how hootsuite manages its growing microservice landscape
TRANSCRIPT
![Page 1: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/1.jpg)
Service ExplosionHOW HOOTSUITE MANAGES ITSGROWING MICROSERVICE LANDSCAPE
Specialist Software Developer - Mobile Web and APIsAdam Arsenault
@Adam_Arsenault
![Page 2: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/2.jpg)
![Page 3: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/3.jpg)
● Road to SOA
● Service Graph
● Voltron
● Demo
● Lessons Learned
What We’ll Talk aboutApp
S2
S3
S4S1
![Page 4: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/4.jpg)
● Founded in 2008
● PHP Monolith
● SOA started in 2013
● Hyper growth
● Continuous Integration
● ~20 services and counting
Road to SOA
![Page 5: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/5.jpg)
“Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations”- Melvin Conway
![Page 6: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/6.jpg)
Engagement MobilePublisher Analytics
Platform Tools Labs
Stable Teams
![Page 7: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/7.jpg)
KABOOM!!!!
![Page 8: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/8.jpg)
Ex. 1 - Integration Failures
● Dev merges changes
● Changes go to staging
● Integration tests fail
● Release pipeline frozen
KABOOM!!!!
![Page 9: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/9.jpg)
Ex. 2 - Production Downtime
● Service goes down in production
● On call / teams affected get notifications
● Sift through flood of notifications to figure out what’s broken
KABOOM!!!!
![Page 10: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/10.jpg)
VISIBILITY
![Page 11: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/11.jpg)
The Service Graph
![Page 12: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/12.jpg)
The Service Graph
App
![Page 13: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/13.jpg)
The Service Graph
App
S1 S2
![Page 14: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/14.jpg)
The Service Graph
App
S2 S3 S4S1
![Page 15: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/15.jpg)
The Service GraphApp
S2
S3
S4S1
![Page 16: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/16.jpg)
The Service GraphApp
S2
S3
S4S1
S5S6
S7
![Page 17: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/17.jpg)
The Service GraphApp
S2
S3
S4S1
S5S6
S7
![Page 18: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/18.jpg)
The Service GraphApp
S2
S3
S4S1
S5S6
S7
![Page 19: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/19.jpg)
![Page 20: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/20.jpg)
The API
![Page 21: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/21.jpg)
DependencySomething that a service needs to function properly.
Types:
1. Internal
2. Traversable
App
Cachedb S1
![Page 22: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/22.jpg)
/status/aboutReturns metadata information about the service or app such as version, description, maintainers, links to documentation, and gets the status of each individual dependency.
![Page 23: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/23.jpg)
/status/:dependencyA configured status endpoint at '/status/:dependency
Examples:● '/status/service-core'● '/status/db'
OK
![Page 24: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/24.jpg)
/status/aggregateReturns the overall status by checking all registered status checks and giving a simple response.
Examples:● OK● CRIT - error message
OK
![Page 25: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/25.jpg)
/status/traverseEnables service graph traversal and execution of an "action" at the last level of traversal.
App
S2
S3
S4S1
![Page 26: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/26.jpg)
Usage
![Page 27: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/27.jpg)
MonitorOK / CRIT
![Page 28: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/28.jpg)
Debug
S1
db S2
![Page 29: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/29.jpg)
Explore and LearnApp
S2
S3
S4S1
S5S6
S7
![Page 30: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/30.jpg)
Document
![Page 31: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/31.jpg)
Status of single machine Overall status of application and services
Alerts / notifications Troubleshoot by drilling down
Monitoring Strategy
![Page 32: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/32.jpg)
![Page 33: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/33.jpg)
Technologies
![Page 34: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/34.jpg)
Architecture
Browser 1 Browser N...
Play App
Status Poller Actor
WS Actor 1
WS Actor N
...
![Page 35: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/35.jpg)
DEMO
![Page 36: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/36.jpg)
“When there is a production issue, I see lots of people go to Voltron to perform diagnostics on what might be wrong”Geordie Henderson - VP Software Development
![Page 37: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/37.jpg)
“Voltron is often the first to tell us when snowflake is down”Brandon Okert - Junior Software Developer Publisher
![Page 38: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/38.jpg)
“When a critical service goes down, everything starts alerting and reporting problems, but Voltron gets through the noise by letting you drill down”Michael Reid - Senior Software Developer Platform
![Page 39: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/39.jpg)
“We suspected the connection between dashboard and Billing Service was broken, but Voltron told us the communication channel was okay.”Martin Jung - Software Developer Mobile Web and APIs
![Page 40: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/40.jpg)
Lessons Learned
![Page 41: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/41.jpg)
• Productivity
• Happiness
Visibility Empowers
![Page 42: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/42.jpg)
• Automate all the things
• Identify problems early and fix
• 10x factor
SOA Tools Early
![Page 43: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/43.jpg)
• Standardize
• Add to a service framework
• Share common status checks
Make Checking Status Easy
![Page 44: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/44.jpg)
• Synchronized views
• Performance
Websockets for Real time
![Page 45: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/45.jpg)
Future Work
![Page 46: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/46.jpg)
Analytics
![Page 47: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/47.jpg)
Real Time Graph View
![Page 48: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/48.jpg)
Open Source
![Page 49: How Hootsuite Manages Its Growing Microservice Landscape](https://reader035.vdocuments.us/reader035/viewer/2022070603/5870a4261a28abcb078b561f/html5/thumbnails/49.jpg)
Thank you!Questions?
Specialist Software Developer - Mobile Web and APIs@Adam_Arsenault
Adam Arsenault