![Page 1: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/1.jpg)
![Page 2: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/2.jpg)
We The Few
Critical Team Composition and Responsibilities For Day 2 Operations
![Page 3: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/3.jpg)
WhoAmI?
KeithStrini…
FieldFacingSolutionsArchitectthatservesasatechnologyanalystfortheUSDepartmentofDefenseandIntelligencecommunities.Iarchitect,developandfieldinformationsystemsacrosstheJointServicesbothCONUSandOCONUS(Korea,Japan,Europe,andtheMiddleEast)andNATO
![Page 4: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/4.jpg)
End Vision
![Page 5: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/5.jpg)
Not End Vision
![Page 6: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/6.jpg)
Options Getting to End Vision
![Page 7: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/7.jpg)
Release Engineering Stratification
AppOperator
DeveloperEnablement Release PlatformReliability
AppProfilesProdManifestUnit/SmokeTestPipelines
ReleaseRepository
CadenceCalendar
PlatformProductManager
SelfServiceDeployment
NoobDevTeam
unit/smoke security uat
pass
fail
![Page 8: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/8.jpg)
Platform Reliability
Coordination Point For All Platform Environment Changes.
■ Creates/Coordinates Cadence Meeting ■ Continuously Develops Resiliency
Probes based on Post Mortems ■ Maintains Environment Parity ■ Enforces Strict Runtime Version Control ■ Communicates Environment Adversity ■ Creates/Coordinate Resiliency Exercise ■ Instruments Distributed Tracing in Ops
Release Engineering
Coordination Point For All New Releases.
■ Attends Final Pre-Release Demo
of Apps ■ Verifies Release Artifacts ■ Coordinates Initial Release Date ■ Collaborates on Downstream
Environment Triage
Developer Enablement
Coordination Point For All New Development Efforts.
■ Creates/Coordinates Platform on
boarding meeting ■ Provides Latest Information about
Platform Environments ■ Provides tooling around Dev
Services, CI/CD environment, governs code repositories, etc
Coordination Execution Resiliency
PRACTICES PRACTICES PRACTICES
![Page 9: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/9.jpg)
Troubleshooting Complex System Failure
![Page 10: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/10.jpg)
Simple System Failure (Daddy I want to build a car)
![Page 11: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/11.jpg)
Simple Complexity
![Page 12: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/12.jpg)
Failure is Inevitable, Hope is Not a Strategy
Care And Feeding Of New Releases To Ensure Early Intervention
Decides on Feature Maturity From a Stability Perspective
Been There, Done That, Got the swag!
Deep Understanding of How New Efforts Deploy Into Operations
Unit/Smoke
Focuses Developers on Contract Based Testing For Integration
Contract Feature Flagging Canary Distributed Tracing
Developers DevEnablement Release Operations Operations
![Page 13: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/13.jpg)
Decoupled Integration – Contract Testing
• IfSpeedIsWhatyouWant,End-To-EndTestingisnothowyougetthere.
• Gettingfeedback…thisweek?• Areyoumockingme?
• SingleSourceofTruth…• VerifyingtheGoods
• IsolationtestingofSingleServices(ProviderorConsumer)
• Idonotthinkthatmeanswhatyouthinkitmeans(SemanticTesting)
• ComplexityFromSimplicity,NotComplexityFromComplexity
• TestData…let’snotignoretheelephantintheroom
• Stability,Stability,Stability…we’retalkingOperationsnotScienceExperiments
• AhSunsets…• PayingoffTechnicaldebtbysubtractionand
addition• Yougetme…youreallygetme
• ConsumerdefinedAPIs
![Page 14: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/14.jpg)
Maintaining Operational Velocity – Feature Flagging
• Idunno.Youtellmewhatyouwant.• NonTechsgettinginontheAction
• OksomostofitworksbutIgottasenditback?• Beautyofcontextencapsulation• Waitingforafeaturelikeyou
• Iseehowwedo1appbuthowdoImanage1000s?• Sowhatifwedon’tknowexactlywhatouruserswant?• AhSunsets…
• PayingoffTechnicaldebtbysubtractionandaddition
![Page 15: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/15.jpg)
• Almost trust you • Canaries – Profiling the CPU,
memory, disk usage, cache synch • Rollback/Roll Forward Strategies
• Blue/Green Deployments • The case of stateless • The case of stateful
(transactions, migrating data) • Infrastructure Isolation
• A/B testing
Predictive Fire Fighting In Operations – Canary/Distributed Tracing • It not you, its me
• Distributed Tracing • Yes we are talking scale here • But that’s a lot of instrumentation • Correlation is tough
• Good definitions of SLO/SLIs
• Threshold tuning
![Page 16: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/16.jpg)
Operations as the Caretaker of Code? • Yourbabyisugly,Ourbabyiscute
• PlatformasProducthelpsalignourinterests• Automationhelpsusberesponsiveasateamtoourendusers
• Lot’sofupfrontpainisbetterthanchronicpainindefinitely
• Successasdefinedbyrhythm
![Page 17: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/17.jpg)
• Change Inherently Creates Failure
• Alignment of Values • One Team One Fight
• Joint SLOs • Platform SLIs • Application SLIs • Instrumentation
Get Off My Lawn!
• Growing up is hard to do • Graduating Product Teams to
Self-Service Deployments
![Page 18: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/18.jpg)
Growing Up Is Hard to Do • Resiliency Exercises as the litmus test • Communicating the attitude that Stability is a
team sport • Starting the Cycle Over
• Capturing Lessons Learned from every class • Knowledge Transparency aides the greater
team
![Page 19: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/19.jpg)
In Conclusion
“When learning something new you have to practice going slow, if you want to eventually go fast forever”
![Page 20: We The Few - Linux Foundation Events€¦ · • Test Data…let’s not ignore the elephant in the room • Stability, Stability, Stability… we’re talking Operations not Science](https://reader033.vdocuments.us/reader033/viewer/2022053021/5f850ba44b1f5c57db7c5129/html5/thumbnails/20.jpg)
Questions?