john reese - usenix · software engineer security training / experience ... java servlet filter)...
TRANSCRIPT
![Page 1: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/1.jpg)
John Reese
![Page 2: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/2.jpg)
No Haunted Graveyardsa lightning talk by jtr of Google
image courtesy of https://www.pexels.com/photo/night-building-forest-trees-42263/CC0 license
![Page 3: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/3.jpg)
It's easy in this line of work to become superstitious: "this part of the system is risky -- let's not touch it." That’s a haunted graveyard.
I will argue, I assume compellingly, that we should invade and re-consecrate any area that is showing signs of turning into a haunted graveyard.
image courtesy of https://www.pexels.com/photo/night-dark-halloween-horror-782/ CC0 license
![Page 4: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/4.jpg)
Examples are everywhereThe race condition you usually win.
Putting functionality into the wrong system because you aren’t confident changing the right system.
Leaving sharp edges on an API because you don’t know who’s using it.
The legacy system nobody wants to work on that continues lurching along till somebody rewrites it from scratch.
image courtesy of https://xkcd.com/1172/Creative Commons Attribution-NonCommercial 2.5 License
![Page 5: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/5.jpg)
The haunted system nameActual text from a 5-year-old file called ZOMG_DO_NOT_CREATE_system-zone:
The (system-zone) cell was turned down using some deep
magic that makes it dangerous to create a new cell with
the same name…
(The name rendered here as “system-zone” was the instance of the given system in a specific zone. We reuse zone names.)
image courtesy of https://www.pexels.com/photo/dry-animal-gift-dangerous-38438/ CC0 license
![Page 6: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/6.jpg)
Our job is control, not to avoid all danger at all costs.
Things you’re afraid to change are serious existential risks.
Otherwise, superstition eats away at the edges of our world and we lose the ability to control it.
Use your error budget and find out what happens when you do that thing everyone’s scared of.
image courtesy of https://www.pexels.com/photo/white-caution-cone-on-keyboard-211151/ CC0 license
![Page 7: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/7.jpg)
But beware Chesterton’s Fence
“There [is] a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, ‘I don’t see the use of this; let us clear it away.’ To which the more intelligent type of reformer will do well to answer: ‘If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.’” -- G.K. Chesterton image courtesy of
https://www.pexels.com/photo/penguin-on-wooden-dock-during-daytime-172206/ CC0 license
![Page 8: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/8.jpg)
Raja Selvaraj
![Page 9: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/9.jpg)
Measuring Reliability through VALET metricsRaja Selvaraj, The Home Depot
![Page 10: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/10.jpg)
The Home Depot
❑ #1 Home Improvement Retailer in the world❑ 94B+ annual revenue❑ Top 10 e-commerce Website ($5B+ annual revenue)❑ 2016 Top 50 Most innovative companies❑ Fast company
❑ 2015 Internet Retailer of the year❑ Internet Retailer
❑ We are hiring
![Page 11: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/11.jpg)
Why measure Reliability
❑ Quantify the health of your service❑ Express quality ❑ No measurement, No improvement
![Page 12: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/12.jpg)
What is VALETA Framework for Reliability
Volume: A measure of how much demand is being placed on your service
Availability: uptime requirement of the service; typically at least three 9s
Latency: how quickly the service should respond, usually at the 90th percentile or higher
Errors: normal rate of errors vs good requests. Define what is an error for this service.
Tickets: Incidents or bugs introduced into production. Ideally zero. A good indicator of quality and technical debt of the service.
![Page 13: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/13.jpg)
Thank You
❑ Thank You
![Page 14: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/14.jpg)
Cezar Alevatto Guimaraes
![Page 15: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/15.jpg)
Lessons Learned from Transforming System Engineers
into SRE at Microsoft Azure
cezarguimaraes.com
![Page 16: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/16.jpg)
EliteSoftware Engineering
Service Engineering
SRE
![Page 17: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/17.jpg)
EliteSoftware Engineering
Service Engineering
SRE
![Page 18: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/18.jpg)
SRE managerTech leadEngineer
![Page 19: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/19.jpg)
SRE manager
![Page 20: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/20.jpg)
EliteSoftware Engineering
Service Engineering
SRE
![Page 21: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/21.jpg)
Pair up with mentors
![Page 22: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/22.jpg)
EliteSoftware Engineering
Service Engineering
SRE
![Page 23: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/23.jpg)
It’s ok to move on
![Page 24: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/24.jpg)
Tech Lead
![Page 25: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/25.jpg)
Make space for teaching
![Page 26: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/26.jpg)
Manage expectation
![Page 27: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/27.jpg)
Create our culture
![Page 28: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/28.jpg)
Engineer
![Page 29: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/29.jpg)
Different learning tools
![Page 30: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/30.jpg)
Be open to work differently
![Page 31: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/31.jpg)
Final thoughts
![Page 32: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/32.jpg)
Start smallCreate your culture
cezarguimaraes.com
![Page 33: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/33.jpg)
Lei Lopez
![Page 34: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/34.jpg)
4 Min Deploys
Lei Lopez @emojineeer
No Engineers Necessary
![Page 35: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/35.jpg)
Deploy Pipeline
MergeBuild
containerRun CI
Click to deploy
Ship to production
Before
https://github.com/Shopify/shipit-engine
![Page 36: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/36.jpg)
Autodeploy?
UNSHIPPED CHANGES
DEPLOY LOGJAM
![Page 37: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/37.jpg)
Deploy Pipeline
MergeBuild
containerRun CI
Click to deploy
Ship to production
After
https://github.com/Shopify/shipit-engine
![Page 38: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/38.jpg)
High Expectations• Since developers are expected to stick around and verify their
changes, the system must be fast and reliable
Trust• As the pull request is the last stop before production, developers
are trusted to test and review to deliver high-quality code
Fearlessness• Given that all it takes is a click of the merge button
Impact on culture
![Page 39: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/39.jpg)
Keeping devs in the loop
![Page 40: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/40.jpg)
Something goes wrong!?1. Lock deploys
• Via ChatOps to prevent new deploys from shipping
2. Rollback• Specific commits via Shipit UI
3. Verify• That things are back in a good state
4. Revert commit• On GitHub without waiting for CI
5. Unlock deploys• To start shipping again
![Page 41: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/41.jpg)
Deploy Pipeline
MergeBuild
containerRun CI
Ship to production
Merge queue
Enqueue merge
https://github.com/Shopify/shipit-engine
Merge
![Page 42: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/42.jpg)
Future Work
● Speed
● Stability
● Auto-rollback
● Canaries
● https://github.com/Shopify/kubernetes-deploy
![Page 43: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/43.jpg)
Thank you! @emojineeer
![Page 44: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/44.jpg)
Office HoursMeet members of Shopify’s production engineering team at our booth and chat with us about the following topics:
Monday10 am to 1 pm Handling Massive Flashes of High-Write Traffic1 pm to 3:45 pm Road to an SRE Model3:45 pm to 6:30 pm Tools for Tracking Service Infrastructure at Scale
Tuesday10 am to 1 pm Auto-deploying Anywhere and At Any Time1 pm to 3:45 pm Road to an SRE Model3:45 pm to 7:00 pm Automating Data Center Deployments
Keep In Touch- Check out our blog at shopify.com/engineering
- Follow us on Twitter at @shopifyeng
![Page 45: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/45.jpg)
Tom Schmidt
![Page 46: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/46.jpg)
SR (securit) ETom Schmidt • March 12, 2017
![Page 47: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/47.jpg)
Security Engineers
What is a security engineer?
○ Software Engineer○ Security training / experience○ “Belief in and aptitude for developing software
systems to solve complex problems”
… why is it worth hiring a team of them?
![Page 48: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/48.jpg)
But...
● … system administrators are less expensive!● … we don’t have time for that!● … I already have a security focal!● … none of my engineers want to install patches!
![Page 49: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/49.jpg)
Security is not a checkbox
● As scope expands, so too do security and compliance requirements
● Many security and compliance requirements are ongoing
● Toil: “Manual, repetitive, automatable, devoid of enduring value, and scales linearly”
![Page 50: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/50.jpg)
Monitoring and Logging: Meeting the Requirement
Dump all logs to event manager (QRadar)
![Page 51: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/51.jpg)
Monitoring and Logging: Scaling out of control
Mandate components to send relevant, well-formatted logs
![Page 52: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/52.jpg)
Monitoring and Logging: Engineering a solution
● Design and develop a common solution (Node middleware, Java servlet filter)
● Easy integration / configuration for every component
● Scalable, efficient, effective● Bonus: Contribute to the open
source community
![Page 53: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/53.jpg)
Secure Engineering = (Maintainable)
Velocity
● Say NO to “Stop and Go”● (Also, say no to slow)
● Automation and common solutions as the DEFAULT approach
● Consistency, Efficiency● (Tempered) Acceleration
![Page 54: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/54.jpg)
Thanks!
![Page 55: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/55.jpg)
Dale Neufeld
![Page 56: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/56.jpg)
How three changes led to big increases
in on-call health
![Page 57: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/57.jpg)
57
“Devops-focused” operationsTeam of 12 supporting:● ~1000 servers● ~175 engineers● Dozens of services and 5-10 deploys per day
On-call• Operations team responsible for primary on-call,
supported by single dev on-call rotation
Looking back to 2014
![Page 58: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/58.jpg)
58
• Heavy toil workload
• Breadth of expert knowledge needed to support at scale:
rails, mysql, redis, nginx, memcached, elasticsearch, kafka, etc.
• Engineering teams building new infra and “dumping” it on operations
The Problems
![Page 59: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/59.jpg)
59
Change #1: fix team composition
![Page 60: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/60.jpg)
60
• Set expectation for engineering teams to own their services in production, supported by production engineering rather than the inverse
Change #2: expect more
![Page 61: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/61.jpg)
61
• Commit to 6 person on-call teams minimum (even though at the time some were only 3-4 people)
• Give time back. After a week of on-call go ahead and take the following Friday off
Change #3: recognize the burden
![Page 62: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/62.jpg)
62
Results
Detect
Protect
Reflect
Correct
![Page 63: John Reese - USENIX · Software Engineer Security training / experience ... Java servlet filter) Easy integration / configuration for every component Scalable, efficient, effective](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ec9ec5c6dfdd772c809569a/html5/thumbnails/63.jpg)
63
Office HoursMeet members of Shopify’s production engineering team at our booth and chat with us about the following topics:
Monday10 am to 1 pm Handling Massive Flashes of High-Write Traffic1 pm to 3:45 pm Road to an SRE Model3:45 pm to 6:30 pm Tools for Tracking Service Infrastructure at Scale
Tuesday10 am to 1 pm Auto-deploying Anywhere and At Any Time1 pm to 3:45 pm Road to an SRE Model3:45 pm to 7:00 pm Automating Data Center Deployments
Keep In Touch- Check out our blog at shopify.com/engineering
- Follow us on Twitter at @shopifyeng