rollback: the impossible dream

Download Rollback: The Impossible Dream

Post on 13-May-2015




2 download

Embed Size (px)


Roll back doesn’t exist. It’s not real. It’s a fantasy, a dream, a delusion. Any vendor who tells you they have a roll back capability is lying to you. And lying to you in a downright dangerous way that will come back to haunt you at 4am in a war room when someone says:“We can’t fix this. Let’s roll back the deployment.”This talk is designed to explain and demonstrate to Operations staff:Why roll back is a fantasy and explained with a dash of Werner HeisenbergWhy it is dangerous and how you can recognize when you’re about to get trappedHow you can avoid falling into that trap of considering it an appropriate compensating control.It’ll also explain what you can actually do operationally instead of “rolling back”. This will cover other alternative compensating controls that can help you get running again and resolve your outage whilst still allowing you to find root cause.


  • 1.RollbackThe Impossible Dreamby James Turnbull jamtur01 @ github kartar @ twitter jamesturnbull on freenodejames @

2. About MeVP Technical Operations at Puppet Labs Puppet guyRuby guy Talks funny 3. A show of hands 4. Who thinks theyknow what rollback is? 5. Last set of hands 6. YMMV 7. Definitions 8. Traditional 9. Modern 10. Fact or Fiction? 11. Accept certain constraints 12. Constraint #1Apply sufficientcapital 13. Constraint #2 Idempotent 14. Constraint #3Cascade-less failure 15. Constraint #4 Resources 16. A PhilosophicalDigression 17. If I know where I amI dont know how I got there If I know how I got there I dont know where I am 18. Very few systemsare truly deterministic 19. A MathematicalDigression 20. On system rollback and totalised elds An algebraic approach to system changeMark Burgess and Alva Couch 20th June 2011 21. So whats wrong with rollback? 22. Risk 23. Learning frommistakes 24. Complex systems are complex 25. Human error 26. What is the problemrollback is trying to solve? 27. What is the problem YOU are trying tosolve? 28. So how can wemitigate Rollback shortcomings? 29. PreventativeDesign 30. Rollback is (often) anarchitecture problem 31. Increase Resilience 32. OperationalIntelligence 33. A little bit of DevOps in every byte 34. Small, iterativechanges 35. Accept that failurehappens 36. We cant test that? Okay we can roll it back if it breaks 37. Assumption is themother of all fuckups* 38. But the system cant be{run|upgraded|deployed} like that because 39. Conclusions 40. Rollback is possible but not probable 41. If you have to haverollback accept constraints 42. You can mitigate the need for it 43. Thank you!Questions/Insults? jamtur01 @ github kartar @ twitter jamesturnbull on freenodejames @