upside of downtime - lenny rachitsky

http://gapingvoid.com/

Sunday, June 20, 2010



The Upside of DowntimeTurning disaster into opportunity


Who’s had a site go down?


Who’s hasn’t had a site go down?


There’s always that one guy!


Downtime sucks

Source: http://www.motivatedphotos.com/?id=8080


http://www.motivatedphotos.com/?id=8080

http://www.motivatedphotos.com/?id=8080

Why downtime sucks

Business

$0

$750

$1,500

$2,250

$3,000

0 2 4 6 8 10 12 14 16 18 20 22

Sales


Why downtime sucks

Business

Brand


Why downtime sucks

Business

Brand

You


Why downtime sucks

Business

Brand

You

Users


Downtime = Bad! (Duh)


Approach #1Don’t fail


Source: http://kansansforlife.files.wordpress.com/2009/12/titanic.jpg


http://kansansforlife.files.wordpress.com/2009/12/titanic.jpg

http://kansansforlife.files.wordpress.com/2009/12/titanic.jpg

“Everything fails all the time”-- Werner Vogels (Amazon, CTO)


Your site will fail

Werner Vogels (Amazon, CTO)


Why?!?


Risk Homeostasis

Why Failure Happens

Source: http://joshuahind.files.wordpress.com/2009/09/bicycle-crash.jpg


http://joshuahind.files.wordpress.com/2009/09/bicycle-crash.jpg

http://joshuahind.files.wordpress.com/2009/09/bicycle-crash.jpg

Risk Homeostasis

Black Swan

Why Failure Happens

Source: Amazon.com


Risk Homeostasis

Black Swan

Unknown unknowns

Why Failure Happens

Source: http://www.apoliticus.com/wp-content/uploads/2009/01/6_21_080306_rumsfeld.jpg


http://www.apoliticus.com/wp-content/uploads/2009/01/6_21_080306_rumsfeld.jpg

http://www.apoliticus.com/wp-content/uploads/2009/01/6_21_080306_rumsfeld.jpg

Risk Homeostasis

Black Swan

Unknown unknowns

Change

Why Failure Happens

Source: http://bozark.net/wordpress/wp-content/uploads/2008/09/barack_obama_change_fairey.jpg


http://bozark.net/wordpress/wp-content/uploads/2008/09/barack_obama_change_fairey.jpg

http://bozark.net/wordpress/wp-content/uploads/2008/09/barack_obama_change_fairey.jpg

Risk Homeostasis

Black Swan

Unknown unknowns

Change

Many small failures

Why Failure Happens

Source: http://www.biojobblog.com/uploads/image/dominos.jpg


http://www.biojobblog.com/uploads/image/dominos.jpg

http://www.biojobblog.com/uploads/image/dominos.jpg

Risk Homeostasis

Black Swan

Unknown unknowns

Change

Many small failures

Humans

Why Failure Happens

Source: http://www.librarian.net/talks/clc/CLC.key/SJ_Shoulder_Shrug.jpg


http://www.librarian.net/talks/clc/CLC.key/SJ_Shoulder_Shrug.jpg

http://www.librarian.net/talks/clc/CLC.key/SJ_Shoulder_Shrug.jpg

Not unusual

Polisherblocked

Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm


http://www.gladwell.com/1996/1996_01_22_a_blowup.htm


Not unusual Not expected

Polisherblocked

Moisture leaks into air system





Not unusual

Polisherblocked


Flow of cold water stopped

Not expected Not good





Not unusual

Polisherblocked



Not expectedBackup disabled





Not unusual

Polisherblocked




Indicator blockedDoh!





Not unusual

Polisherblocked




Indicator blocked

Relief valve broken

Doh!

Dammit





Not unusual

Polisherblocked




Indicator blocked

Relief valve broken

Gauge broken

Doh!

Dammit

WTF





Not unusual

Polisherblocked



Meltdown


Indicator blocked

Relief valve broken

Gauge broken

Doh!

Dammit





Source: http://support.rightscale.com/09-Clouds/AWS/02-Amazon_EC2/Designing_Failover_Architectures_on_EC2/03-Advanced_Failover_Architecture


http://support.rightscale.com/09-Clouds/AWS/02-Amazon_EC2/Designing_Failover_Architectures_on_EC2/03-Advanced_Failover_Architecture

http://support.rightscale.com/09-Clouds/AWS/02-Amazon_EC2/Designing_Failover_Architectures_on_EC2/03-Advanced_Failover_Architecture

“accidental power failure”

Source: http://www.datacenterknowledge.com/archives/2010/06/16/power-failure-kos-intuit-sites-for-24-hours/


http://www.datacenterknowledge.com/archives/2010/06/16/power-failure-kos-intuit-sites-for-24-hours/

http://www.datacenterknowledge.com/archives/2010/06/16/power-failure-kos-intuit-sites-for-24-hours/

“traffic accident damaged a nearby utility transformer”

Source: http://www.datacenterknowledge.com/archives/2007/11/13/truck-crash-knocks-rackspace-offline/


http://www.datacenterknowledge.com/archives/2007/11/13/truck-crash-knocks-rackspace-offline/

http://www.datacenterknowledge.com/archives/2007/11/13/truck-crash-knocks-rackspace-offline/

“unfortunate code change”Source: http://www.datacenterknowledge.com/archives/2010/06/11/errant-code-change-crashes-10-million-blogs/


http://www.datacenterknowledge.com/archives/2010/06/11/errant-code-change-crashes-10-million-blogs/

http://www.datacenterknowledge.com/archives/2010/06/11/errant-code-change-crashes-10-million-blogs/

“Unhappy customers may get some attention, but unhappy networked customers can quickly impact your business”

-- Clay Shirky

Source: http://happenupon.files.wordpress.com/2009/02/technology-guru-clay-shir-001.jpg, http://scholarlykitchen.sspnet.org/2010/03/02/shirky-at-nfais-how-abundance-breaks-everything/


http://happenupon.files.wordpress.com/2009/02/technology-guru-clay-shir-001.jpg

http://happenupon.files.wordpress.com/2009/02/technology-guru-clay-shir-001.jpg

http://scholarlykitchen.sspnet.org/2010/03/02/shirky-at-nfais-how-abundance-breaks-everything/

http://scholarlykitchen.sspnet.org/2010/03/02/shirky-at-nfais-how-abundance-breaks-everything/

http://labs.webmetrics.com/crowdsourceduptimeSunday, June 20, 2010

http://labs.webmetrics.com/crowdsourceduptime

http://labs.webmetrics.com/crowdsourceduptime

Recap


Your site will fail


Your site will fail+Downtime is bad


Your site will fail+Downtime is bad+Everyone will find out


Your site will fail+Downtime is bad+Everyone will find out=Screw it, I’ll become a lumberjack

Source: http://sbadrinath.files.wordpress.com/2009/03/different26rqcu3.jpg


http://sbadrinath.files.wordpress.com/2009/03/different26rqcu3.jpg

http://sbadrinath.files.wordpress.com/2009/03/different26rqcu3.jpg

“Embrace fear of outages and degradation. Use it to guide your architecture, your code, your infrastructure. So lean into it.”

-- John Allspaw, VP Tech. Ops at Etsy


Approach #2Prepare for downtime


Disclaimer: Try hard to avoid downtime


Learning by example...


Case Study #1Facebook


“The larger issue here isn't just that a portion of Facebook's platform has gone down - numerous web services have issues from time to time, including everything from Gmail to Twitter. An outage of this length, however, with no official communication from the company itself is disturbing.”

-- N.Y. Times


Downtime Disturbing

Facebook


Case Study #2Google App Engine


Downtime Kudos

Google App Engine


Case Study #3Atlassian


Downtime

Atlassian

Bravo


http://atlassian.com/


http://atlassian.com

http://atlassian.com

Downtime:Opportunity to Build Trust


Downtime:Opportunity to Destroy Trust


How To: Prepare for Downtime


Something > Nothing


Upside of Downtime Framework 1.0

Oh crapLife is good That sucked

Time



CommunicatePrepare Explain

Time



1. Communication channel



Something is wrong

Can’t tell if it’s me or you

I’ll assume it’s you

You suck



Something is wrong

Can’t tell if it’s me or you

I’ll assume it’s you

I know it’s youTell me when you’re back

You suck a lot less





1. Communication channel Easy to find




Hosted off-site




Hosted off-site

Real-time / automated


7 keys for public health dashboards

1. Must show current status for each “service”

2. Data must be accurate and timely

3. Must be easy to find

4. Must provide details for events in real time

5. Provide historical uptime and performance data

6. Provide a way to be notified of status changes

7. Provide details on the data is gathered

Source: http://www.transparentuptime.com/2008/11/rules-for-successful-public-health.html


http://www.transparentuptime.com/2008/11/rules-for-successful-public-health.html

http://www.transparentuptime.com/2008/11/rules-for-successful-public-health.html



Hosted off-site


2. Process




Hosted off-site


2. Process Authority




Hosted off-site



Mean-Time-To-Communicate (MTTC)




Hosted off-site



Mean-Time-To-Communicate (MTTC)

On-call/drills/escalations/etc.Sunday, June 20, 2010

Your servers


Prepare ExplainCommunicate

1. Communicate



1. Communicate Use communication channel




MTTC




MTTC

Who/what is affected




MTTC


When the incident started




MTTC



ETA




MTTC



ETA

Update regularly




MTTC



ETA

Update regularly

2. Fix it!Sunday, June 20, 2010

Phew, close one!



1. Postmortem



1. PostmortemAdmit failure

Source: http://en.blog.wordpress.com/2010/02/19/wp-com-downtime-summary/


http://en.blog.wordpress.com/2010/02/19/wp-com-downtime-summary/

http://en.blog.wordpress.com/2010/02/19/wp-com-downtime-summary/



Sound like a human

Source: http://www.bureauofcommunication.com/compose/apology


http://www.bureauofcommunication.com/compose/apology

http://www.bureauofcommunication.com/compose/apology


“We apologize for any inconvenience this may

have caused”




Sound like a human

Start time and end time

Source: https://groups.google.com/group/google-appengine/browse_thread/thread/a7640a2743922dcf


https://groups.google.com/group/google-appengine/browse_thread/thread/a7640a2743922dcf




Sound like a human


Who/what was impacted

Source: http://techcrunch.com/2009/11/02/large-scale-downtime-at-rackspace-cloud/


http://techcrunch.com/2009/11/02/large-scale-downtime-at-rackspace-cloud/

http://techcrunch.com/2009/11/02/large-scale-downtime-at-rackspace-cloud/



Sound like a human



What went wrong

Source: http://www.zendesk.com/2010/03/tuesday-double-whammy.html


http://www.zendesk.com/2010/03/tuesday-double-whammy.html

http://www.zendesk.com/2010/03/tuesday-double-whammy.html



Sound like a human



What went wrong

Lessons learned

Source: http://graysky.org/2010/02/downtime-postmortem/


http://graysky.org/2010/02/downtime-postmortem/

http://graysky.org/2010/02/downtime-postmortem/



Sound like a human



What went wrong

Lessons learned



“I was completely overwhelmed by the amount of positive feedback and support I received.”




Sound like a human



What went wrong

Lessons learned

2. Improve for the futureSunday, June 20, 2010

“Google is not just saying sorry, they are actually implementing serious changes which probably represents millions of dollars of development to help make sure this doesn't happen again.”


Source: http://news.ycombinator.com/item?id=1168493


http://news.ycombinator.com/item?id=1168493

http://news.ycombinator.com/item?id=1168493


Source: https://groups.google.com/group/google-appengine/browse_thread/thread/a7640a2743922dcf





Be human



Be authentic



Be transparent



Accept responsibility



Learn and improve


Trust



CommunicatePrepare Explain1. Communication channel - Easy to find - Off-site - Real-time

2. Process - Give authority - M.T.T.C. - On-call/escalations

1. Communicate - Use channel - M.T.T.C. - Who/what affected - When started - ETA to resolution - Update regularly

2. Fix it!

1. Post-mortem - Admit failure - Sound like a human - Start time and end time - Who/what was impacted - What went wrong - Lessons learned

2. Learn and improve






Be HumanBe TransparentBe Prepared + +




2. Fix it!





Be HumanBe TransparentBe Prepared + +

Trust=




2. Fix it!


Disclaimer:Don’t screw up too often


Transparent Not Transparent

Caught

Not Caught

Downtime Prisoner’s Dilemma



Caught

Not Caught Win




Caught

Not Caught

Big Loss

Win




Caught

Not Caught

Big Win Big Loss

Win




Caught

Not Caught

Big Win Big Loss

Win Win



BenefitsGain trust

Reduce churn, increase loyalty

Reduce support costs

Ability to control the message

Competitive advantage

More time to focus on the actual problem

Reduce stress


Change != Easy


Change != Impossible


Keys to Adoption

Getting past a culture of “hide the problem”


Keys to Adoption


Overriding commitment to want to improve


Keys to Adoption



Available resources to improve


Keys to Adoption




Pain


Keys to Adoption




Pain

Buy-in


Product Management

Support

Sales/Marketing

Engineering/Operations


Product Management

Support

Default: Lets wait for complaints

Sales/Marketing



Product Management

Support


Reality: Proactiveness => Forgiveness

Sales/Marketing



Product Management

Support


Default: Too much work

Sales/Marketing




Product Management

Support



Reality: More upfront, less when it matters

Sales/Marketing




Product Management

Support




Default: Don’t want to look bad

Sales/Marketing





Product Management

Support





Reality: Opportunity to learn/improve

Sales/Marketing



Product Management

Support






Default: I don’t want my customers to knowSales/Marketing




Product Management

Support






Default: I don’t want my customers to know

Reality: They’ll find out, better from usSales/

Marketing




Source: http://delicious.com/lennysan/healthdashboard


http://delicious.com/lennysan/healthdashboard

http://delicious.com/lennysan/healthdashboard

Simple as that!


Your site will still fail!


“The measure of a society is how well it transforms pain and suffering into something worthwhile.”

-- Fredrick Nietzsche


“The measure of a company is how well it transforms pain of downtime into something worthwhile.”

-- Lenny Rachitsky

Source: Original quote inspired by Fredrick Nietzsche


Bare minimum:Register a Twitter account


Lenny Rachitsky@lennysanhttp://www.transparentuptime.com/

Webmetrics/Neustar@webmetricshttp://www.webmetrics.com/

Slides: http://bit.ly/upside-of-downtime

Thank You


http://www.transparentuptime.com

http://www.transparentuptime.com

http://www.webmetrics.com

http://www.webmetrics.com

http://bit.ly/upsideofdowntime

http://bit.ly/upsideofdowntime

Bonus





2. Fix it!








2. Fix it!




"Unlikely that an accidental surface or subsurface oil spill would occur from the proposed activities"

-- Exploration and environmental impact plan

Source: http://en.wikipedia.org/wiki/Deepwater_Horizon_drilling_rig_explosion


http://en.wikipedia.org/wiki/Deepwater_Horizon_drilling_rig_explosion

http://en.wikipedia.org/wiki/Deepwater_Horizon_drilling_rig_explosion




2. Fix it!





“Be not afraid of transparency; some are born transparent, some achieve transparency, and others have transparency thrust upon them.”

-- Burrowed from William Shakespeare


Making change1. Find the bright spots - (this presentation has a bunch)



2. Script the critical moves - (framework)




3. Point to the destination - (W.W.G.D.)





4. Find the feeling - (how would you feel?)






5. Shrink the change - (start small)







6. Grow your people - (everyone is learning as they go)








7. Tweak the environment - (create a simple process)









8. Build habits - (build process organically)









8. Build habits - (build process organically)

9. Rally the herd - (get buy in, rest will follow)


upside of downtime - lenny rachitsky

Documents

unusual source

risk homeostasis source

clayshir http

backup disabled source

small failures source

expectednot good source

small failures humans

unfortunate code change