cloud expo - flying two mistakes high

Flying Two Mistakes HighA Guide to Not CrashingLee Atchison, Principal Cloud Architect and Advocate at New Relic, Inc.

©2008-16NewRelic,Inc.Allrightsreserved.

2

SafeHarbor


This document and the information herein (including any information that may be incorporated by reference) isprovided for informational purposes only and should not be construed as an offer, commitment, promise orobligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code,functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not bereplicated or disclosed without New Relic’s express written permission.

Such information may contain forward-looking statements within the meaning of federal securities laws. Anystatement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates,goals, or other characterizations of future events is a forward-looking statement. These forward-lookingstatements can often be identified as such because the context of the statement will include words such as“believes,” “anticipates,”, “expects” or words of similar import.

Actual results may differ materially from those expressed in these forward-looking statements, which speak onlyas of the date hereof, and are subject to change at any time without notice. Existing and prospective investors,customers and other third parties transacting business with New Relic are cautioned not to place undue relianceon this forward-looking information. The achievement or success of the matters covered by such forward-lookingstatements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantialrisks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, orachievements to differ materially from those expressed or implied in any forward-looking statement. Furtherinformation on factors that could affect such forward-looking statements is included in the filings we make withthe SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relationswebsite at http://ir.newrelic.com or the SEC’s website atwww.sec.gov.

New Relic assumes no obligation and does not intend to update these forward-looking statements, except asrequired by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, withrespect to the information provided.

3

WhoamI?


Specializein:

Cloudcomputing

Services&Microservices

Scalability, Availability

28yearsinindustry7inAmazonRetail&AWS(BuiltSW/VGAppStore,AWSElasticBeanstalk)

4inNewRelic(ArchitectureLead,Cloud,ServiceMigration)

@leeatchison leeatchison

4

Iwanttotellyouastory…


5



Youtellmeifthisisokornot…

6



Thiswasarecentlyoverheardconversation…

Youtellmeifthisisokornot…

7

Isthisok?


“Wewerewonderinghowchangingasettingon

ourMySQLdatabasemightimpactourperformance…

8

Isthisok?


“Wewerewonderinghowchangingasettingon

ourMySQLdatabasemightimpactourperformance…

…butwewereworriedthatthechangemaycauseourproductiondatabasetofail…”

9

Isthisok?


“…Sincewedidn’twanttobringdownproduction,wedecidedtomakethechangetoourbackup

(replica)databaseinstead…

UnderConstruction

…butwewereworriedthatthechangemaycauseourproductiondatabasetofail…”

10

Isthisok?


“…Sincewedidn’twanttobringdownproduction,wedecidedtomakethechangetoourbackup

(replica)databaseinstead…

…Afterall,itwasn’tbeingusedforanything

atthemoment.”

UnderConstruction

11

Isthisok?


Until,ofcourse,thebackupwasneeded…

UnderConstructionX

12

Isthisok?


Until,ofcourse,thebackupwasneeded…

Thiswasatruestory

UnderConstruction!!!!X

X

13

Iflyradiocontrolledmodelairplanes


“Keepyourplaneatleasttwomistakeshigh.”

There’sanoldadage:

14

ButWhy?



15

WhyTwoMistakesHigh?


Youperformsomestunt,anditfails…Youlosealtitude

Youalwayswanttobehighenoughtomakeamistake,

evenifyou’vejustmadeamistake…

16

WhyTwoMistakesHigh?



Now,youarelower,andyouaretryingtorecover



17

WhyTwoMistakesHigh?




Youwanttostillbehighenough, sothatifyoumakeanothermistake,youwon’tcrash



18

WhyTwoMistakesHigh?




Youwanttostillbehighenough, sothatifyoumakeanothermistake,youwon’tcrash



19

Putanotherway…


…evenifyouarecurrentlyrecovering

fromamistake

…flyingtwomistakeshigh,youcanalwayshaveabackupplanfor

recovering fromamistake

20©2008-16NewRelic,Inc.Allrightsreserved.

Don’tscrewup...

…whileyouarescrewingup

Thissameapplieswhenbuildinghighlyavailable,highscaleapplications

21©2008-16 New Relic, Inc. All rights reserved.

22

Howdowekeep“TwoMistakesHigh”inanapplication?


Walkthroughramifications andrecoveryplan



Makesurerecoveryplanworks§ Hasnomistakes§ Hasitsownrecoveryplan




Ifrecoveryplandoesn’twork…

it’snotagoodrecoveryplan

Makesurerecoveryplanworks§ Hasnomistakes§ Hasitsownrecoveryplan



EXAMPLEHowmanynodesdoweneed?

26



HowmanynodesdoIneedtohandlemytrafficdemands?

BuildingaService§ Designedtohandle1,000req/sec(assumesinglenode=300req/sec)

27



Right???

§ ceil[1,000/300]=4nodes§ Withfournodes,canhandleourtraffic§ PLUS wehaveenoughnodesthatwecanloseone!Wehaveredundancy!

28

EXAMPLEWellno…


Youthink4nodesgivesyouredundancy,butitdoesn’t...

Ifyouloseoneofthosenodes:§ Remainingnodescanonlyhandle300*3=900req/sec

§ Cannothandlethe1,000req/secload

29

EXAMPLEHowmanydoweneed?


4nodes...allowshandlingourtrafficbutwecannothandleanodefailure

5nodes...allowshandlingasinglenodefailure

But…

Noupgrading

6nodes...amulti-nodefailure,

Or…

Handleafailureduringanupgrade

30

LESSONFlyTwoMistakesHigh


Evenifyouthinkyouhaveredundancy…

§ Thinkthroughthefailuremodes§ …and makesure


EXAMPLERollingupgrades

32

EXAMPLERollingupgrades


Areyousafe?

Youneed10nodestorunyourapplication

Youhave11nodes,sothatyoucandorollingupgrades

§ Bringonenodedownatatimetoupgrade…

§ Alwaysatleast10available...

33

EXAMPLEWellno…


Withthefailedservertocontendwith…youhavenoroomtodoanupgradeorrollback,

andyouareatriskforanotherfailure

§ Whatifthatnodefailsduringupgrade?§ Whatifyounowhavetorollback?

34



Makesureyoucanhandlefailures

§ Evenduring“exceptional”events,suchasupgrades

§ Exceptionaleventscancausefailures


EXAMPLEUnknowndependencies

? ?

36

EXAMPLEUnknowndependencies


Areyousafe?

Youhaveyourapplicationrunningon20servers…§ Youcanrunon15serversifnecessary

§ Plentyofredundancy

37

EXAMPLEWell,depends…


Areanyofthe20servers in

thesamerack?

38




thesamerack?

Sharethesamepowersupply?

39




thesamerack?


Sharethesamepowersource?

40




thesamerack?


Sharethesamepowersource?

SharethesameA/Csystem?

41



Redundancyisnotredundancywhentheresourcesarenotindependent


EXAMPLEFailureloop

43

EXAMPLEFailureloop


Areyousafefrompoweroutages?

Youliveinanapartment…§ Theapartmentprovidesanenclosedgaragetostorethingsin

§ Thepowergoesoutinyourplacealot…§ ...youbuyagenerator,storeitinthegarage

44

EXAMPLEFailureloop


Oops

Oops…thegarage:§ Hasasingledoor,thebiggaragedoor§ Ithasagaragedooropener§ Thatrequireselectricitytoopen...§ Thegeneratorisonlyavailable...whenyoualreadyhavepower…

45



Makesureyourrecoveryplansactuallyareoperationalwhenyouareinafailuremode


EXAMPLEHighredundancyinaction

47

EXAMPLEArealsystem…


Highlyindependent

Multi-levelerror recovery

Highlyrecoverablesystem

Redundant

48

EXAMPLEArealsystem…


Infact,oneoftheveryfirstlargescalesoftwareapplicationsutilizingextremeredundancyandfailuremanagement

Highlyindependent

Multi-levelerror recovery

Highlyrecoverablesystem

Redundant

49

EXAMPLEWhatisthissystem?


50

EXAMPLEUSSpaceShuttleProgram


§ Theyhadproblems…seriousmechanicalproblems...

§ Butthesoftwaresystemutilizedstateoftheart:• Redundancytechniques• Errorrecoverytechniques

51

EXAMPLEUSSpaceShuttleSystem


Five onboardcomputers§ Fourwereidentical(fifthtalkaboutlater)

§ Allfour:– Rantheexactsameprogramduringcriticalperiods

– Givensamedata– Expectedtogeneratethesameresult

52

EXAMPLEFourcomputers


Computersvotedontheproperoutcome

Ifanyonecomputerdidnotgeneratethesameresults:

53




Thosethatdisagreedwiththeoutcomewereturnedoffforremainderoftheflight


54



Ultimateindemocraticsystems…


Thosethatdisagreedwiththeoutcomewereturnedoffforremainderoftheflight


55



CouldFLYwithonlyTHREE computersworking

CouldLANDwithonlyTWO computersworking

56

EXAMPLETies


Whatifthefourcomputerscouldn’tdecide?

(softwarebugormultiplefailures)

57

EXAMPLETies


Whatifthefourcomputerscouldn’tdecide?

(softwarebugormultiplefailures)

Fifthcomputerwasusedasatiebreaker

§ Muchsimplerversionofsoftware…onlyusedforkeydecisions

§ Softwarewrittenbyindependentsoftwareteam,unconnectedwithrestofsoftwaredevelopers

§ (Intheory)wouldnotintroducesamesoftwareerrors…

58

HighlySuccessful

©2008-16 New Relic, Inc. All rights reserved.

30-yearoperationofSpaceShuttle:§ Neveracasewhereaseriouslifethreateningproblemoccurredthatwasaresultofasoftwareproblem

§ Eventhoughsoftwarewasthemostcomplexsoftwareeverbuiltforaspaceprogram

59

USSpaceShuttle


Thisisextreme(notneededbymostprojects)§ Showswhatispossible...§ Independenceiscriticaltohighavailability

60



Useavailabilitysolutionconsistentwiththerisk

61




Highertherisk,higherthefocusonavailability

62





Don’toverinvest,don’tunder

invest

63





Don’toverinvest,don’tunder

invest

Butthinkahead,avoidthesurprise

64

Andremember…



ArchitectingforScaleBy:LeeAtchisonPublished by:O’ReillyMedia,Available:June2016www.architectingforscale.com

WanttoLearnMore?

©2008-15 New Relic, Inc. All rights reserved.

Thank you.

LeeAtchisonPrincipalCloudArchitectandAdvocateatNewRelic,Inc.

Architecting forScalePublished by:O’ReillyMedia,Available: June2016www.architectingforscale.com

@leeatchison leeatchison

cloud expo - flying two mistakes high

Technology