the chubby lock service for loosely‐coupled distributed ... · the chubby lock service for...
TRANSCRIPT
![Page 1: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/1.jpg)
TheChubbylockservicefor
loosely‐coupleddistributedsystems
PresentedbyPetkoNikolovCornellUniversity
3/12/09
![Page 2: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/2.jpg)
BeforeChubbyCameAbout…
• Widerangeofdistributedsystems• Clientsinthe10,000s• HowtodoprimaryelecJon?– Adhoc(noharmfromduplicatedwork)– OperatorintervenJon(correctnessessenJal)
• Disorganized• Costly• Lowavailability
![Page 3: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/3.jpg)
MoJvaJon
• Needforservicethatprovides– synchronizaJon(leaderelecJon,sharedenv.info.)– reliability– availability– easy‐to‐understandsemanJcs
– performance,throughput,latencyonlysecondary
• NOTresearch
![Page 4: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/4.jpg)
Outline
• PrimaryElecJon– Paxos
• Design• UseandObservaJons• RelatedWork
![Page 5: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/5.jpg)
PrimaryElecJon
• Distributedconsensusproblem• AsynchronouscommunicaJon– loss,delay,reordering
• FLPimpossibilityresult
• SoluJon:Paxosprotocol
![Page 6: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/6.jpg)
Paxos:Problem
• CollecJonofprocessesproposingvalues– onlyproposedvaluemaybechosen
– onlysinglevaluechosen– learnofchosenvalueonlywhenithasbeen
• Proposers,acceptors,learners• Asynchronous,non‐ByzanJnemodel– arbitraryspeeds,failbystopping,restart– messagesnotcorrupted
![Page 7: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/7.jpg)
Paxos:Algorithm
Phase1(a)Proposersendspreparerequestwith#n(b)Acceptor:ifn>#ofanyotherprepareit
has repliedto,respondwithpromise.Phase2(a) Ifmajorityreply,proposersendsaccept
with valuev(b) Acceptoracceptsunlessitrespondedto
preparewith#higherthann
![Page 8: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/8.jpg)
Paxos:Algorithm
![Page 9: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/9.jpg)
Paxos:Algorithm
• Learningofchosenvalue– disJnguishedlearneropJmizaJon
• haspi_alls
• Makingprogress– disJnguishedproposer
• Usually,everyprocessplaysallroles– primaryasdisJnguishedproposerandlearner
![Page 10: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/10.jpg)
Paxos:StateMachines
• Replicatedstatemachine– samestateifsamesequenceofops.performed
• Clientsendsrequeststoserver– replicatedwithPaxos
• Paxosusedtoagreeonorderofclientops.– canhavefailures/morethan1master
– Paxosguaranteesonly1valuechosen&replicated
![Page 11: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/11.jpg)
Paxos:ViewChange
![Page 12: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/12.jpg)
Design
• Lockservice(andnotconsensuslibrary)• Servesmallfiles
• Supportlarge‐scaleconcurrentfileviewing• EventnoJficaJonmechanism
• Cachingoffiles(consistentcaching)• Security(accesscontrol)• Course‐grainedlocks
![Page 13: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/13.jpg)
Design:RaJonale
• Lockservicevs.Paxoslibrary• Advantages– maintainprogramstructure,comm.pacers
– mechanismforadverJsingresults– persuadingprogrammerstouseit– reduce#ofclientserversneededtomakeprogress
![Page 14: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/14.jpg)
Design:RaJonale
• Course‐grainedlocks– lessloadonlockserver– lessdelaywhenlockserverfails– shouldsurvivelockserverfailures– lesslockserversandavailabilityrequired
• Fine‐grainedlocks– heavierlockserverload,moreclientstallingonfail– canbeimplementedonclientside
![Page 15: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/15.jpg)
Design:SystemStructure
• Twomaincomponents:– server(Chubbycell)– clientlibrary– communicateviaRPC
• Proxy– opJonal– moreonthislater
![Page 16: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/16.jpg)
Design:ChubbyCell
• Setofreplicas(typically5)• UsePaxostoelectmaster– promisenottoelectnewmasterforsomeJme(masterlease)
• Maintaincopiesofsimpledatabase
• WritessaJsfiedbymajorityquorum
• ReadssaJsfiedbymasteralone
• Replacementsystemforfailedreplicas
![Page 17: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/17.jpg)
Design:ChubbyClients
• Linkagainstlibrary• MasterlocaJonrequeststoreplicas
• Allrequestssentdirectlytomaster
![Page 18: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/18.jpg)
Design:Files,Dirs,Handles
• FSinterface– /ls/cs6464‐cell/lab2/test– specializedAPI– alsoviainterfaceusedbyGFS
• Doesnotsupport/maintain/reveal– movingfiles
– path‐dependentpermissionsemanJcs– dirmodifiedJmes/filelast‐accessJmes
![Page 19: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/19.jpg)
Design:Nodes
• permanentvs.ephemeral• Metadata– threenamesofACLs(R/W/changeACLname)
• authenJcaJonbuiltintoRPC– 4monotonicallyincreasing64‐bitnumbers
• instance,contentgeneraJon,lockgeneraJon,ACLgen.– 64‐bitfile‐contentchecksum
![Page 20: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/20.jpg)
Design:Handles
• AnalogoustoUNIXfiledescriptors• Checkdigits– preventclientcreaJng/guessinghandles
• Supportforuseacrossmasterchanges– sequencenumber– modeinformaJonforrecreaJngstate
![Page 21: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/21.jpg)
Design:LocksandSequencers
• Anynodecanactaslock(sharedorexclusive)• Advisory(vs.mandatory)– protectresourcesatremoteservices
– debugging/admin.purposes– novalueinextraguardsbymandatorylocks
• Writepermissionneededtoacquire– preventsunprivilegedreaderblockingprogress
![Page 22: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/22.jpg)
Design:LocksandSequencers
• Complexinasyncenvironment• Usesequence#’sininteracJonsusinglocks• Sequencer– opaquebyte‐string– stateoflockimmediatelyajeracquisiJon– passedbyclienttoservers,serversvalidate
• AlternaJve:lock‐delay
![Page 23: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/23.jpg)
Design:Events
• ClientsubscribeswhencreaJnghandle• Deliveredasyncviaup‐callfromclientlibrary• Eventtypes– filecontentsmodified– childnodeadded/removed/modified– Chubbymasterfailedover– handle/lockhavebecomeinvalid– lockacquired/conflicJnglockrequest(rarelyused)
![Page 24: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/24.jpg)
Design:API
• Open()(onlycallusingnamednode)– howhandlewillbeused(accesscheckshere)– eventstosubscribeto– lock‐delay– whethernewfile/dirshouldbecreated
• Close()vs.Poison()• Otherops:
– GetContentsAndStat(),SetContents(),Delete(),Acquire(),TryAcquire(),Release(),GetSequencer(),SetSequencer(),CheckSequencer()
![Page 25: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/25.jpg)
Design:API
• PrimaryelecJonexample• Candidatesacempttoopenlockfile/getlock– winnerwritesidenJtywithSetContents()– replicasfindoutwithGetContentsAndStat(),possiblyajerfile‐modificaJonevent
• Primaryobtainssequencer(GetSequencer())
![Page 26: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/26.jpg)
Design:SessionsandKeepAlives
• SessionmaintainedthroughKeepAlives• Handles,locks,cacheddataremainvalid– clientmustacknowledgeinvalidaJonmessages
• Terminatedexplicitly,orajerleaseJmeout
• LeaseJmeoutadvancedwhen– sessioncreated– masterfail‐overoccurs
– masterrespondstoKeepAliveRPC
![Page 27: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/27.jpg)
Design:SessionsandKeepAlives
• MasterrespondsclosetoleaseJmeout• ClientsendsanotherKeepAliveimmediately
![Page 28: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/28.jpg)
Design:SessionsandKeepAlives
• Handles,locks,cacheddataremainvalid– clientmustacknowledgeinvalidaJonmessages
• CacheinvalidaJonspiggybackedonKeepAlive– clientmustinvalidatetomaintainsession– RPC’sflowfromclienttomaster– allowsoperaJonthroughfirewalls
![Page 29: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/29.jpg)
Design:SessionsandKeepAlives
• ClientmaintainslocalleaseJmeout– conservaJveapproximaJon
– mustassumeknownrestricJonsonclockskew
• Whenlocalleaseexpires– disablecache– sessioninjeopardy,clientwaitsingraceperiod– cacheenabledonreconnect
• ApplicaJoninformedaboutsessionchanges
![Page 30: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/30.jpg)
Design:Caching
• Clientcachesfiledata,nodemeta‐data– write‐throughheldinmemory
• InvalidaJon– masterkeepslistofwhatclientsmayhavecached– writesblock,mastersendsinvalidaJons– clientsflushchangeddata,ack.withKeepAlive– datauncachableunJlinvalidaJonacked
• allowsreadstohappenwithoutdelay
![Page 31: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/31.jpg)
Design:Caching
• Invalidatesdatabutdoesnotupdate– updaJngarbitrarilyunefficient
• Strictvs.weakconsistency– weakermodelshardertouseforprogrammers– donotwanttoalterpreexisJngcomm.protocols
• Handlesandlockscachedaswell– eventinformsclientofconflicJnglockrequest
• Absenceoffilescached
![Page 32: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/32.jpg)
Design:Fail‐overs
• In‐memorystatediscarded– sessions,handles,locks,etc.
• LeaseJmer“stops”
• Quickre‐elecJon– clientreconnectbeforeleasesexpire
• Slowre‐elecJon– clientsdisablecache,entergraceperiod– allowssessionsacrossfail‐overs
![Page 33: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/33.jpg)
Design:Fail‐overs
![Page 34: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/34.jpg)
Design:Fail‐overs
Stepsofnewly‐electedmaster:1. Picknewepochnumber
2. RespondonlytomasterlocaJonrequests3. Buildin‐memorystateforsessions/locksfromDB4. RespondtoKeepAlives5. Emitfail‐overeventstocaches6. Waitforacknowledgements/sessionexpire7. AllowalloperaJonstoproceed
![Page 35: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/35.jpg)
Design:Fail‐overs
Stepsofnewly‐electedmaster(cont’d):8. Handlecreatedpre‐fail‐overused– masterrecreatesinmemory,honorscall
– ifclosed,recordthatinmemory
10. Deleteephemeralfilesw/oopenhandlesajeraninterval
• Fail‐overcodesourceofmanybugs
![Page 36: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/36.jpg)
Design:Database
• FirstChubbyusedreplicatedBerkeleyDB– withmasterleaseaddedon
• ReplicaJoncodewasnew– didnotwanttotaketherisk
• Implementedownsimpledatabase– distributedusingconsensusprotocol
![Page 37: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/37.jpg)
Design:Backup
• Everyfewhours• SnapshotofdatabasetoGFSserver– differentbuilding
• buildingdamage,cyclicdependecies
• Disasterrecovery• IniJalizenewreplica– avoidloadonin‐servicereplicas
![Page 38: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/38.jpg)
Design:Mirroring
• CollecJonoffilesmirroredacrosscells• MostlyforconfiguraJonfiles– /ls/global/mastermirroredto/ls/cell/slave
• globalcell’sreplicasspreadaroundworld– Chubby’sownACLs– FilesadverJsingpresence/locaJon– pointerstoBigtablecells– etc.
![Page 39: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/39.jpg)
MechanismsforScaling
• Clientsindividualprocesses(notmachines)– observed90,000clientsforasinglemaster
• ServermachinesidenJcaltoclientones• MosteffecJvescaling:reducecommunicaJon• Regulate#ofChubbycells• IncreaseleaseJme• Caching• Protocol‐conversionservers
![Page 40: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/40.jpg)
Scaling:Proxies
• Proxiespassrequestsfromclientstocell• CanhandleKeepAlivesandreads• Notwrites,buttheyare<<1%ofworkload• KeepAlivetrafficbyfarmostdominant
• Disadvantages:– addiJonalRPCforwrites/firstJmereads– increasedunavailabilityprobability– fail‐overstrategynotideal(willcomebacktothis)
![Page 41: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/41.jpg)
Scaling:ParJJoning
• NamespaceparJJonedbetweenservers• NparJJons,eachwithmasterandreplicas
• NodeD/CstoredonP(D/C)=hash(D)modN– meta‐dataforDmaybeondifferentparJJon
• Liclecross‐parJJoncomm.desirable– permissionchecks– directorydeleJon– cachinghelpsmiJgatethis
![Page 42: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/42.jpg)
UseandObservaJons
• Manyfilesfornaming
• Config,ACL,meta‐datacommon
• 10clientsuseeachcachedfile,onavg.
• Fewlocksheld,nosharedlocks
• KeepAlivesdominateRPCtraffic
![Page 43: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/43.jpg)
Use:Outages
• Sampleofcells– 61outagesoverfewweeks(700cell‐days)– duetonetworkcongesJon,maintenance,overload,errorsinsojware,hardware,operators
• 52outagesunder30s – applicaJonsnotsignificantlyaffected
• Fewdozencell‐yearsofoperaJon– dataloston6occasions(bugs&operatorerror)
![Page 44: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/44.jpg)
Use:JavaClients
• MostofGoogleinfrastructureisinC++• Growing#ofJavaapplicaJons• GooglersdislikeJNI– wouldrathertranslatelibrarytoJava– maintainingitwouldrequiregreatexpense
• Javausersrunprotocol‐conversionserver– exportsprotocolsimilartoChubby’sclientAPI
![Page 45: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/45.jpg)
Use:NameService
• MostpopularuseofChubby– providesnameserviceformostGooglesystems
• DNSusesTTLvalues– entriesmustberefreshedwithinthatJme– huge(andvariable)loadonDNSserver
• Chubby’scachingusesinvalidaJons,nopolling– clientbuildsupneededentriesincache– nameentriesfurthergroupedinbatches
![Page 46: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/46.jpg)
Use:NameService
• Nameservice– nofullconsistencyneeded– reduceloadwithprotocol‐conversionserver
• ChubbyDNSserver– namingdataavailabletoDNSclients– easestransiJonbetweennames
– accommodatesbrowsers
![Page 47: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/47.jpg)
Use:Fail‐overProblems
• MasterwritessessionstoDBwhencreated– startofmanyprocessesatonce=overload
• DBmodified–storesessionatfirstwriteop.– read‐onlysessions:atrandomonKeepAlives– spreadoutwritestoDBinJme
• Youngread‐onlysessionsmaybe“discarded”– mayreadstaledataforawhileajerfail‐over– verylowprobability
![Page 48: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/48.jpg)
Use:Fail‐overProblems
• Newdesign–nosessionsindatabase– recreatethemlikehandlesajerfail‐over
– newmasterwaitsfullleaseJmebeforeops.• licleeffect–verylowprobability
• Proxyserverscanmanagesessions– allowedtochangesessionalockisassociatedwith
• permitstakeoverofsessionbyanotherproxyonfail
– mastergivesnewproxychancetoclaimlocksbeforerelinquishingthem
![Page 49: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/49.jpg)
Use:AbusiveClients
• Companyenvironmentassumed• RequeststouseChubbythoroughlyreviewed• Abuses:– lackofaggressivecaching
• absenceoffiles,openfilehandles– lackofquotas
• 256kBlimitonfilesizeintroduced• encourageduseofappropriatestoragesystems
– publish/subscribe
![Page 50: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/50.jpg)
Use:LessonsLearned
• Developersrarelyconsideravailability– shouldplanforshortChubbyoutages– crashedapplicaJonsonfail‐overevent
• Fine‐grainedlockingnotessenJal• PoorAPIchoices– handlesacquiringlockscannotbeshared
• RPCuseaffectstransportprotocols– forcedtosendKeepAlivesbyUDPforJmeliness
![Page 51: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/51.jpg)
RelatedWork
Chubby• locks,storagesystem,
session/leaseinoneservice• targetaudience–wide
range• higher‐levelinterface• lostlockexpensivefor
clients• coulduselocksand
sequencerswithothersystems
Boxwood
• 3separateservices– lock,Paxos,failuredetecJon– couldbeusedindependently
• fewer,moresophisJcateddevelopers
• differentdefaultparameters• lacksgraceperiod• useslocksprimarilywithin
![Page 52: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/52.jpg)
Summary
• Distributedlockservice– course‐grainedsynchronizaJonforGoogle’sdistributedsystems
• Designbasedonwell‐knownideas– distributedconsensus,caching,noJficaJons,file‐systeminterface
• Primaryinternalnameservice
• Repositoryforfilesrequiringhighavailability
![Page 53: The Chubby lock service for loosely‐coupled distributed ... · The Chubby lock service for loosely‐coupled distributed systems Presented by Petko Nikolov Cornell University 3/12/09](https://reader034.vdocuments.us/reader034/viewer/2022052007/601bd2744865f81a70151398/html5/thumbnails/53.jpg)
References
TheChubbylockserviceforloosely‐coupleddistributedsystems,MikeBurrows.AppearsinProceedingsofthe7thUSENIXSymposiumonOperaJngSystemsDesignandImplementaJon(OSDI),November,2006.hcp://labs.google.com/papers/chubby‐osdi06.pdf
PaxosMadeSimple,LeslieLamport.AppearsinACMSIGACTNews(DistributedCompuJngColumn),Vol.32,No.4(December2001),pages51‐58.hcp://research.microsoj.com/en‐us/um/people/lamport/pubs/paxos‐simple.pdfAlso,
PaxosMadePracJcalbyDavidMaziereshcp://www.cs.cornell.edu/courses/cs6464/2009sp/papers/paxos_pracJcal.pdf