group replication: a journey to the group communication core · replication plugin api mysql server...

GroupReplication:AJourneytotheGroupCommunicationCore

Alfranio Correia(alfranio.correia@oracle.com)PrincipalSoftwareEngineer

4thofFebruary Oracle/Fosdem2017 1

SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirection.Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andtimingofanyfeaturesorfunctionalitydescribedforOracle’sproductsremainsatthesolediscretionofOracle.

24thofFebruary Oracle/Fosdem2017

ProgramAgenda

Background

GroupCommunicationInterface

Group Communication Engine

Performance

Conclusion

ProgramAgenda

4thofFebruary Oracle/Fosdem2017 4

Background

MySQLInnoDB Cluster

S1 S2 S3 S4 S…

MySQLConnectorApplication

MySQLRouter

MySQLShell

ReplicaSet

S1 S2 S3 S4 S…

MySQLRouter

ReplicaSet2

ReplicaSet3

MySQLRouter

S1 S2 S3 S4

MySQLGroupReplication• WhatisMySQLGroupReplication?“Multi-masterupdateeverywhere replicationpluginforMySQLwithbuilt-inautomaticdistributedrecovery,conflictdetection andgroupmembership.”

• WhatdoestheMySQLGroupReplicationplugindofortheuser?– Automates serverfailover inSinglePrimary– Providesfault tolerance– Enablesupdateeverywhere setups– Automatesgroupreconfiguration(handlingofcrashes,failures,re-connects)– Providesahighlyavailablereplicated database

MajorBuilding Blocks

M M M M MCom.API

ReplicationPlugin

MySQLServer

Group Comm.System (Corosync)GroupCom.Engine

TheCompleteStack

ReplicationPlugin

MySQLServer

PerformanceSchemaTables:Monitoring

APIs:Lifecycle/Capture/Applier

InnoDBReplicationProtocol

GroupCom.API

GroupCom.Engine

Network

PluginCapture ApplierConflicts

Handler

GroupComm.System(Corosync)GroupCom.Engine

GroupCom.Binding

Recovery

Group Communication Interface

Design• Abstract interfacetosupport different solutions– Reconfigurethe group and get membership information– Send and receive messages

• Usesthe observer pattern–MySQLGroupReplication listens toevents

• Different implementations perCommunication Systems• Made the transition from Corosync easy

Semantics• Closed Group–Only group members cansend and receive messages

• TotalOrder–Messages aretotally ordered among each other

• SafeDelivery–One cannot deliver amessage if the majority can’t doso

• View Synchrony– Changes tomembership aretolltaly ordered with messages

Group Communication Engine

Built-inCommunicationEngine• Based on provendistributedsystemsalgorithms(Paxos)– Compression,multi-platform,dynamicmembership,SSL,IPwhitelisting

• Nothird-partysoftwarerequired• Nonetworkmulticastsupport required–MySQLGroupReplicationcanoperateoncloudbasedinstallationswheremulticastisunsupported

PaxosFamily and Friends

Multi-Paxos

Fast Paxos

Disk Paxos

Cheap Paxos

VerticalPaxos

Generalized Paxos

Mencius

Flexible Paxos

Egalitarian Paxos

Byzantine Paxos

BasicPaxos

Prepare/Election Phase

Accept Phase

Learn Phase

• Get agreement on avalue:– Next message/transaction tobedelivered

• Members may have different roles:– Usually all members areproposers,acceptors and learners

• Need aquorum tomake progress– Usually amajority

PreparePhase

• Proposer sends apreparerequest with number “n”tomembers (i.e.acceptors)• If an acceptor has not received arequest with anumber greater than “n”,it will respond• It will promise not toaccept arequest numberedless than “n”• If the reply has anon-empty value,the leaderwillusethat with the highest number

Prepare1.1

Promise1.2

(y,value)

(x,value)

Accept Phase

• If the leaderfinds outthat anon-empty value hasbeen previously proposed,it will useit• Otherwise,it will propose anew value• Requires anetworkround-triptoget agreement

Accept2.1

Accepted2.2

(n,value)

Learn Phase

• It will inform other members about the decision• Only one learner is required tohave progress• If the member already has the value,an ack isenough

Learn3

(value)

Multi-Paxos

slot 0

Accept/Learn0

Election0

Accept/Learn0

Election

slot 1 slot 2 slot 3 ...

• Consensus roundtodecideon each slot’s content

• Replicated LogStream

So what?• They caneasily become abottleneck• Multiple leaders:eXtended COMmunications

How doesXCOMwork?

slot 0

Accept/Learn

slot 1 slot 2 slot 3

Accept/Learn

slot 4 slot 5 ......

Accept/Learn0

Accept/Learn

• Every member is aleaderso noleaderelection

• Every member owns aIn-Memory Replicated Log

Nothing toPropose

slot 0

Accept/Learn

nop slot 2 slot 3

Accept/Learn

nop slot 5 ......

Accept/Learn0

Learn0

Accept/Learn0

• Only alearn message with a“nop”is enough

How is the optimization possible?• Member “1”sends alearn message “(0,nop)”tomember “4”and dies• Non-leaderscanonly propose “nop”(s)on behalf of others• They mustgo through all Paxosphases

(1,nop)

Prepare Promise Accept Accepted

(0,nop)

HandlingFailures/Suspicions

slot 0

Accept/Learn0

Prep./Accept/Learn

slot 1 slot 2 nop

Accept/Learn0

Accept/Learn

slot 4

Accept/Learn

slot 5 ......

Implemented Optimizations inXCOM• Pipeline– Proposes several “transactions”inparallel– Improvesperformanceinhigh latency networks– Current value is “10”

• Batch– ImprovesCPUusage– Improvesperformanceinhigh latency/low bandwidth networks– Current value is “5”

Implemented Optimizations inBiding• Compression– Reduces bandwith consumption

• Automatically reconfigureagroup– Faulty members areexpelled

Performance

Configuration• Multipe writers – One perServer• Singlewriter – Just one client• OracleServerX5-2Lwith two IntelXeon E5-2660-V3processors– 20Cores– 40HardwareThreads

• OracleEnterprise Linux7,kernel 3.8.13-118.13.3• 10Gbps ethernet• Used “tc”tothrottle network

Multiple writers (256Bytes)

3members 5members 7members 3members 5members 7members

Uncompressed256bytepayload Compressed256bytepayload

100000

120000

140000

16000010Gbpsnetworkwith0.1mslatency

200Mbpsnetworkwith7mslatency

• Compression improvesperformanceinMetropolitan

• Headers arenot compressed (~200bytes)though

Messagesp

Multiple writers (1KBytes)

• Check whether compression may help or not

• Usually helps when bandwidth is aproblem

Uncompressed1Kpayload Compressed1Kpayload

100000

Messagesp

SingleWriter (1KBytes)

Uncompressed1Kpayload Compressed1Kpayload

100000

• The scale outeffect with multiple writers is small

• Compression doesnot help here

Messagesp

Conclusion

Current Status• Has made into MySQL 5.7.17release• GAinDecember 2016

Future• Configurable Paxosrole(s)– Leader/Acceptor/Learner or Acceptor/Learner or Learner

• Multiple leadersonly if needed:– Avoids the skip message– ImprovesCPUand networkusage

• Not all members need tomake messages networkdurable– Reduces resilience but improvesperformance

Future• Expose someconfiguration options:– Batch– Pipeline

• Compression at low level layers aswell• Write tonetworkinparallel• Overlay networks

Wheretogofromhere?• Packages– http://www.mysql.com/downloads/

• Documentation– http://dev.mysql.com/doc/refman/5.7/en/group-replication.html

• BlogsfromtheEngineers(news,technicalinformation,andmuchmore)– http://mysqlhighavailability.com

group replication: a journey to the group communication core · replication plugin api mysql server...

Documents

upstream downstream - fosdem · ha audit mysql 5.6 mysql...

understanding mysql group...

mysql replication: latest developments · mysql 5.1.30...

14603869 mysql replication tutorial

mysql developer day conference: mysql replication and...

mysql 5.7 replication news

mysql replication: what's new in mysql 5.7 and mysql 8 ·...

mysql replication: what’s new in mysql 5.7 and beyond

mysql replication tutorial presentation (pdf) - o'reilly...

mysql replication @ gnugroup

mysql replication update mysql 5.5 (ga) & mysql 5.6.2 (dev...

managing mysql replication - jeremy d....

mysql replication for beginners

mysql enterprise monitor€¦ · mysql-monitor.log mysql...

mysql applier for apache hadoop: real-time event streaming...

advanced mysql replication techniques

mysql replication: what's new in mysql 5.7 and mysql 8 ›...

mysql replication...

high availability using mysql group replication ·...

section viii: mysql data replication · 2006. 1. 12. ·...