con2178 clement-oow 15 cluster talk2

8/17/2019 CON2178 Clement-OOW 15 Cluster Talk2

1/44

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |


2/44



OpenWorld 2015

200 Million QPS on Commodity Hardware

Getting Started with MySQL Cluster 7.4

Fraer Cle!ent"y#$% Cl&ster 'echnical %ead

(ernd Oc)lin*irector, "y#$% Cl&ster +ngineering

Octoer 2-th, 2015



3/44


200 "illion $# on Co!!odity ardareetting started ith "y#$% Cl&ster .3

4sers, Feat&res and eleases1

2

3

4

*esign for Availaility and #cale

erfor!ance, getting to 200" 7&eries/s

o to get started ith "y#$% Cl&ster


4/44


#afe aror #tate!ent 'he folloing is intended to o&tline o&r general prod&ct direintended for infor!ation p&rposes only, and !ay not e incointo any contract. 8t is not a co!!it!ent to deliver any !ateor f&nctionality, and sho&ld not e relied &pon in !a)ing p&rdecisions. 'he develop!ent, release, and ti!ing of any feat

f&nctionality descried for Oracle9s prod&cts re!ains at the discretion of Oracle.


5/44Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

:eynote; "onday, 3.00 "y#$% +ngineering, Oracle

C&sto!er +@periences

Hari Tatrakal, *irector of *ataase #ervices, %ive ation

Olaniyi Osinowo, "y#$% ? Open #o&rce 'echnologies %ea!rnie So"rada ? #o$ W"lts%, *ataase +ngineers, int



"y#$% Cl&ster content B Openorld

&"lly !lasti% #eal'

Time Ser(i%es witMySQ) Cl"ster

(ernd Oc)lin, Oracle

Conference #ession

'&esday 11a!."oscone #o&th, 2-2

MySQ) Ser(er and

MySQ) Cl"ster at*ndia+s &inan%ial*n%l"sion ,atewaySer(i%e

+C et al

Conference #ession

'&esday 5.15p!"oscone #o&th, 250

,et Start

MySQ) Cl

(enedita >

Oracleands On

'h&rsday, otel i))



"y#$% Co!!&nity eception B Open

Cele$rate- Ha(e &"n and Min.le wit Ora%le+s MySQ)/ o"r Peers

• '&esday, Octoer 2th, p!

• Eillian9s at "etreon; 15 Fo&rth #treet, #an Francisco C

At the corner of oard and 3th st. only 2


8/44


"y#$% Cl&ster deploy!ents

Web

Telecoms

igh vol&!e O%'eCo!!erce

On


9/44


hoLs &sing "y#$% Cl&sterM


10/44


"y#$% Cl&ster highlights

HighThroughputReads & Writes

Carrier-GradeAvailability

Real-TimeResponsiveness

On-ine! inear"calability

o# TCO!Open plat$orm

%istributed! arallel architecture

Transactional! AC'%-compliant relational database

"hared-nothing design! synchronous data replication

"ub-second $ailover & sel$-healing recovery

%ata structures optimi(ed $or RA)* Real-time e+tensio

redictable lo# latency! bounded access times

'ncrementally scale out! scale up and scale on-lineinearly scale #ith distribution a#areness

G & Commercial editions! scale on COT"

,le+ible A's ".! C//! 0ava! Open0A! %A & HT


11/44



SQL

Joins, Foreign Keys, Transactions! Ro# loc1s! Tri

"tored procedures! 3lobs! 1eyless tables! ne#".! )y". compatible*** conneclanguages! OR)s etc***

oSQL ,ull C!! "#i $or best control and per$ormance 4)y".% "5 built onJa$a ! 0A! ode.%s! )emcache****

&" 77*7778 uptime systems '(i$e nines), o single #oint o( (ai

Heartbeating! cluster membership! auto*atic (ailo$er ! reco$ery! aut$ailover! transactional %%! C! async replication! advanced e+ception logging***

+er(or*ance and #arallelis* &igh through#ut, low ound4-M read t/0s6* 3atching! optimised protocols! 'ntra and 'nter 9uery paralleli

parallel $ilters! #ushed #arallel %oins! non1loc2ing e$ent dri$en *u

HA! High per$ormance! Relational! Transactional! %istributed! arallel! ".! :o".! "hared-:othing! Commodity ***


12/44



Scalaility Scale1out nodegroups or stateless A' clients online! Scale1u#clients online #ith *ultithreading! scale up hard#are online

3e#lication Synchronous two #hase co**it internally! Transacasync re#lication bet#een clusters! con$lict detection/resolution***Storage %ata trans#arently distriuted and alanced y hash, 'nde+

memory! others on dis1 or memory! "econdary uni9ue and ordered inde+es! Redundant 3

and #eriodic chec2#oints...Manageaility nline add ! dro# 'inde/, colu*n)! Online consistent bac1u#grade! Online O" or hard#are upgrade! consolidated cluster logs! C management AShared nothing, Co**odity o need (or shared storage! 'n-memory data u

TC over 5thernet ; 'n$iniband etc! :o special layer < re9uirements* #en source

HA! High per$ormance! Relational! Transactional! %istributed! arallel! ".! :o".! "hared-:othing! Commodity ***


13/44


=*>

"y#$% Cl&ster eleases

=*< =*?

< *istri&tedparallel Noins

< "&lti


14/44


"y#$% Cl&ster .3

Active


15/44


"y#$% Cl&ster Architect&re

5ata

odes

ode Grou# 6

,@

,?

,?

,@

ode Grou# -

,<

,

,

,<

"##lication odes

Cluster

Mg*t

3STJ+"

ode Grou# 8

,>

,B

,B

,>

,@

,<

,>

,?

,

,B

Table @

d"#i #rotocol

Tables and 'ndices are hori(ontally partitioned! distributed across anreplicated #ithin the :odeGroups* Application :odes including )y".%

:dbApi to per$orm transactional operations and 9ueries on data*

)ost Application :o"ervers $or variou


16/44


"y#$% Cl&ster Architect&re for Availaili

5ata

odes

ode Grou# 6

,@

,?

,?

,@

ode Grou# -

,<

,

,

,<

"##lication odes

Cluster

Mg*t

3STJ+"

ode Grou# 8

,>

,B

,B

,>

,@

,<

,>

,?

,

,B

Table @

Redundancy $or availability - All nodes in each nodegroup store the same data - Can survive data node $ailures so long as one node per nodegroup is

available* - oad balanced! "ynchronous


17/44



5ata

odes

ode Grou# 6

,@

,?

,?

,@

ode Grou# -

,<

,

,

,<

"##lication odes

Cluster

Mg*t

3STJ+"

ode Grou# 8

,>

,B

,B

,>

,@

,<

,>

,?

,

,B

Table @

Redundancy $or availability - T#o 4or more6 management servers* - sed $or con$iguration! node startup;shutdo#n! triggering

bac1ups! logging / Dsplit-brainD arbitration - :ot critical E not involved in transaction processing ;9uerying

Redundant

components

)anagement nodes act as light#eight arbitrators! avoiding the cost o$odd-si(ed data node 9uorums to cope #ith single $ailures*

d"#i #rotocol


18/44



5ata

odes

ode Grou# 6

,@

,?

,?

,@

ode Grou# -

,<

,

,

,<

"##lication odes

Cluster

Mg*t

3STJ+"

ode Grou# 8

,>

,B

,B

,>

,@

,<

,>

,?

,

,B

Table @

Redundancy $or availability - A' nodes are stateless and consistent! can use n / m sparing #ithsimple $ront end load balancing* - :dbApi automatically balances! $ails over and bac1 on data node $ailures*

- :et#or1 needs no "O, too E no single $ailure ta1es out F @ clustermember*

Redundant

components

d"#i #rotocol

Availability also comes $rom support $or online operations "chemachanges! Hard#are and O" upgrades! "o$t#are upgrades! Cluster scal


19/44


"y#$% Cl&ster Architect&re for #cale O"

5ata

odes

ode Grou# 6

,@

,?

,?

,@

ode Grou# -

,<

,

,

,<

"##lication odes

Cluster

Mg*t

3STJ+"

ode Grou# 8

,>

,B

,B

,>

,@

,<

,>

,?

,

,B

Table @

+er(or*ance ! Ca#acity

Online scale out o$ bac1 end by adding #hole node groups

4Read / Write scaling6

%ata :odes can be added online! #hile transactions and 9ueriesare running* 5+isting data is rebalanced across all nodegroups*

d"#i #rotocol


20/44


"y#$% Cl&ster Architect&re for #cale O"

5ata

odes

ode Grou# 6

,@

,?

,?

,@

ode Grou# -

,<

,

,

,<

"##lication odes

Cluster

Mg*t

3STJ+"

ode Grou# 8

,>

,B

,B

,>

,@

,<

,>

,?

,

,B

Table @

+er(or*ance ! Ca#acity

Online scale out o$ bac1 end by adding #hole node groups

4Read / Write scaling6

Application :odes can be added and removed online! all havee9ual! consistent access to the data stored by the data nodes*

+er(or*ance ! &"

Online scale out o$ $ront end ; Api nodes

d"#i #rotocol


21/44


"y#$% Cl&ster Architect&re for #cale p

Replication thread

)ain thread

%) instances"hared nothing

TC instances"hared nothing

"end threads

Re9uesth

TC and most #or$ed by "

t

Receive threads 'O threads

Connect threads Watch

ndTC 'ransaction

coordinator)M%ocal data!anager G'ale K8nde@ partitionsH

Generally no more than one re9uest processing thread per HT

%ata node


22/44


"y#$% Cl&ster Architect&re for #cale p

Replication thread

)ain thread

%) instances"hared nothing

TC instances"hared nothing

"end threads

Re9uesth

TC and most #or$ed by "

t

Receive threads 'O threads

Connect threads Watch

nd

%ata node

Con$igurable parallelism #ithin a %ata node

TC 'ransaction

coordinator)M%ocal data!anager G'ale K8nde@ partitionsH


23/44


"y#$% Cl&ster Architect&re for #cale pApplicationnode

%atabase ; ersistence layer

3usiness logic ; "tate machines

rotocol decoding

)anyI threads

:dbApi

A' conn

Clients Clients Clients

libndbclie

:dbApi calls

Client memc

Drotocol BD

A' conn A' conn

Mysqld

MemcachNode.js

Java

Slapd

...

< Can scale then&!er of threadsto !eet de!and

< Can scale then&!er of dApiconnections toavoid ottlenec)s


24/44


"y#$% Cl&ster Architect&re for #cale pApplicationnode

%atabase ; ersistence layer

3usiness logic ; "tate machines

rotocol decoding

)anyI threads

:dbApi

A' conn

Clients Clients Clients

libndbclie

:dbApi calls

Client memc

Drotocol BD

A' conn A' conn

Mysqld

MemcachNode.js

Java

Slapd

...

< Can scale then&!er of threadsto !eet de!and

< Can scale then&!er of dApiconnections toavoid ottlenec)s

" #$% Cl f


25/44


"y#$% Cl&ster erfor!ance

istri$"ted

ei%ien%y

rotocol design,opti!isation, pac)ing,

!&litple@ing.*ata *istri&tion

aareness%ocality < &shed don

filtering and Noining

Coordination

a(oidan%e

on loc)ing readsarallel co!!it

alan%e

ash partitioning

)o%alei%ien%y

O# call a!ortisaon loc)ing e@e

Cache friendly dstr&ct&res

%oc) free sharedstr&ct&res

%ocal data str&ct"&lti gran&larity

#cale O"t #cale p

"ee )y". Connect


26/44



". Koins!

aggregates

Opti!isations &ild in layers

:o". R;W o$ single ro#s

:o". R;W o$ multi ro#s

". R;W o$ multi ro#s

)y". "erver ". optimisations

%istributed parallel $ilter / Koin

3atching hints! distribution

a#areness! read removal

Optimised


27/44



=*< =*> =*?

&e$ 20121 illion

o#$% readsper !in&te

"n 2013 J.5@ etter

perfor!anceper dApiconnection

&e$ 2015 200 !illion

o#$% readsper second

50 etter#ysench readperfor!ance

"l 20121 illion ritesper !in&te

2.5 !

#$%state!per se


28/44


=*>


=*< =*?

ata node "&ltiple

'ransactionCoordinatorG'CH threads

6d$7pi Connectionthreadcontentionred&ction

ata node #can K :loo)&popti!isations,#end K ecv

opti!isations

eg&lar i!prove!ents co!po&nd over releases

" #$% Cl t f


29/44



6oSQ) "lk $en%marks < etting to !illions of re7&ests per second on a

distri&ted syste! is often a !atter of efficient!&ltiple@ing and de!&ltiple@ing of individ&alre7&ests < "odern hardare is very capale and so it isi!portant to )eep o&t of the ay, avoidingconte@t sitches, threads, loc) contention, s!all

!essages, e@tra hops, and &nnecessaryco!!&nication or coordination. < "any s!all re7&ests !&st e gatheredtogether and handled in &l), itho&t adverselyaffecting latency or application se!antics.



30/44



< *elivered as part of so&rce distri&tion

< "&ltithreaded CKK dApi application < 4ses the asyn%rono&s feat&res of dApi hich allo athread to participate in !&ltiple conc&rrent dataase transa < o operations &sing the f&ll pri!ary )ey < Can !a)e &se of dApi *istri&tion Aareness hints to !co!!&nication

< ara!eters ; Number of API connections, Number of Number of parallel transactions per thread, Number

per transaction, Number of columns, Size of each c

Lockmode, Distribution Awareness, Thread partition

o#$% ench!ar) tool flexAsynchnli1e e*g* )y".% ;)emcached! has no

upstream clients to servso simpler

%etails http;;mi1aelronstrom*blogspot*co*u1;;@@;ho#-to-ma1e-e$$icient-s

"y#$% Cl&ster 200 !illion o#$% reads/


31/44



2 A8 client !achines r&nning flexAsynch

62 *ata node!achines

r&nning ndbmtd

1 "anage!ent node

- @JJ bytes da

- @7 G3;s agg

rate - B*? ) reads;

- B@< )3;s da

rate

-


32/44



2 A8 client !achines r&nning flexAsynch

62 *ata node!achines

r&nning ndbmtd

1 "anage!ent node

- @JJ bytes da

- @7 G3;s agg

rate - B*? ) reads;

- B@< )3;s da

rate

-


33/44



'he8nfiniand

Clo&d '"

10 !illionconc. reads

2 @ 25-threads

2 @ 6 A8connections

flexAsynch ndbmtd

62 @ 12 'C K%*" thread



34/44



'he8nfiniand

Clo&d '"

flexAsynch ndbmtd

ot distri&tion aare,e@tra hop to data



35/44



'he8nfiniand

Clo&d '"

flexAsynch ndbmtd

*istri&tion aare,!ini!al hops



36/44



'he8nfiniand

Clo&d '"

flexAsynch ndbmtd


(atching ofre7&ests



37/44



'he8nfiniand

Clo&d '"

flexAsynch ndbmtd


(atching ofre7&ests

artitionedclient threads



38/44



9ntel hardware la 4Than1sM6

@J machines! each #ith -: cores 4B

HT threads6 - < soc1ets 'ntel Neon DHas#ellD 5-

processors

5ach soc1et

- @? cores 4 :dbApi connections per

machine

-


39/44


"y#$% Cl&ster o#$% #cale O&t

J @J @

J

J

@JJ

@J


40/44


etting started ith "y#$% Cl&ster

Try Cl"ster at OOW8(eneditaLs ands


41/44


etting started ith "y#$% Cl&sterTips

#tart s!all and si!ple

< "ini!al nodes K config&ration < GT 10" conc&rrent readsIH < #tart on localhost to r&le o&t fireall iss&es

et it &p and r&nning, then add co!ple@ity

+@peri!ent ith !ys7l / !ys7ld, node fail&res, applications

Consider &sing "y#$% Cl&ster "anager Ghttps;//edelivery.or

As) for help ; for&!s.!ys7l.co!

"y la

:eep )earnin. wit Ora%le

https://edelivery.oracle.com/https://edelivery.oracle.com/


42/44


Classroo!

'raining

%earning

#&scription

%ive >irt&al Class

'raining On

*e!and

:eep )earnin. wit Ora%leni(ersity

ed"%ation;ora%le;%om

Clo&

'ech

App

8nd&

:eynote; "onday, 3.00


43/44


:eynote; "onday, 3.00 -.00 p!, =(CA '#tate of the *olphin

• ich "ason, #> ? eneral "anager "y#$% (4, Oracle• 'o!as 4lin, > "y#$% +ngineering, Oracle

C&sto!er +@periences

Hari Tatrakal, *irector of *ataase #ervices, %ive ationOlaniyi Osinowo

, "y#$% ? Open #o&rce 'echnologies %ea!rnie So"rada ? #o$ W"lts%, *ataase +ngineers, int

"y#$% Cl&ster erfor!ance ains


44/44


"y#$% Cl&ster erfor!ance ains

#ynchrono&s A8 < Operation definition and execution are separated.

< #ingle &ser thread can define a $at% of operations, thenthe! together, ith only one A8 U *( ro&nd trip < A transaction can contain one or !ore atches of operati < 1 "ser tread < 1 e=e%"tin. transa%tion

Asynchrono&s A8 adds ;

< #ingle &ser thread can define, e@ec&te and ait for the re!&ltiple independent transactions. < 1 "ser tread < n e=e%"tin. transa%tions

Async Api allo#s the number o$ client threads to be reduced giving e$

con2178 clement-oow 15 cluster talk2

Documents