mysql 8.0: the new replication features...–much more data to store –e.g. social media, “look...
TRANSCRIPT
![Page 1: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/1.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved.
MySQL 8.0: The New Replication Features
Luís SoaresSoftware Development DirectorMySQL Replication
2nd October, 2019 Percona Live Europe 1
![Page 2: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/2.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
2nd October, 2019 Percona Live Europe 2
![Page 3: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/3.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Agenda
2nd October, 2019 Percona Live Europe 3
![Page 4: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/4.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Introduction
Use Cases
Enhancements in MySQL 8 (and 5.7)
Roadmap
Conclusion
Program Agenda
2nd October, 2019 Percona Live Europe
1
2
3
4
4
5
![Page 5: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/5.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Introduction
2nd October, 2019 Percona Live Europe
1
5
![Page 6: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/6.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Today...
– Technology mesh.
– All things distributed.
– Large amounts of data to handle, transform, store.
–Offline periods are horribly expensive, simply unaffordable.
– Go green requires dynamic and adaptative behavior.
–Much more data to store – e.g. social media, “Look at all of my pictures!”; Monitoring – Keeping logs for N years! ; IoT – and much more.
–Moving, transforming and processing data quicker than anyone else means having an edge over competitors.
– It is a zoo. Distributed coordination and monitoring is key.
6Percona Live Europe2nd October, 2019
![Page 7: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/7.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Database Replication
2nd October, 2019 Percona Live Europe
ServerB
ServerA
App
INSERT ...
INSERT ... INSERT ...
Replication“The process of generating and reproducing multiple copies of data at one or more sites.”,Database Systems: A Practical Approach to Design, Implementation, and Management, Thomas M. Connolly, Carolyn E. Begg, Third Edition, 2002.
7
![Page 8: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/8.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
MySQL Database Replication: Overview
2nd October, 2019 Percona Live Europe
INSERT ...Server
B
binary log
INSERT ...
relay log
INSERT ...Server
A
binary log
App
INSERT ...
Comm. Framework
8
PRIMARY SECONDARY
![Page 9: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/9.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
MySQL Database Replication: Some Notes
• Logical replication log recording master changes (binary log).
• Row or statement based format (may be intermixed).
• Each transaction is split into groups of events.
• Control events: Rotate, Format Description, Gtid, and more.
Binary Log
2nd October, 2019 Percona Live Europe
Layout of the Binary Log.
BEGIN ...E1 E2 COMMIT BEGIN ...E1 E2 COMMITGTID GTID
9
![Page 10: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/10.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Coordination Between Servers
MySQL Database Replication: Some Notes
102nd October, 2019 Percona Live Europe
A B Since 3.23
A B
semi-synchronous (plugin)
A CB
group replication (plugin)
transactions
transactions
Since 5.5
Since 5.7.17
asynchronous (native)
transactions, membership, coordination
acks
And in MySQL 8 as of 8.0.1
![Page 11: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/11.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Use cases
2nd October, 2019 Percona Live Europe
2
11
![Page 12: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/12.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Clustering Made Practical
2nd October, 2019 Percona Live Europe
RAISE
eplicate
utomate
ntegrate
cale
nhance
12
![Page 13: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/13.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Replicate
• For highly available infrastructures where:
– the number of servers has to grow or shrink dynamically;
– with as little pain as possible.
Group Replication
2nd October, 2019 Percona Live Europe
A B C D E
13
![Page 14: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/14.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Automate
• Single-primary mode
– Automatic PRIMARY/SECONDARY role assignment
– Automatic new PRIMARY election on PRIMARY failures
– Automatic setup of read/write modes on PRIMARY and SECONDARIES
– Automatic global consistent view of which server is the PRIMARY
Group Replication
2nd October, 2019 Percona Live Europe
S S S* S P S
Next PrimaryOld Primary
14
![Page 15: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/15.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
IntegrateBinary Log
2nd October, 2019 Percona Live Europe
• Logical replication log
– Extract, transform and load.
–MySQL fits nicely with other technologies.
INSERT ...Server
A
Replication log
App
INSERT ...
Hadoop
Kafka
Solr
Lucene
...
Transform
15
![Page 16: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/16.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Scale
• Replicate between clusters
– For disaster recovery
• Read Replicas
– For read-scale out. Deploy asynchronous read replicas connected to the cluster
Asynchronous Replication
2nd October, 2019 Percona Live Europe
A B C
J K L
D
I
M
Z
.
.
.
.
.
.
16
![Page 17: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/17.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
InnoDB Cluster
Enhance
2nd October, 2019 Percona Live Europe
S1 S2 S3 S4 S…
M
M M
MySQL Connector
Application
MySQL Router
MySQL Connector
Application
MySQL Router
MySQL Shell
HA
Rep
licaS
et1
• InnoDB Cluster Integrated Solution
– Group Replication for high availability.
– Asynchronous Replication for Read Scale-out.
–One-stop shell to deploy and manage the cluster.
– Seamlessly and automatically route the workload to the proper database server in the cluster.
– Hide failures from the application.
17
![Page 18: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/18.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Enhancements in MySQL 8 (and 5.7)
2nd October, 2019 Percona Live Europe
Consistency3.1
3
18
Operations3.2
Monitoring3.3
Performance3.4
Security3.5
Other3.6
![Page 19: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/19.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Consistency Levels
• Eventual Consistency (default)
– Transaction does not wait at all.
– Executes on the current snapshot of the data on that member.
• Before Consistency (Synchronize on Reads)– Transaction waits for all preceding transactions to complete.
– Executes on the most up to date snapshot of the data in the group.
• After Consistency (Synchronize on Writes)
– Transaction waits until all members have executed it.
– Executes on the current snapshot of the data on that member.
2nd October, 2019 Percona Live Europe 19
New in 8.0.14
![Page 20: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/20.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Consistency Levels
• Before and After (Yes, you can combine both)
– Transaction waits for all preceding transactions and for all members to execute it.
– Executes on the most up to date snapshot of the data in the group and updates everywhere before returning to the application.
• Before On Primary Fail-over– Transaction waits for all transactions in the new primary’s replication backlog to be
executed.
– Executes on the snapshot of the data that the old primary was in when it stepped down (or crashed).
202nd October, 2019 Percona Live Europe
New in 8.0.14
![Page 21: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/21.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Transaction Does Not Wait for to Complete Before it Executes On Member A.
Accessing Data: Synchronize Execution Eventually
212nd October, 2019 Percona Live Europe
B CA
App
X=5 WHERE X=1
B CA
App
READ X=?
B CA
App
X = 1
New in 8.0.14
SET @@session.group_replication_consistency=EVENTUAL
![Page 22: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/22.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Transaction i Waits for to Complete Before it Executes On Member A.
Accessing Data: Synchronize Before Execution
222nd October, 2019 Percona Live Europe
B CA
App
X=5 WHERE X=1
B CA
App
X = 5
B CA
App
READ X=?
waits until member A receives and commits A.
New in 8.0.14
SET @@session.group_replication_consistency=BEFORE
![Page 23: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/23.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Waits For All Members to Execute. Reads Updated Value Without Waiting.
Accessing Data: Synchronize After Execution
232nd October, 2019 Percona Live Europe
B CA
App
X=5 WHERE X=1
B CA
App
READ X? X=5
B CA
App
commits after all membershave received and executed it.
reads X = 5 immediately.
Commit OK.
New in 8.0.14
SET @@session.group_replication_consistency=AFTER
![Page 24: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/24.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Waits For All Members to Execute. Reads Updated Value Without Waiting.
Accessing Data: Synchronize Before on Primary Fail-over
242nd October, 2019 Percona Live Europe
B CA
App
X=5 WHERE X=1
CA
App
READ X? X=5
B CA
App
A (new primary) holds executionuntil is executed.
New in 8.0.14
SET @@session.group_replication_consistency=BEFORE_ON_PRIMARY_FAILOVER
![Page 25: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/25.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Consistency Levels – User Interface
• System Variable Controls the Behavior: group_replication_consistency.
• Global and Session scope
– Can be set per transaction.
• Values:
– EVENTUAL
– BEFORE_ON_PRIMARY_FAILOVER
– BEFORE
– AFTER
– BEFORE_AND_AFTER
252nd October, 2019 Percona Live Europe
New in 8.0.14
![Page 26: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/26.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Consistency Levels
• Consistency has an impact on throughput.
• Usually not all transactions are executed under strong consistency requirements...
262nd October, 2019 Percona Live Europe
1 8 16 32 64 128 256 512
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
Sysbench RW Sustained Throughput
Durable settings
EVENTUAL BEFORE AFTER BEFORE&AFTER
number of client threads
tra
nsa
ctio
ns p
er
se
co
nd
1 8 16 32 64 128 256 512
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
Sysbench Update Index Sustained Throughput
Durable settings
EVENTUAL BEFORE AFTER BEFORE&AFTER
number of client threadstr
an
sa
ctio
ns p
er
se
co
nd
![Page 27: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/27.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Enhancements in MySQL 8 (and 5.7)
2nd October, 2019 Percona Live Europe
Consistency3.1
3
27
Operations3.2
Monitoring3.3
Performance3.4
Security3.5
Other3.6
![Page 28: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/28.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Automatically Setting Server to Read-Only
Preventing (Involuntary) Writes on Stale Members
282nd October, 2019 Percona Live Europe
B C
A
B CA
A joins a single primary cluster andit allows update operations priorto joining.
A becomes a secondary and isautomatically set to read only.
B C
A
A leaves the group and remainsread only until the DBA reverts.
Backported to 5.7.20
![Page 29: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/29.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Automatically Shoot Member in the Head (ST*NITH)
Preventing (Involuntary) Reads/Writes on Stale Members
292nd October, 2019 Percona Live Europe
B C
A
A shuts itself down, thus becomingeffectively unavailble for both readsand write operations.
B CA
A is a secondary and wasautomatically set to read only.
B C
A
A leaves the group involuntarily(failed to apply changes, networkpartitioned, etc).
New in 8.0.12
@@group_replication_exit_state_action={ READ_ONLY | ABORT_SERVER }
Backported to 5.7.24
![Page 30: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/30.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Choose next primary by assigning election weights to the candidates.
Control Primary Promotion: Priorities
302nd October, 2019 Percona Live Europe
B CAw=100 w=90w=50
B
CA
w=100
w=90w=50
CAw=90w=50
B is the primary.
B is not in the group anymore.C has higher weight than A.
C is elected the new primary.
Backported to 5.7.20
![Page 31: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/31.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
User tells current primary to give up its role and assign it to another server.
Control Primary Election: Choose Your Primary.
312nd October, 2019 Percona Live Europe
B CA B CA
B is the primary.
Now I want A to be the primary, not B.
New in 8.0.13
A CB
A is the primary. B stepped down.
Ops
![Page 32: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/32.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
User can specify, online, on which mode the group operates.
Single-Primary to Multi-Primary and Back Online
322nd October, 2019 Percona Live Europe
B CA B CA
Single-primary mode,B is the primary.
Now I want the group operating inmulti-primary mode.
New in 8.0.13
A CB
Return to single-primary mode.
Ops
![Page 33: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/33.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
User controls the amount of time to wait until others decide to evict a member from the group.
Dealing With Unreliable Networks: Relaxed Member Eviction
332nd October, 2019 Percona Live Europe
B CA B CA
I think A is not around anymore.Lets kick it out of the group!
New in 8.0.13
C
See… I told you. A is back!Good thing we waited.
Hold on. Lets wait for some time!Maybe it is just too busy to talk to us
right now...
OK. Lets give it somemore time!
UNREACHABLE
A B
![Page 34: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/34.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Member tries to rejoin automatically in case it gets evicted.
Automatic Cluster-Rejoin on Partitions
342nd October, 2019 Percona Live Europe
B C
A automatically tried (and succeeded)to rejoin the cluster after it had beenevicted. No user intervention.
B CA
A is a secondary and wasautomatically set to read only.
B C
A
A leaves the group involuntarilyBecause of network issues.
New in 8.0.16
A
A
![Page 35: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/35.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Enhancements in MySQL 8 (and 5.7)
2nd October, 2019 Percona Live Europe
Consistency3.1
3
35
Operations3.2
Monitoring3.3
Performance3.4
Security3.5
Other3.6
![Page 36: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/36.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Through the entire asynchronous topology
Monitor Lag With Microsecond Precision
362nd October, 2019 Percona Live Europe
A B C D
How much time does my data take to reachD coming from A?
![Page 37: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/37.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
From the immediate master
Monitor Lag With Microsecond Precision
372nd October, 2019 Percona Live Europe
A B C D
How much time does my data originated inA takes to flow from B to C?
![Page 38: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/38.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
For each stage of the replication applier process
Monitor Lag with Microsecond Precision
382nd October, 2019 Percona Live Europe
D
Data
Worker ThreadsRelay Log
CoordinatorThread
ReceiverThread
• Per Stage Timestamps
– User can monitor how much time it takes for a specific transaction to traverse the pipeline.
![Page 39: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/39.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Version, Role and more
Global Group Stats Available on Every Server
392nd October, 2019 Percona Live Europe
A B C
A is primary.B, C are secondaries.A: 5.7.20B, C: 8.0.3
A is primary.B, C are secondaries.A: 5.7.20B, C: 8.0.3
A is primary.B, C are secondaries.A: 5.7.20B, C: 8.0.3
• Query one Replica, Get status of all
– Every replica reports group-wide information about roles and versions of the members of the group.
– Also available at any replica are group-wide status.
![Page 40: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/40.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Group Replication Message Cache Memory Usage
• GCS/XCom’s Paxos message cache is instrumented.
• GCS/XCom’s Paxos message cache memory usage is exposed in performance schema.
402nd October, 2019 Percona Live Europe
-- This is a session open on ServerA and the user is reading stats on GCS_Xcom message cacheServerA> select * from memory_summary_global_by_event_name where event_namelike "%GCS_XCom%"\G*************************** 1. row ***************************
EVENT_NAME: memory/group_rpl/GCS_XCom::xcom_cacheCOUNT_ALLOC: 28890317COUNT_FREE: 28840318
SUM_NUMBER_OF_BYTES_ALLOC: 24499151783SUM_NUMBER_OF_BYTES_FREE: 24470424555
LOW_COUNT_USED: 0CURRENT_COUNT_USED: 49999
HIGH_COUNT_USED: 50000LOW_NUMBER_OF_BYTES_USED: 0
CURRENT_NUMBER_OF_BYTES_USED: 28727228HIGH_NUMBER_OF_BYTES_USED: 135676530
1 row in set (0.01 sec)
New in 8.0.12
![Page 41: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/41.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Enhancements in MySQL 8 (and 5.7)
2nd October, 2019 Percona Live Europe
Consistency3.1
3
41
Operations3.2
Monitoring3.3
Performance3.4
Security3.5
Other3.6
![Page 42: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/42.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Highly Efficient Replication Applier
• Delivers the best throughput of the three dependency trackers, at anyconcurrency level.
• WRITESET dependency tracking allows applying a single threaded workload in parallel.
• Fast Group Replication recovery – time to catch up.
Write set parallelization
422nd October, 2019 Percona Live Europe
Backported to 5.7.22
![Page 43: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/43.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Write set parallelization
Highly Efficient Replication Applier
432nd October, 2019 Percona Live Europe
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
50000
1 2 4 8 16 32 64 128 256
Up
dat
es/
seco
nd
Ap
plie
s o
n t
he
Re
plic
a
Number of Clients on the Master
Applier Throughput: Sysbench Update Index
COMMIT_ORDER WRITESET WRITESET_SESSION
Backported to 5.7.22
![Page 44: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/44.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Replica quickly online by using WRITESET
Fast Group Replication Recovery
442nd October, 2019 Percona Live Europe
0
1
2
3
4
5
6
7
8
9
10
Sysbench RW at 33% capacity(workload: 9K TPS on 64 threads)
Sysbench RW at 66% capacity(workload: 18K TPS on 64 threads)
Tim
e to
cac
th-u
p p
er t
ime
of
wo
rlo
ad m
issi
ng
(rat
io)
Group Replication Recovery Time: Sysbench Update Index (durable settings)
MySQL 5.7.20 MySQL 8.0.3
0
1
2
3
4
5
6
7
8
9
10
Sysbench RW at 33% capacity(workload: 4K TPS on 64 threads)
Sysbench RW at 66% capacity(workload: 8K TPS on 64 threads)
Tim
e to
cac
th-u
p p
er t
ime
of
wo
rlo
ad m
issi
ng
(rat
io)
Group Replication Recovery Time: Sysbench RW (durable settings)
MySQL 5.7.20 MySQL 8.0.3
Backported to 5.7.22
![Page 45: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/45.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Empty or delayed replica quickly online by using Automatic Cloning and WRITESET
Even Faster Group Replication Recovery: Clone Support
452nd October, 2019 Percona Live Europe
New in 8.0.17
B CA
D
D is empty or has very old data (a lot to catch up)
B CA D
D automatically takes a snapshot of A (clones A and restores the
image into itself). D’s old data is forever gone.
B CA D
D is has recoverd and has caught up using a snapshot of A and a
small amount of binary logs
Requires that the clone plugin is installed. The clone plugin is shipped with MySQL 8.0.17.
![Page 46: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/46.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Even Faster Group Replication Recovery
• Recovery using binary logs only vs recovery using clone and binary logs together.
– There are cases binary logs are quicker and cases clone together with binary logs take less time.
• No Network involved.
– Network adds latency
– Network may not impact throughput (if it is not a bottleneck).
Empty or delayed replica quickly online by using Automatic Cloning and WRITESET
462nd October, 2019 Percona Live Europe
New in 8.0.17
0 1 2 3 4 5 6 7 8 9 10
0
4
8
12
16
20
Database State Transfer Throughput(TPCC Workload, 100GB Database, local copy)
replication time (slave available) clone time (slave unavailable)
Size of the changes in the binlog (GB)
Tim
e t
o p
roce
ss (
min
ute
s)
![Page 47: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/47.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
More transactions per second while sustaining zero lag on any replica
High Cluster Throughput
472nd October, 2019 Percona Live Europe
8 16 32 64
Number of Clients on the Master
Asynchronous Replication Sustained Throughput(Sysbench Update Index, non-durable settings)
MySQL 5.7 MySQL 8.0.3
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
8 16 32 64
Up
dat
es
pe
r se
con
d
Number of Clients on the Master
Asynchronous Replication Sustained Throughput(Sysbench Update Index, durable settings)
MySQL 5.7 MySQL 8.0.3
![Page 48: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/48.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Replicate only changed fields of documents (Partial JSON Updates)
Efficient Replication of JSON Documents
482nd October, 2019 Percona Live Europe
0
10000
20000
30000
40000
50000
60000
Entire JSON Partial JSON
Byt
es p
er
tran
sact
ion
Binary Log Space Per Transaction
FULL MINIMAL
• Numbers are from a specially designed benchmark:• tables have 10 JSON fields,• each transaction modifies
around 10% of the data
![Page 49: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/49.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
0,0
500,0
1000,0
1500,0
2000,0
2500,0
1 2 3 4
Tran
sact
ion
s p
er s
eco
nd
Throughput on the Master: Partial JSON vs Complete JSON
4 8 16 32 64
Replicate only fields of the document that changed (Partial JSON Updates)
Efficient Replication of JSON Documents
492nd October, 2019 Percona Live Europe
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
Throughput on the Slave: Partial JSON vs Complete JSON
4 8 16 32 64
FULL MINIMAL FULL MINIMAL FULL MINIMAL FULL MINIMAL
COMPLETE JSON PARTIAL JSON COMPLETE JSON PARTIAL JSON
![Page 50: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/50.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Enhancements in MySQL 8 (and 5.7)
2nd October, 2019 Percona Live Europe
Consistency3.1
3
50
Operations3.2
Monitoring3.3
Performance3.4
Security3.5
Other3.6
![Page 51: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/51.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Binary Log Encryption on Disk
2nd October, 2019 Percona Live Europe
App
INSERT ...
Comm. Framework
51
INSERT ...Server
B
binary log
INSERT ...
relay log
INSERT ...Server
A
binary log
<empty>
relay log
PRIMARY SECONDARY
New in 8.0.14
![Page 52: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/52.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Binary Log Encryption on Disk
2nd October, 2019 Percona Live Europe
App
INSERT ...
Comm. Framework
52
INSERT ...Server
B
binary log
INSERT ...
relay log
INSERT ...Server
A
binary log
<empty>
relay log
PRIMARY SECONDARY
Relay logs are also encrypted.
Binary logs are encrypted.
New in 8.0.14
![Page 53: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/53.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Binary Log Encryption on Disk
• Protects Binary Log Data at rest.
• Controllable Using a System Variable: binlog_encryption
• Two tier encryption, one master key and one key per file.
• Rotate the Binary Log Encryption Key:
ALTER INSTANCE ROTATE BINLOG KEY
2nd October, 2019 Percona Live Europe 53
New in 8.0.14
New in 8.0.16
![Page 54: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/54.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 54
1 8 16 32 64 128 256 1 8 16 32 64 128 256Sysbench RW Sysbench Update Index
0
8000
16000
24000
32000
40000
Sustained Asynchronous Replication Throughput with Encryption
No encryption Encrypted binlog
benchmark, client threads
Tra
nsa
ctio
ns p
er
se
co
nd
(T
PS
)
1 8 16 32 64 128 256 1 8 16 32 64 128 256Sysbench RW Sysbench Update Index
0
8000
16000
24000
32000
40000
Sustained Group Replication Throughput with Encryption
No encryption Encrypted binlog
benchmark, client threadsT
ran
sa
ctio
ns p
er
se
co
nd
(T
PS
)
Performance
• Marginal impact on throughput
–More visible when the commit rate is higher.
2nd October, 2019 Percona Live Europe 54
![Page 55: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/55.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Enhancements in MySQL 8 (and 5.7)
2nd October, 2019 Percona Live Europe
Binary Log Enhancements 3.1
3
55
Operations3.2
Monitoring3.3
Performance3.4
Security3.5
Other3.6
![Page 56: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/56.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Changes to defaults in MySQL 8
• Binary log is on by default.
• Logging of slave updates is on by default.
• Replication metadata is stored in InnoDB tables by default instead of files.
• Row-based applier falls back into hash scans to find rows instead of table scans.
• Transaction write-set extraction is on by default.
• Binary log expiration is set to 30 days by default.
• Server-id is set to 1 by default instead of 0.
High performance replication enabled out-of-the-box
562nd October, 2019 Percona Live Europe
![Page 57: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/57.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Other MySQL 8 Group Replication Enhancements
• Monitoring: Group Replication threads instrumented and shown in performance schema
• Monitoring: Group Replication conditional variables and mutexesinstrumented and shown in performance schema
• Operations: SAVEPOINT support when write sets are being extracted
• Operations: Support hostnames in Group Replication whitelist
• Operations: New options to fine tune the cluster automatic flow control.
• Operations: Cross-version replication policies for GR
572nd October, 2019 Percona Live Europe
Backported to 5.7.19
Backported to 5.7.21
New in 8.0.17
![Page 58: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/58.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Other MySQL 8 Group Replication Enhancements
• Performance: More efficient code path between network layer and replication layer.
• Performance: Configurable communication pipeline size for group replication.
• Performance: Enhanced large transactions support for group replication.
• Performance: support for protocol compression in mysqlbinlog
• Troubleshooting: Dynamic and high performance debugging of group replication inter-node messaging
• Security: Encryption of transient replication files.
• Infrastruture: IPv6 support.
582nd October, 2019 Percona Live Europe
New in 8.0.13
New in 8.0.16
New in 8.0.17
New in 8.0.17
New in 8.0.14
![Page 59: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/59.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Roadmap
2nd October, 2019 Percona Live Europe
4
59
![Page 60: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/60.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
The Road to MySQL 8 Group Replication and InnoDB Clusters
2nd October, 2019 Percona Live Europe
MySQL 5.7.95.7 is GAlifecycle interfacesP_S tables for GRserver side changes
GR 0.2.0 labsHello world!
MySQL 5.7.17GR is GA
MySQL 8.0.1 DMRGR is released with 8InnoDB Cluster is GALots of replication enhancements
MySQL 8.0.3 RC1bug fixespartial json updatesmonitoring enhancements
60
MySQL 8.0.4 RC2bug fixesmonitoring
MySQL 8.0.11 GAbug fixesMySQL 8.0.0 DMR
MySQL 8.0.2 DMRMore replicationenhancements
MySQL 8.0.12 GAMember auto-shutdownInstrumentation
MySQL 8.0.13 GASelect PrimaryMonitoringPerformanceRelax Eviction
MySQL 8.0.14 GAConsistencyEncryption
MySQL 8.0.16 GALarge TrxsAuto-rejoin
......
MySQL 8.0.17 GAClone SupportEncryptionenhancements
![Page 61: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/61.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Conclusion
2nd October, 2019 Percona Live Europe
5
61
![Page 62: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/62.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Conclusion
Latest MySQL 8 GA is out:
• Performance/efficiency improvements
– Automatic Cloning of donors in Group Replication means one less provisioning step required from the user.
• More encryption features
– Encrypt even transient replication data that touches the disk.
• Improved Operatinos and DBA experience– Mysqlbinlog supports protocol compression
– Enhanced cross-version replication protection in Group Replication.
– Enhanced distributed recovery by integrating the clone plugin. Seamless and automatic snapshotting, provisioning and catch up. Reduced operations overhead.
2nd October, 2019 Percona Live Europe 62
![Page 63: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/63.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Where to go from here?
• Packages
– http://www.mysql.com/downloads/
• Documentation– https://dev.mysql.com/doc/refman/8.0/en/
• Blogs from the Engineers (news, technical information, and much more)
– http://mysqlhighavailability.com
2nd October, 2019 Percona Live Europe 63
![Page 64: MySQL 8.0: The New Replication Features...–Much more data to store –e.g. social media, “Look at all of my pictures!”; Monitoring –Keeping logs for N years! ; IoT –and much](https://reader034.vdocuments.us/reader034/viewer/2022051916/60081817409946022d62ce9a/html5/thumbnails/64.jpg)
Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |