open source india - mysql labs: multi-source replication
DESCRIPTION
This session was presented in Open source India 2013. The presentation covers MySQL multi-source replication which is released under MySQL Labs.TRANSCRIPT
![Page 1: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/1.jpg)
1 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
MySQL Labs : Multi Source Replication
Shivji Kumar Jha, Software Developer,MySQL Replication Team
![Page 2: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/2.jpg)
2 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Safe Harbour Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or functionality, and should not
be relied upon in making purchasing decisions. The development, release, and
timing of any features or functionality described for Oracle’s products remains at the
sole discretion of Oracle.
![Page 3: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/3.jpg)
3 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
MySQL Labs features are not fit for production.
They are provided solely for testing purposes, to try the latest bug fixes and generally to keep up with the development.
Please, do not use these binaries in production.
Instead, install them on a spare server.
If you are looking for production ready binaries, please visit MySQL Downloads.
![Page 4: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/4.jpg)
4 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Agenda
Background: Why Use Replication?
Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitoring
Reading More about Multi-source Replication
![Page 5: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/5.jpg)
5 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Read scale-out
M SS
S
S
M
write clients read clientsread clients
write clients
Morereads?More
slaves!
Background: What is Replication Used For?
![Page 6: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/6.jpg)
6 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
M M?
write clients write clients
Morewrites?More
Masters?
What about Write scale-out?
M?
M?
Enter MySQL Fabric ...
![Page 7: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/7.jpg)
7 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Redundancy: If master crashes, promote slave to master
C
B
A
Background: What is Replication Used For?
C
B
ACrash
C
B
A
B is thenew master
Uh Oh! Whew!
![Page 8: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/8.jpg)
8 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
On-line Backup and Reporting
Background: What is Replication Used For?
M
S
write clients
business intelligent client apps reporting client apps big queries client apps
![Page 9: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/9.jpg)
9 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Image fromwww.ginkgomaps.com
Background: What is Replication Used For?
CB
BAAC
Image fromwww.ginkgomaps.com
![Page 10: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/10.jpg)
10 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Agenda
Background: Why Use Replication?
Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitoring
Reading More about Multi-source Replication
![Page 11: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/11.jpg)
11 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Multi-Source Replication: Introduction
A Slave can have more than one master.
S
M4
M1
M3
M2
![Page 12: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/12.jpg)
12 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
A Slave can have more than one master.
M1db1db1db1
Multi-Source Replication: Introduction
![Page 13: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/13.jpg)
13 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
A Slave can have more than one master.
M1
M2
db1db1db1
db2
Multi-Source Replication: Introduction
![Page 14: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/14.jpg)
14 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
A Slave can have more than one master.
M4
M1
M3
M2
db1db1db1
db4
db3
db2
Multi-Source Replication: Introduction
![Page 15: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/15.jpg)
15 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
A Slave can have more than one master.
S db1db1db1 db4db3db2
M4
M1
M3
M2
db1db1db1
db4
db3
db2
Multi-Source Replication: Introduction
![Page 16: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/16.jpg)
16 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Agenda
Background: Why Use Replication?
Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitoring
Reading More about Multi-source Replication
![Page 17: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/17.jpg)
17 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Multi-Source Replication: Use Cases
The main use cases of Multi-source replication are related to data aggregation.
![Page 18: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/18.jpg)
18 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Business IntelligenceData analytics
Backupetc
SM2
M1
M3
Database 2
Database 3
Database 1, 2, 3
Database 1
Multi-Source Replication: Use Cases
![Page 19: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/19.jpg)
19 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Multi-Source Replication: Use Cases
SM
M
M
Shard 2
Shard 3
Full table
Shard 1
![Page 20: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/20.jpg)
20 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Agenda
Background: Why Use Replication?
Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitoring
Reading More about Multi-source Replication
![Page 21: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/21.jpg)
21 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Multi-Source Replication: More details..
S
M4
M1
M3
M2
Slave can have more than one master.
– Receive transactions from several MySQL servers simultaneously.
– Apply transactions from different masters simultaneously.
– No conflict detection or resolution.
![Page 22: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/22.jpg)
22 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Insert...
Insert...
B
binary log
Insert...
relay log
Insert...
A
binary log
Client
Network
Multi-Source Replication: What is a Channel?
Senderthread
Receiverthread Applier
thread
A channel is an an abstraction for a sender-receiver-applier path.
Channel
![Page 23: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/23.jpg)
23 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
S
relay log1M1 binary log Network
Multi-Source Replication: How Many Channels?
Number of channels on slave = Number of sources.
relay log2M2 binary log Network
relay log3M3 binary log Network
binary log
![Page 24: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/24.jpg)
24 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
relay log1M1 binary log Network
Multi-Source Replication: Slave Appliers
Each channel has its own single-threaded slave applier.
relay log2M2 binary log Network
relay log3M3 binary log Network
binary logS
![Page 25: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/25.jpg)
25 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
binary log
M1 binary log Network
Multi-Source Replication: Slave Appliers
Each channel has its own multi-threaded slave applier.
SM2
M3
Network relay log2
relay log3Network relay log3
binary log
binary log
relay log1
![Page 26: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/26.jpg)
26 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Adding / Configuring A Channel
Each channel can be configured individually.
To add or alter a channel configuration use:
CHANGE MASTER TO master_def …
FOR CHANNEL=”<channel_name>”
![Page 27: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/27.jpg)
27 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
The FOR CHANNEL=”<channel_name>” clause is consistent across different replication commands.
– start slave [...] FOR CHANNEL=”<channel_name>"
– stop slave [thread_types] FOR CHANNEL=”<channel_name>”
– reset slave [all] FOR CHANNEL=”<channel_name>”
– flush relay logs FOR CHANNEL=”<channel_name>”
– show relay log events FOR CHANNEL=”<channel_name>”
...
Replication Commands on a Channel
![Page 28: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/28.jpg)
28 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
The FOR ALL CHANNELS clause.
– START SLAVE [thread_type] FOR ALL CHANNELS
– STOP SLAVE [thread_type] FOR ALL CHANNELS
Working With All Channels
![Page 29: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/29.jpg)
29 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Compatibility With Other Replication Features
Multi-Threaded Slaves Yes
Global Transaction Identifiers Yes
Semi-Synchronous Replication Yes
Crash Safe Slaves Yes
Filters Yes
![Page 30: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/30.jpg)
30 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Agenda
Background: Why Use Replication?
Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitoring
Reading More about Multi-source Replication
![Page 31: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/31.jpg)
31 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Replication Monitoring: P_S Replication Tables
Connection Configuration
Connection Status
Execution Configuration
Execution Status
Slave Status
Applier / CoordinatorStatus
WorkersStatus
![Page 32: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/32.jpg)
32 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
We have six performance schema tables for replication (MySQL-5.7.2):
– replication_connection_configuration
– replication_connection_status
– replication_execute_configuration
– replication_execute_status
– replication_execute_status_by_coordinator
– replication_execute_status_by_worker
Consistent semantics across tables. Lets explore one of them.
Replication Monitoring: P_S Replication Tables
One row per CHANNEL
One row per CHANNEL per WORKER
![Page 33: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/33.jpg)
3333 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
mysql> select * from performance_schema.replication_connection_status\G
*************************** 1. row ***************************CHANNEL_NAME : CHANNEL1SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12bTHREAD_ID : 13SERVICE_STATE : ONRECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4LAST_ERROR_NUMBER : 0LAST_ERROR_MESSAGE :LAST_ERROR_TIMESTAMP : 0000-00-00 00:00:00*************************** 2. row ***************************CHANNEL_NAME : CHANNEL2...
Replication Monitoring: Connection Status
One row per CHANNEL
![Page 34: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/34.jpg)
3434 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
mysql> select * from performance_schema.replication_connection_status\G
*************************** 1. row ***************************CHANNEL_NAME : CHANNEL1SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12bTHREAD_ID : 13SERVICE_STATE : ONRECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4LAST_ERROR_NUMBER : 0LAST_ERROR_MESSAGE :LAST_ERROR_TIMESTAMP : 0000-00-00 00:00:00*************************** 2. row ***************************CHANNEL_NAME : CHANNEL2...
Replication Monitoring: Connection Status
Receiver thread & its service state
![Page 35: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/35.jpg)
3535 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
mysql> select * from performance_schema.replication_connection_status\G
*************************** 1. row ***************************CHANNEL_NAME : CHANNEL1SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12bTHREAD_ID : 13SERVICE_STATE : ONRECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4LAST_ERROR_NUMBER : 0LAST_ERROR_MESSAGE :LAST_ERROR_TIMESTAMP : 0000-00-00 00:00:00*************************** 2. row ***************************CHANNEL_NAME : CHANNEL2...
Replication Monitoring: Connection Status
Set of transactions receivedthrough this channel
![Page 36: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/36.jpg)
3636 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
mysql> select * from performance_schema.replication_connection_status\G
*************************** 1. row ***************************CHANNEL_NAME : CHANNEL1SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12bTHREAD_ID : 13SERVICE_STATE : ONRECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4LAST_ERROR_NUMBER : 1045LAST_ERROR_MESSAGE : error connecting to master '[email protected]:13000' ...LAST_ERROR_TIMESTAMP : 2013-11-04 13:37:23*************************** 2. row ***************************CHANNEL_NAME : CHANNEL2...
Replication Monitoring: Connection Status
Oops! There was an error in the receiver thread on this channel
![Page 37: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/37.jpg)
3737 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
mysql> select * from performance_schema.replication_connection_status\G
*************************** 1. row ***************************CHANNEL_NAME : CHANNEL1SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12bTHREAD_ID : NULLSERVICE_STATE : OFFRECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4LAST_ERROR_NUMBER : 1045LAST_ERROR_MESSAGE : error connecting to master '[email protected]:13000' ...LAST_ERROR_TIMESTAMP : 2013-11-04 13:37:23*************************** 2. row ***************************CHANNEL_NAME : CHANNEL2...
Replication Monitoring: Connection Status
Receiver thread on this channel stopped serving...
![Page 38: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/38.jpg)
38 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Agenda
Background: Why Use Replication?
Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitoring
Reading More about Multi-source Replication
![Page 39: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/39.jpg)
39 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Read More About Multi Source Replication
Read more about Multi-source replication on Rith's blog:http://on-mysql-replication.blogspot.in/2013/09/mysql-labs-multi-source-replication.html
Read more about Multi-source replication from the design notes:http://dev.mysql.com/worklog/task/?id=1697
Read a feature preview of Multi-source replication on Rith's blog:http://on-mysql-replication.blogspot.in/2013/09/feature-preview-mysql-multi-source-replication.html
![Page 40: Open source India - MySQL Labs: Multi-Source Replication](https://reader034.vdocuments.us/reader034/viewer/2022042816/55964aea1a28ab8f1d8b46e8/html5/thumbnails/40.jpg)
40 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
Read more about replication P_S tables on Shiv's blog:http://shivjijha.blogspot.com/2013/09/Monitoring-Replication-with-the-NEW-performance-schema-tables.html
Read More About Multi Source Replication
Official MySQL documentation for replication P_S tables:http://dev.mysql.com/doc/refman/5.7/en/performance-schema-replication-tables.html