Tungsten Replicator Tutorial Uc 2013

Continuent 2013
Using Tungsten Replicator

to solve replication
problems
Neil Armitage, Cluster implementation Engineer, Continuent
Giuseppe Maxia, QA Director, Continuent
1
1 Monday, April 22, 13
Continuent 2013
ABOUT US
Neil Armitage
Continuent Tungsten Deployment and Support

Engineer, Continuent, Inc
20 years development and DB experience
Giuseppe Maxia, a.k.a. "The Data Charmer"
QA Director, Continuent, Inc
25 years development and DB experience
long timer MySQL community member. Oracle

ACE Director
2
Continuent 2013
Tungsten replicator
Global transaction ID
Multiple masters
Multiple sources
Flexible topologies
Parallel replication
Heterogeneous replication
... and more

3
Continuent 2013
What Tungsten Replicator is NOT
Automated management
Automatic failover
Transparent connections
All the above (and more) are available with a

commercial solution named Continuent
Tungsten (a.k.a. Tungsten Enterprise)
4
Continuent 2013
What are we talking about?
Requirements
Components
Installation
Topologies
Administration
Troubleshooting
5
Continuent 2013
Tungsten Replicator Concepts
6
Role
service
Replicator
Master, slave, direct slave
A.k.a. "pipeline"
The replication engine
stage
extract,queue,apply
Continuent 2013
Tungsten Replicator Components
7
THL
service schema
properties file
Transaction History Log
Makes the node crash proof
service denition
tools
Ruling from a centralized
location
Continuent 2013
Tungsten Replicator in a nutshell
binlog
THL
slave
master
host1
host2
THL
trep_commit_seqno trep_commit_seqno trep_commit_seqno
origin seqno eventid
trep_commit_seqno trep_commit_seqno trep_commit_seqno
origin seqno eventid
global
transaction ID
8
Continuent 2013
Planning
9
Hosts
Topology
Stand-alone or taking over

Continuent 2013
star
master!slave
Heterogeneous
fan!in slave all!masters
MySQL
Oracle
Oracle
MySQL Oracle
Oracle
MySQL
MySQL
Continuent 2013
Installation
11
Continuent 2013
Installation
System Requirements
Validate !rst
Deploying from a single location

12
Continuent 2013
Installation - tools
tools/ tungsten-installer
tools/ con!gure-service
tools/update
(Using the cookbook recipes, you hardly see

them)
13
Continuent 2013
Tungsten in practice
Installation
14
Continuent 2013
Installation
Check the requirements
Get the binaries
Expand the tarball
Run cookbook
15
Continuent 2013
REQUIREMENTS
Java JRE or JDK (Sun/Oracle or Open-jdk)
Ruby 1.8 (only during installation)
ssh access to the same user in all nodes
MySQL user with all privileges

16
Continuent 2013
Installation - Choices
--master-slave
--direct
17
Continuent 2013
binlog
THL
THL
slave
slave
master
host1
host2
host3
THL
18
master-slave
Continuent 2013
binlog
THL
slave
master
relay log
host1
host2
host3
THL
slave
relay log
19
direct
Continuent 2013
Overview of Virtual Machines
Copy zip !les from USB Key
Expand on local disk
Start all 4 Machines in VirtualBox

20
Continuent 2013
Virtual Machines
4 Nodes host1->host4
Running centos 6.3 and Percona 5.5
Root and tungsten password = password
localhost port 2222 redirects to 22 on hosts

21
!!" $% &&&& '()*!'+),-./0-".!'
Continuent 2013
VERY important de!nitions
Staging directory:
Where you unpack the software and run the

installer.
There is generally only one, in one host;
Can be discarded after installation
Installation directory:
Where your installed software will go;
There is one for every host;

22
Continuent 2013
Example
23
host1
host3
Staging directory:
$HOME/tungsten-replicator-2.0.8-167
host2
/opt/replication
/opt/replication
/opt/replication
Continuent 2013
Requirements : how to
step by step: how it happened

24
Continuent 2013
installing VMs
Step-by-step demo
25
Continuent 2013
Overview of Tungsten
cookbook
26
Continuent 2013
tungsten cookbook
tungsten-replicator-2.0.8-167
|
+--/cluster-home
+--/cookbook
+--/tools
+--/tungsten-replicator
27
Continuent 2013
tungsten cookbook
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_MASTER_SLAVE.sh
+--install_master_slave
+--show_cluster
+--test_cluster

...
28
Continuent 2013
tungsten cookbook
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_ALL_MASTERS.sh
+--install_all_masters
+--show_cluster
+--test_cluster

...
29
Continuent 2013
tungsten cookbook
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_STAR.sh
+--install_star
+--show_cluster
+--test_cluster

...
30
Continuent 2013
tungsten cookbook
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_FAN_IN.sh
+--install_fan_in
+--show_cluster
+--test_cluster

...
31
Continuent 2013
tungsten cookbook
$ cat COMMON_NODES.sh
export NODE1=host1
export NODE2=host2
export NODE3=host3
export NODE4=host4
32
Continuent 2013
tungsten cookbook
$ cat USER_VALUES.sh
# User defined values for the cluster to be
installed.
export TUNGSTEN_BASE=$HOME/installs/cookbook
export DATABASE_USER=tungsten
export BINLOG_DIRECTORY=/var/lib/mysql
export MY_CNF=/etc/my.cnf
export DATABASE_PASSWORD=secret
export DATABASE_PORT=3306
export TUNGSTEN_SERVICE=cookbook
export RMI_PORT=10000
export THL_PORT=2112
export START_OPTION=start
33
Continuent 2013
Getting started: VALIDATE FIRST
export VERBOSE=1
./cookbook/check_cookbook
./cookbook/validate_cluster
34
Continuent 2013
sample master-slave installation
edit cookbook/COMMON_NODES.sh
edit cookbook/USER_VALUES.sh
run cookbook/install_master_slave
and then:
run cookbook/show_cluster
run cookbook/test_cluster
35
Continuent 2013
What does the installation do
host4
1: Validate all servers
host1 host2 host3
Report all errors
36
Continuent 2013
host4
1: (again) Validate all servers
host1 host2 host3
37
Continuent 2013
2: install Tungsten in all servers
host3
$HOME/
tinstall/
config/
releases/
relay/
thl/
tungsten/
backups/
host4
host1
host2
38
Continuent 2013
example (from manual installation)
ssh r2 chmod 444 $HOME/tinstall
./tools/tungsten-installer \
--master-slave --master-host=r1 \
--datasource-user=tungsten \
--datasource-password=secret \
--service-name=dragon \
--home-directory=$HOME/tinstall \
--thl-directory=$HOME/tinstall/logs \
--relay-directory=$HOME/tinstall/relay \
--cluster-hosts=r1,r2,r3,r4 --start
ERROR >> qa.r2.continuent.com >> /home/tungsten/
tinstall is not writeable
39
Continuent 2013
example
ssh r2 chmod 755 $HOME/tinstall
--master-slave --master-host=r1 \
--datasource-user=tungsten \
--datasource-password=secret \
--service-name=dragon \
--home-directory=$HOME/tinstall \
--thl-directory=$HOME/tinstall/logs \
--relay-directory=$HOME/tinstall/relay \
--cluster-hosts=r1,r2,r3,r4 --start
# no errors
40
Continuent 2013
After installation.
A tour of the cookbook
utilities
41
Continuent 2013
General principles (1)
42
Scripts without extension are designed to be

launched by users
e.g. ./cookbook/help
./cookbook/install_master_slave
Scripts with extension ".sh" are either for

internal use only or deprecated.
./cookbook/install_* scripts can be used

before installing. Most everything else
require an installed topology
Continuent 2013
General principles (2)
43
After installation there is a !le

CURRENT_TOPOLOGY in the staging
directory
cookbook scripts can be used either from the

staging directory or from the installation
directory.
Continuent 2013
Cookbook tour: help and checks
44
./cookbook/check_cookbook
./cookbook/help
./cookbook/readme
Continuent 2013
Cookbook tour: Getting information
45
./cookbook/show_cluster
./cookbook/paths
./cookbook/backups
./cookbook/services
./cookbook/query_node {node} {query}
./cookbook/query_all_nodes {query}
Continuent 2013
Cookbook tour: Inspecting replication
46
./cookbook/replicator
./cookbook/trepctl
./cookbook/thl
./cookbook/show_conf
./cookbook/edit_conf
./cookbook/show_log
./cookbook/vimlog
./cookbook/emacslog
Continuent 2013
Cookbook tour: testing tools
47
./cookbook/test_cluster
./cookbook/start_load [start|stop]
./cookbook/test_all_topologies
Continuent 2013
Cookbook tour: powerful admin tools
48
./cookbook/heartbeat
./cookbook/switch
./cookbook/add_node_master_slave
./cookbook/add_node_star
./cookbook/copy_backup
./cookbook/clear_cluster # <--- CAUTION!
Continuent 2013
More installation
49
Continuent 2013
DRY-RUN
50
Method to simulate installation;
Does NOT perform installation;
Does NOT even do validation;
It only shows the commands used to install;
Allows you to get the commands and do an

installation manually (e.g. when you can't ssh
between nodes)
Continuent 2013
DRY-RUN
51
export DRYRUN=1
Continuent 2013
Intro to multi-master installation
52
Continuent 2013
How tungsten-installer Works for
Basic Master/Slave Deployment
53
db1
db2
db3
Staging copy
of les
check prereqs
copy code
congure
Continuent 2013
From Master/Slave Replication ...
54
db1
Replicator
db3
Service alpha
db2
Replicator
Service alpha
Replicator
Service alpha
Install master and slaves on the whole cluster
tungsten-
installer
tungsten-
installer
tungsten-
installer
Continuent 2013
To Multi-Master
55
db1
Replicator
Service alpha
Service bravo
db2
Replicator
Service bravo
Service alpha
Install master on
db1
tungsten-
installer
install master
on db2
tungsten-
installer
install slave
service on db1
con!gure-
service
install slave
service on db2
con!gure-
service
Continuent 2013
tungsten-installer master 1
56
TUNGSTEN_HOME=/home/tungsten/installs/cookbook
./tools/tungsten-installer
--master-slave
--master-host=$MASTER1
--datasource-port=3306
--datasource-user=tungsten
--datasource-password=secret
--datasource-log-directory=/var/lib/mysql
--service-name=alpha
--home-directory=$TUNGSTEN_HOME
--cluster-hosts=$MASTER1
--start
creating service 'alpha'
Notice: --cluster-hosts has only one host
Continuent 2013
tungsten-installer master 2
57
./tools/tungsten-installer
--master-slave
--master-host=$MASTER2
--datasource-port=3306
--datasource-user=tungsten
--datasource-password=secret
--datasource-log-directory=/var/lib/mysql
--service-name=bravo
--home-directory=$TUNGSTEN_HOME
--cluster-hosts=$MASTER2
--start
creating service 'bravo'
Notice: --cluster-hosts has only one host
Continuent 2013
Con!gure Service master 1
58
$TUNGSTEN_HOME/tungsten/tools/configure-service -C
--quiet
--host=$MASTER1
--datasource=$MASTER1
--local-service-name=alpha
--role=slave
--service-type=remote
--release-directory=$TUNGSTEN_HOME/tungsten
--skip-validation-check=THLStorageCheck
--master-thl-host=$MASTER2
--master-thl-port=2112
--svc-start bravo
Notice: bravo is the master service in host 2
Continuent 2013
Con!gure Service master 2
59
$TUNGSTEN_HOME/tungsten/tools/configure-service -C
--quiet
--host=$MASTER2
--datasource=$MASTER2
--local-service-name=bravo
--role=slave
--service-type=remote
--release-directory=$TUNGSTEN_HOME/tungsten
--skip-validation-check=THLStorageCheck
--master-thl-host=$MASTER1
--master-thl-port=2112
--svc-start alpha
Notice: alpha is the master service in host 1
Continuent 2013
From Master/Slave Replication ...
60
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
./cooobook/install_master_slave
Continuent 2013
How Do I Install Fan-In Replication?
61
db1
Replicator
db3
Service db1
db2
Replicator
Service db2
Replicator
Service db1
Service db2
./cooobook/install_fan_in
Continuent 2013
How Do I Install Multi-Master?
62
db1
Replicator
Service db1
Service db2
db2
Replicator
Service db1
Service db2
./cooobook/install_all_masters
Continuent 2013
How Do I Extend Multi-Master?
63
db1
Replicator
Service db1
Service db2
Service db3
db3
Service db1
Service db2
Service db3
db2
Replicator
Service db1
Service db2
Service db3
Replicator
Continuent 2013
How Do I Extend Multi-Master?
64
db1
db3
Service db1
Service db2
Service db3
db2
Replicator
db4
Service db1
Service db2
Service db4
Replicator
Service db3
Service db4
Service db1
Service db2
Service db3
Replicator
Service db4
Service db1
Service db2
Service db3
Replicator
Service db4
Continuent 2013
How Do I Install a Star Topology?
65
db1
Replicator
Service db1
Service db3
db3
Service db1
Service db2
Service db3
db2
Replicator
Service db2
Service db3
HUB
Replicator
./cooobook/install_star
Continuent 2013
How Do I Extend a Star Topology?
66
db1
Replicator
Service db1
Service db3
db3
Service db1
Service db2
Service db3
db2
Replicator
Service db2
Service db3
db4
Replicator
Service db3
Service db4
HUB
Service db4
Continuent 2013
How Do I Extend a Star Topology?
67
db1
Replicator
Service db1
Service db3
db3
Service db1
Service db2
Service db3
db2
Replicator
Service db2
Service db3
db4
Replicator
Service db3
Service db4
HUB
Service db4
db5
Replicator
Service db5
Service db3
Service db5
Continuent 2013
BI-DIR: the painless way
edit cookbook/COMMON_NODES.sh
edit cookbook/USER_VALUES.sh
remove two nodes
edit the variables in cookbook/

NODES_ALL_MASTERS.sh
cookbook/install_all_masters
68
Continuent 2013
Multiple masters
fan-in
Steps:
install a master service in each node
install a slave service for each master in the fan-

in node
or :
cookbook/install_fan_in
69
Continuent 2013
Multiple masters
star topology
Steps:
install a master service in each server
in the hub, install a slave service for each spoke
in each spoke, install a slave service for the hub,

using bypass option
cookbook/install_star
70
Continuent 2013
Taking Over from Standard
Replication
cookbook/install_standard_replicaton
cookbook/takeover
71
Continuent 2013
Replication Management
72
Continuent 2013
Common Commands
replicator
trepctl
thl
the Tungsten service schema

73
Continuent 2013
replicator
Its the service provider
You launch it once when you start
You may restart it when you change con!g

74
Continuent 2013
trepctl
Tungsten Replicator ConTroLler
Its the driving seat for your replication
You can start, update, and stop services
You can get speci!c info

75
Continuent 2013
trepctl
Tungsten Replicator Controller
put services online or o"ine
check status
skip events
inspect internals
change roles
heartbeat
backup/restore
... and a lot more

76
Continuent 2013
thl
Transaction History List
Gives you access to the Tungsten transaction

history logs
77
Continuent 2013
thl
Transaction History Log
info
index
list (total or a speci!c event, or by range)
purge
78
Continuent 2013
Tungsten service schema
one for each service
named "tungsten_SERVICE_NAME"
e.g. tungsten_alpha, tungsten_dragon
Most important table: trep_commit_seqno

79
Continuent 2013
Looking at the tungsten service db
select * from tungsten_dragon.trep_commit_seqno\G
******************* 1. row *******************
task_id: 0
seqno: 102
fragno: 0
last_frag: 1
source_id: qa.r1.continuent.com
epoch_number: 0
eventid: mysql-bin.
000002:0000000000018903;0
applied_latency: 0
update_timestamp: 2012-02-06 05:56:12
shard_id: tungsten_dragon
extract_timestamp: 2012-02-06 05:56:09
80
Continuent 2013
Where are the tools
in the tungsten directory:
$TUNGSTEN_BASE/tungsten/tungsten-replicator/bin
replicator # the daemon
trepctl # replicator controller
thl # transaction history log tool
81
Continuent 2013
Starting and stopping the replicator
cd $TUNGSTEN_BASE/tungsten/tungsten-replicator/bin
./replicator status
Tungsten Replicator Service is running (PID:32400).
./replicator stop
Stopping Tungsten Replicator Service...
Stopped Tungsten Replicator Service.
./replicator start
Starting Tungsten Replicator Service...
.... or ./cookbook/replicator ...
82
Continuent 2013
checking replicator vitals
trepctl services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: -1 # bad sign?
appliedLatency : -1.0
role : slave
serviceName : dragon
serviceType : local
started : true
state : ONLINE
Finished services command...
83
Continuent 2013
sending a heartbeat
trepctl -host $MASTER_HOST heartbeat
trepctl services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: 102
appliedLatency : 3.139
role : slave
serviceType : local
started : true
state : ONLINE
Finished services command...
84
Continuent 2013
replicator status (1)
trepctl status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysql-bin.000002:0000000000018903;0
appliedLastSeqno : 102
clusterName : default
currentEventId : NONE
currentTimeMillis : 1328504342058
dataServerHost : qa.r4.continuent.com
extensions :
latestEpochNumber : 0
masterConnectUri : thl://qa.r1.continuent.com:2112/
masterListenUri : thl://qa.r4.continuent.com:2112/
maximumStoredSeqNo : 102
minimumStoredSeqNo : 0
[...]
85
Continuent 2013
replicator status (2)
[...]
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
resourcePrecedence : 99
rmiPort : 10000
role : slave
seqnoType : java.lang.Long
serviceType : local
simpleServiceName : dragon
siteName : default
sourceId : qa.r4.continuent.com
state : ONLINE
timeInStateSeconds : 245.215
uptimeSeconds : 245.539
Finished status command...
86
Continuent 2013
A failover scenario
1: MySQL native replication
87
Continuent 2013
1. one Master, two slaves
Loading the employees test database

88
Continuent 2013
2. Master goes away
* Stop replication
* Slaves are updated at di"erent levels
# 2
select count(*) from titles
333,145
# 3
443,308
89
Continuent 2013
3. Look into Slave #2 binary logs
!nd the last transaction

90
Continuent 2013
4. Look into Slave #3 binary logs
1. !nd the transaction that was last in slave #2
2. Recognize that last transaction in the log of
slave #3 (This can actually take you a
LOOOONG TIME)
3. Get the position immediately after this
transaction
4. (e.g. 134000 in !le mysql-bin.000018)
91
Continuent 2013
5. promote Slave #3 to master
* in slave #2
CHANGE MASTER TO
master_host=slave_3_IP,
master_user=slavename,
master_password=slavepassword,
master_log_file=mysql-bin.000018,
master_log_pos=134000;
92
Continuent 2013
A failover scenario
1I: Tungsten Replicator
93
Continuent 2013
1. one master, two slaves
loading the employees test database

94
Continuent 2013
2. Master goes away
* Stop replication
* Slaves are updated at di"erent levels
# 2
333,145
# 3
443,308
95
Continuent 2013
3. no need to !nd the last transaction
# simply change roles
trepctl -host slave3 setrole -role master
trepctl -host slave2 setrole \
-role slave -uri thl://slave3
trepctl -host slave3 online
State: ONLINE
trepctl -host slave2 online
State: GOING-ONLINE:SYNCHRONIZING
96
Continuent 2013
4. Check that the slave has
synchronized
# new master
select seqno from tungsten.trep_commit_seqno;
78
# new slave
64
97
Continuent 2013
4. Tell the replicator to hurry up
# new master
trepctl -node slave3 flush
Master log is synchronized with database at log
sequence number: 78
# new slave
trepctl host slave2 wait -applied 78
ONLINE
78
98
Continuent 2013
4. ... and were done
# new master
select count(*) from employees.titles
count(*)
443308
# new slave:
count(*)
443308
99
Continuent 2013
planned role switch
cookbook/install_master_slave
cookbook/switch
100
Continuent 2013
Switching roles in master/slave
replication (1)
101
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
!
online
!
online
!
online
Continuent 2013
replication (2)
102
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
o"ine
"
!
online
!
online
Continuent 2013
replication (3)
103
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
o"ine
"
!
online
!
online
Wait for transactions to be applied
Continuent 2013
replication (4)
104
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
o"ine
"
o"ine
"
o"ine
"
Slaves go ofine
Continuent 2013
replication (5)
105
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
o"ine
"
o"ine
"
o"ine
"
Slave is promoted.
Notice: 2 masters, but
o"ine
Continuent 2013
replication (6)
106
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
o"ine
"
o"ine
"
o"ine
"
old master becomes slave
Continuent 2013
replication (7)
107
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
o"ine
"
o"ine
"
o"ine
"
slaves are directed to new master
Continuent 2013
replication (8)
108
db1
Replicator
db3
Service db1
db2
Replicator
Service db1
Replicator
Service db1
!
online
!
online
!
online
all nodes go online, using new master
Continuent 2013
Tungsten GTID vs MySQL 5.6 GTID
What is GTID
How it works in Tungsten
How it works (or not) in MySQL 5.6

109
Continuent 2013
without global transaction ID
110
slave
master
slave
A
B C
commit
commit
commit
commit
binlog
position
binlog
position
position
binlog
Continuent 2013
with global transaction ID
111
slave
master
slave
A
B C
commit
commit
commit
commit
id#200
id#200
id#200
Continuent 2013
Tungsten and global transaction ID:
activation
(none)
active by default
112
Continuent 2013
status
trepctl status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysql-bin.000002:0000000000001442;0
appliedLastSeqno : 6
clusterName : default
currentEventId : NONE
currentTimeMillis : 1354304680923
dataServerHost : qa.r4.continuent.com
113
Continuent 2013
seeing transactions
thl list -seqno 6
SEQ# = 6 / FRAG# = 0 (last frag)
- TIME = 2012-11-30 20:44:35.0
- EPOCH# = 0
- EVENTID = mysql-bin.000002:0000000000001442;0
- SOURCEID = qa.r1.continuent.com
- SQL(0) = insert into test.v1 values (1, 'inserted
by node #1') /* ___SERVICE___ = [cookbook] */
114
Continuent 2013
changing master connection
trepctl offline
trepctl online -seqno 105
115
Continuent 2013
Tungsten and Global transaction ID:
crash-safe slave tables
mysql -e 'select * from tungsten_cookbook.trep_commit_seqno\G'
*************************** 1. row ***************************
task_id: 0
seqno: 6
fragno: 0
last_frag: 1
source_id: qa.r1.continuent.com
epoch_number: 0
eventid: mysql-bin.000002:0000000000001442;0
applied_latency: 0
update_timestamp: 2012-11-30 20:44:35
shard_id: test
extract_timestamp: 2012-11-30 20:44:35
116
Continuent 2013
Tungsten and Global transaction ID:
crash-safe tables and parallel replication
mysql -e 'select seqno, source_id, shard_id,update_timestamp from
tungsten_cookbook.trep_commit_seqno'
+-------+----------------------+----------+---------------------+
| seqno | source_id | shard_id | update_timestamp |
+-------+----------------------+----------+---------------------+
| 7 | qa.r1.continuent.com | db1 | 2012-11-30 20:54:14 |
+-------+----------------------+----------+---------------------+
117
Continuent 2013
MySQL 5.6 and global transaction ID
activation
mysqld --log-slave-updates \
--gtid-mode=on \
--enforce-gtid-consistency
WARNING: before MySQL 5.6.10, it was
--disable-gtid-unsafe-statements
118
Continuent 2013
seeing transactions
#121203 11:15:49 server id 1 end_log_pos 344 CRC32 0x45b25c8f
GTID [commit=yes]
SET @@SESSION.GTID_NEXT= '7A77A490-3D3A-11E2-8CC9-7DCF9991097B:
2'/*!*/;
# at 344
#121203 11:15:49 server id 1 end_log_pos 423 CRC32 0x873c8fac
Query thread_id=3 exec_time=0 error_code=0
SET TIMESTAMP=1354533349/*!*/;
BEGIN
/*!*/;
# at 423
#121203 11:15:49 server id 1 end_log_pos 522 CRC32 0xb4bf4372
Query thread_id=3 exec_time=0 error_code=0
SET TIMESTAMP=1354533349/*!*/;
insert into t1 values (1)
119
Continuent 2013
status
show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 127.0.0.1
Master_User: rsandbox
Master_Port: 13233
Connect_Retry: 60
Master_Log_File: mysql-bin.000002
Read_Master_Log_Pos: 1837
Relay_Log_File: mysql_sandbox13234-relay-bin.000005
Relay_Log_Pos: 2047
Relay_Master_Log_File: mysql-bin.000002
...
Retrieved_Gtid_Set: 46E13434-3B28-11E2-BF47-6C626DA07446:1-7
Executed_Gtid_Set: 46E13434-3B28-11E2-BF47-6C626DA07446:1-7
120
Continuent 2013
changing master connection
CHANGE MASTER TO master_log_file='mysql-bin-000003',
master_log_pos='1234'
# No global transaction ID is used
121
Continuent 2013
crash-safe slave table
select * from slave_relay_log_info\G
********************* 1. row ********************
Number_of_lines: 7
Relay_log_name: ./mysql_sandbox13234-relay-bin.000005
Relay_log_pos: 2047
Master_log_name: mysql-bin.000002
Master_log_pos: 1837
Sql_delay: 0
Number_of_workers: 5
Id: 1
# NO Global transaction ID is used!
122
Continuent 2013
crash-safe slave table + parallel
select * from mysql.slave_worker_info\G
Id: 12
Relay_log_name: ./mysql_sandbox13234-relay-bin.000007
Relay_log_pos: 4299
Master_log_name: mysql-bin.000002
Master_log_pos: 7155
Checkpoint_relay_log_name: ./mysql_sandbox13234-relay-bin.000007
Checkpoint_relay_log_pos: 1786
Checkpoint_master_log_name: mysql-bin.000002
Checkpoint_master_log_pos: 4642
Checkpoint_seqno: 9
Checkpoint_group_size: 64
Checkpoint_group_bitmap: ?
# NO Global transaction ID is used!
123
Continuent 2013
Filters
124
Continuent 2013
Tungsten Replication Service
125
Extract Filter Apply
Stage
Stage
Stage
Pipeline
Master
DBMS
Transaction
History Log
In-Memory
Queue
Slave
DBMS
Continuent 2013
Restrict replication to some schemas
and tables
126
--master-slave -a \
...
--svc-extractor-filters=replicate \
"--property=replicator.filter.replicate.do=test,*.foo" \
...
--start-and-report
# test="test.*" -> same drawback as binlog-do-db in MySQL
# *.foo = table 'foo' in any database
# employees.dept_codes,employees.salaries => safest way
Continuent 2013
Exclude some schemas and tables
from replication
127
--master-slave -a \
...
--svc-extractor-filters=replicate \
"--property=replicator.filter.replicate.ignore=test,*.foo" \
...
--start-and-report
# test="test.*" -> same drawback as binlog-ignore-db in MySQL
# *.foo = table 'foo' in any database
# employees.dept_codes,employees.salaries => safest way
# DO NOT MIX .do and .ignore!
# (you can do it, but it may not do what you mean)
Continuent 2013
Change name of replicated schema
128
-a --svc-applier-filters=dbtransform \
--property=replicator.filter.dbtransform.from_regex1=stores \
--property=replicator.filter.dbtransform.to_regex1=playground
# from_regex1=stores -> name of the schema in the master
# to_regex1=playground -> name of the schema in the slave
# WARNING: requires "USE schema_name" to work properly.
Continuent 2013
Multi-master:
Con#ict prevention
129
Continuent 2012
CONFLICTS
130
Continuent 2013
What's a con#ict
Data modi!ed by several sources (masters)
Creates one or more :
data loss (unwanted delete)
data inconsistency (unwanted update)
duplicated data (unwanted insert)
replication break
131
Continuent 2013
Data duplication
132
id name amount
1 Joe 100
2 Frank 110
3 Sue 100
alpha
bravo
charlie
4 Matt 130
4 Matt 140
BREAKS
REPLICATION
Continuent 2013
auto_increment o$sets
are not a remedy
A popular recipe
auto_increment_increment +
auto_increment_offset
They don't prevent con#icts
They hide duplicates

133
Continuent 2013
Hidden data duplication
134
id name amount
1 Joe 100
2 Frank 110
3 Sue 100
alpha
o$set 1
bravo
o$set 2
charlie
o$set 3
13 Matt 130
11 Matt 140
INSERT
INSERT
Continuent 2013
Data inconsistency
135
id name amount
1 Joe 100
2 Frank 110
3 Sue 100
alpha
bravo
charlie
3 Sue 105
3 Sue 108
UPDATE
UPDATE
Continuent 2013
Data loss
136
id name amount
1 Joe 100
2 Frank 110
3 Sue 100
alpha
bravo
charlie
record #3
3 Sue 108
MAY BREAK
REPLICATION
UPDATE
DELETE
Continuent 2012
con#ict handling strategies
resolving
after the fact
Needs information that is missing in async replication
avoiding
requires synchronous replication with 2pc
preventing
setting and enforcing a split sources policy
Transforming and resolving
all records are converted to INSERTs
con"icts are resolved within a given time window

137
used by Tungsten
planned for
future use
planned for
future use
Continuent 2013
Multi-master:
Con!ict prevention
138
Continuent 2013
Tungsten con#ict prevention
in a nutshell
1. de!ne the rules
(which master can update which database)
2. tell Tungsten the rules
3. de!ne the policy
(error, drop, warn, or accept)
4. Let Tungsten enforce your rules
139
Continuent 2013
Tungsten Con#ict prevention facts
Sharded by database
De!ned dynamically
Applied on the slave services
methods:
error: make replication fail
drop: drop silently
warn: drop with warning

140
Continuent 2013
Tungsten con#ict prevention
applicability
unknown shards
The schema being updated is not planned
actions: accept, drop, warn, error
unwanted shards
the schema is updated from the wrong master
actions: accept, drop, warn, error
whitelisted shards
can be updated by any master

141
Continuent 2013
Con#ict prevention directives
--svc-extractor-filters=shardfilter
replicator.filter.shardfilter.unknownShardPolicy=error
replicator.filter.shardfilter.unwantedShardPolicy=error
replicator.filter.shardfilter.enforceHomes=false
replicator.filter.shardfilter.allowWhitelisted=false
142
Continuent 2013
con#ict prevention in a star topology
143
Host1
master: alpha
database: employees
Host2
master: bravo
database: buildings
Host3
master: charlie (hub)
database: vehicles
A
B
C B
A
C
C
alpha updates
employees
!
!
Continuent 2013
con#ict prevention in a star topology
144
Host1
master: alpha
database: employees
Host2
master: bravo
database: buildings
Host3
master: charlie (hub)
database: vehicles
A
B
C B
A
C
C
alpha updates
vehicles
!
Continuent 2013
con#ict prevention in a all-masters
topology
145
Host1
master: alpha
database: employees
Host2
master: bravo
database: buildings
Host3
master: charlie
database: vehicles
A
B
C B
A
C
C
A
B
alpha updates
employees
!
!
Continuent 2013
topology
146
Host1
master: alpha
database: employees
Host2
master: bravo
database: buildings
Host3
master: charlie
database: vehicles
A
B
C B
A
C
C
A
B
charlie updates
vehicles
!
!
Continuent 2013
topology
147
Host1
master: alpha
database: employees
Host2
master: bravo
database: buildings
Host3
master: charlie
database: vehicles
A
B
C B
A
C
C
A
B
bravo updates
employees
!
!
Continuent 2013
topology
148
Host1
master: alpha
database: employees
Host2
master: bravo
database: buildings
Host3
master: charlie
database: vehicles
A
B
C B
A
C
C
A
B
charlie updates
employees
!
!
Continuent 2013
setting con#ict prevention rules
trepctl -host host1 -service charlie \
shard -insert < shards.map
cat shards.map
shard_id master critical
personnel alpha false
buildings bravo false
vehicles charlie false
test whitelisted false
# charlie is slave service in host 1
149
Continuent 2013
trepctl -host host2 -service charlie \
cat shards.map
# charlie is slave service in host 2
150
Continuent 2013
trepctl -host host3 -service alpha \
trepctl -host host3 -service bravo \
cat shards.map
# alpha and bravo are slave services in host 3
151
Continuent 2013
Con#ict prevention demo
152
reminder
Server #1 can update "employees"
Server #2 can update "buildings"
Server #3 can update "vehicles"

Continuent 2013
Sample correct operation (1)
mysql #1> create table employees.names( ... )
# all servers receive the table
# all servers keep working well
153
Continuent 2013
Sample correct operation (2)
mysql #2> create table buildings.homes( ... )
# all servers receive the table
# all servers keep working well
154
Continuent 2013
Sample incorrect operation (1)
mysql #2> create table employees.nicknames( ... )
# Only server #2 receives the table
# slave service in hub gets an error
# slave service in #1 does not receive anything
155
Continuent 2013
sample incorrect operation (2)
#3 $ trepct services | simple_services
alpha [slave]
seqno: 7 - latency: 0.136 - ONLINE
bravo [slave]
seqno: -1 - latency: -1.000 - OFFLINE:ERROR
charlie [master]
seqno: 66 - latency: 0.440 - ONLINE
156
Continuent 2013
sample incorrect operation (3)
#3 $ trepct -service bravo status
NAME VALUE
---- -----
appliedLastEventId : NONE
appliedLastSeqno : -1
appliedLatency : -1.0
(...)
offlineRequests : NONE
pendingError : Stage task failed: q-to-dbms
pendingErrorCode : NONE
pendingErrorEventId : mysql-bin.000002:0000000000001241;0
pendingErrorSeqno : 7
pendingExceptionMessage: Rejected event from wrong shard:
seqno=7 shard ID=employees shard master=alpha service=bravo
(...)
157
Continuent 2013
Fixing the issue
mysql #1> drop table if exists employees.nicknames;
mysql #1> create table if exists employees.nicknames ( ... ) ;
#3 $ trepct -service bravo online -skip-seqno 7
# all servers receive the new table
158
Continuent 2013
Sample whitelisted operation
mysql #2> create table test.hope4best( ... )
mysql #1> insert into test.hope4best values ( ... )
# REMEMBER: 'test' was explicitly whitelisted
# All servers get the new table and records
# But there is no protection against conflicts
159
Continuent 2013
administration
160
Continuent 2013
Viewing THL Events
thl info
log directory = /home/tungsten/installs/master_slave/thl/dragon/
min seq# = 0
max seq# = 101
events = 101
161
Continuent 2013
viewing THL events
thl index
LogIndexEntry thl.data.0000000001(0:102)
162
Continuent 2013
viewing THL events
thl index
[...]
163
Continuent 2013
viewing THL events
thl list -seqno 102
[...]
SEQ# = 102 / FRAG# = 0 (last frag)
- TIME = 2012-02-06 05:56:09.0
- EPOCH# = 0
- EVENTID = mysql-bin.000002:0000000000018903;0
- SOURCEID = qa.r1.continuent.com
- METADATA =
[mysql_server_id=10;is_metadata=true;service=dragon;shard=tung
sten_dragon;heartbeat=NONE]
- TYPE = com.continuent.tungsten.replicator.event.ReplDBMSEvent
- OPTIONS = [##charset = ISO8859_1, autocommit = 1, sql_auto_is_null = 1, foreign_key_checks = 1,
unique_checks = 1, sql_mode = 'IGNORE_SPACE', character_set_client = 8, collation_connection = 8,
collation_server = 8]
- SCHEMA = tungsten_dragon
- SQL(0) = UPDATE tungsten_dragon.heartbeat SET source_tstamp=
"2012-02-06 05:56:09", salt= 2, name= "NONE" WHERE id= 1 /*
___SERVICE___ = [dragon] */
164
Continuent 2013
Skipping a THL Event
trepctl online -skip-seqno 1092
trepctl online -skip-seqno 1092,1093,1094
# see example
165
Continuent 2013
Adding a Member
Let's see the cookbook, and use it

166
Continuent 2013
parallel replication
167
Replicator Pipeline Architecture
THL
Slave
DBMS
Transaction
History Log
MySQL
Binlog
shard.list
le
Apply Extract Extract
Pipeline
Tungsten Replicator Process
Stage
Apply Extract
Apply Extract
Apply Extract
Parallel
Queue
Assign
Shard
ID
Apply
Stage
Stage
channels
Continuent 2013
Parallel replication facts
!
Sharded by database
!
Good choice for slave lag problems
"
Bad choice for single database projects
169
Parallel Replication test
binary logs
MySQL slave
Tungsten slave
OFFLINE
STOPPED
replicator alpha
direct:
alpha
(slave)
Concurrent sysbench
on 30 databases
running for 1 hour
TOTAL DATA: 130 GB
RAM per server: 20GB
Slaves will have 1 hour lag
measuring results
binary logs
MySQL slave
Tungsten slave
ONLINE
START
replicator alpha
direct:
alpha
(slave)
Recording
catch-up time
MySQL native
replication
slave catch up in 04:29:30
Tungsten parallel
replication
slave catch up in 00:55:40
Parallel replication made simpler
FROM HERE ....
TO HERE
Continuent 2013
parallel replication
direct slave facts
!
No need to install Tungsten on the master
!
Tungsten runs only on the slave
!
Replication can revert to native slave with two
commands (trepctl offline; start slave)
!
Native replication can continue on other slaves
"
Failover (either native or Tungsten) becomes a manual
task
177
Continuent 2013
installing parallel replication
MORE_OPTIONS='--channels=10'
178
Continuent 2013
Checking parallel replication
trepctl status
trepctl status -name tasks
trepctl status -name shards
trepctl status -name stores
179
Continuent 2013
Parallel replication demo
180
Continuent 2013
Troubleshooting
181
Continuent 2013
Identify the Failed Component
Steps
1. trepctl services
2. trepctl -service SVC_NAME status
3. look at the logs
4. Take action
182
Continuent 2013
reading the logs
ls $TUNGSTEN_BASE/tungsten/tungsten-replicator/logs/
trepsvc.log user.log
...
or
./cookbook/show_log
# let's see it in practice
183
Continuent 2013
Parting thoughts
184
Continuent 2013
Open source Tungsten
Replicator now includes
Oracle-to-MySQL and
Oracle-to-Oracle extractors
and appliers!
185
Continuent 2012 186
Continuent Website:
http://www.continuent.com
Tungsten Replicator 2.0:
http://code.google.com/p/tungsten-replicator
Our Blogs:
http://scale-out-blog.blogspot.com
http://datacharmer.blogspot.com
http://yingclusters.blogspot.com
560 S. Winchester Blvd., Suite 500
San Jose, CA 95128
Tel +1 (866) 998-3642
Fax +1 (408) 668-1009
e-mail: sales@continuent.com

Tungsten Replicator Tutorial Uc 2013

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Tungsten Replicator Tutorial Uc 2013

Încărcat de

Drepturi de autor:

Formate disponibile

Continuent 2013

Using Tungsten Replicator

Continuent Tungsten Deployment and Support

20 years development and DB experience

Giuseppe Maxia, a.k.a. "The Data Charmer"

QA Director, Continuent, Inc

25 years development and DB experience

long timer MySQL community member. Oracle

... and more

All the above (and more) are available with a

Stand-alone or taking over

Deploying from a single location

(Using the cookbook recipes, you hardly see

Check the requirements

Get the binaries

Expand the tarball

Java JRE or JDK (Sun/Oracle or Open-jdk)

Ruby 1.8 (only during installation)

ssh access to the same user in all nodes

MySQL user with all privileges

Copy zip !les from USB Key

Expand on local disk

Start all 4 Machines in VirtualBox

Running centos 6.3 and Percona 5.5

Root and tungsten password = password

localhost port 2222 redirects to 22 on hosts

Where you unpack the software and run the

There is generally only one, in one host;

Can be discarded after installation

Where your installed software will go;

There is one for every host;

step by step: how it happened

Scripts without extension are designed to be

Scripts with extension ".sh" are either for

./cookbook/install_* scripts can be used

After installation there is a !le

cookbook scripts can be used either from the

Method to simulate installation;

Does NOT perform installation;

Does NOT even do validation;

It only shows the commands used to install;

Allows you to get the commands and do an

remove two nodes

edit the variables in cookbook/

install a master service in each node

install a slave service for each master in the fan-

install a master service in each server

in the hub, install a slave service for each spoke

in each spoke, install a slave service for the hub,

the Tungsten service schema

Its the service provider

You launch it once when you start

You may restart it when you change con!g

Tungsten Replicator ConTroLler

Its the driving seat for your replication

You can start, update, and stop services

You can get speci!c info

Tungsten Replicator Controller

put services online or o"ine

... and a lot more

Transaction History List

Gives you access to the Tungsten transaction

Transaction History Log

list (total or a speci!c event, or by range)