MySQL 5.7 基于复制线程SQL_Thread加快恢复的尝试

1. MySQL 数据恢复常用办法

MySQL恢复的方法一般有三种：

1. 官方推荐的基于全备+binlog ，通常做法是先恢复最近一次的全备，然后通过mysqlbiinlog --start-position --stop-position binlog.000xxx | mysql -uroot -p xxx -S database 恢复到目标数据库做恢复

2. 基于主从同步恢复数据，通常做法是先恢复最近一次的全备，然后恢复后的实例做slave 挂载到现有的master 上面，通过 start slave sql_thread until master_log_pos 恢复到故障前的一个pos。

现在尝试第三种恢复方式，通过原来主库上面的binlog 把数据都恢复到slave 上。

处理思路：

因为relaylog和binlog本质实际上是一样的，所以是否可以利用MySQL自身的sql_thread来增量binlog

1）重新初始化一个实例，恢复全量备份文件。
2）找到第一个binlog文件的position，和剩下所有的binlog。
3）将binlog伪装成relaylog，通过sql thread增量恢复。

应用场景：

1. 最近的一次全备离故障位置比较远，通过上面两种方式的恢复时间太慢

2. 双主keepalived的集群，由于keepalived没有像MHA 那样有日志补全机制，出故障是有可能会有数据丢失的，万一同步有严重的复制延时出现故障切换到slave，这样数据就不一致，需要做日志补全

2. 实验步骤

1. 建立基于主从同步（这里实验基于传统的pos，其实GTID 也一样可行）

M1 ：

root@localhost:mysql3307.sock [(none)]>select * from restore.t1;
+----+------+
| id | c1   |
+----+------+
|  1 | 1    |
|  2 | 3    |
|  3 | 2    |
|  4 | 3    |
|  5 | 6    |
|  6 | 7    |
|  7 | 9    |
| 10 | NULL |
| 11 | 10   |
+----+------+
9 rows in set (0.00 sec)

　M2：（slave）

root@localhost:mysql3307.sock [(none)]>select * from restore.t1;
+----+------+
| id | c1   |
+----+------+
|  1 | 1    |
|  2 | 3    |
|  3 | 2    |
|  4 | 3    |
|  5 | 6    |
|  6 | 7    |
|  7 | 9    |
| 10 | NULL |
| 11 | 10   |
+----+------+
9 rows in set (0.00 sec)

root@localhost:mysql3307.sock [restore]>show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: m1
                  Master_User: repl
                  Master_Port: 3307
                Connect_Retry: 60
              Master_Log_File: 3307-binlog.000002
          Read_Master_Log_Pos: 154
               Relay_Log_File: M2-relay-bin.000004
                Relay_Log_Pos: 371
        Relay_Master_Log_File: 3307-binlog.000002
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes	
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 154
              Relay_Log_Space: 624
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 13307
                  Master_UUID: afeab8d6-b871-11e7-9b2a-005056b643b3
             Master_Info_File: /data/mysql/3307/data/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)

　记录此时slave 的 relay-log 信息

[root@M2 data]# more M2-relay-bin.index
./M2-relay-bin.000003
./M2-relay-bin.000004
 
[root@M2 data]# more relay-log.info
7
./M2-relay-bin.000004
371
3307-binlog.000002
154
0
0
1

　2. 使用sysbench 模拟数据不同步

[root@M1 logs]# mysqladmin create sbtest

[root@M1 sysbench]# sysbench --db-driver=mysql --mysql-host=m1 --mysql-port=3307 --mysql-user=sbtest --mysql-password='sbtest' /usr/share/sysbench/oltp_common.lua --tables=4 --table-size=100000 --threads=2 --time=60 --report-interval=10 prepare

　　在主库导入数据的时候在slave端停止同步，制造数据不一致

root@localhost:mysql3307.sock [mysql]>stop slave

　3. 等sysbench执行完，查看主库的数据和slave 的数据

主库：

root@localhost:mysql3307.sock [sbtest]>select count(1) from sbtest1;
+----------+
| count(1) |
+----------+
|   100000 |
+----------+
1 row in set (0.05 sec)
 
root@localhost:mysql3307.sock [sbtest]>select count(1) from sbtest2;
+----------+
| count(1) |
+----------+
|   100000 |
+----------+
1 row in set (0.05 sec)
 
root@localhost:mysql3307.sock [sbtest]>select count(1) from sbtest3;
+----------+
| count(1) |
+----------+
|   100000 |
+----------+
1 row in set (0.05 sec)
 
root@localhost:mysql3307.sock [sbtest]>select count(1) from sbtest4;
+----------+
| count(1) |
+----------+
|   100000 |
+----------+
1 row in set (0.05 sec)

　　slave 端：

root@localhost:mysql3307.sock [sbtest]>select count(1) from sbtest4;
+----------+
| count(1) |
+----------+
|    67550 |
+----------+
1 row in set (0.06 sec)
 
root@localhost:mysql3307.sock [sbtest]>select count(1) from sbtest3;
+----------+
| count(1) |
+----------+
|    70252 |
+----------+
1 row in set (0.04 sec)

　　可以看到主从不同步。

4. 此时查看slave 的status：

root@localhost:mysql3307.sock [(none)]>show slave status\G
*************************** 1. row ***************************
               Slave_IO_State:
                  Master_Host: m1
                  Master_User: repl
                  Master_Port: 3307
                Connect_Retry: 60
              Master_Log_File: 3307-binlog.000002
          Read_Master_Log_Pos: 76364214
               Relay_Log_File: M2-relay-bin.000004
                Relay_Log_Pos: 64490301
        Relay_Master_Log_File: 3307-binlog.000002
             Slave_IO_Running: No
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 64490084
              Relay_Log_Space: 76364861
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 0
                  Master_UUID: afeab8d6-b871-11e7-9b2a-005056b643b3
             Master_Info_File: /data/mysql/3307/data/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State:
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)

　由于本地的relay log 没有执行完毕，为了保证实验准确性，我们先让本地的relaylog 执行完 , start slave sql_thread

再次检查：

*************************** 1. row ***************************
               Slave_IO_State:
                  Master_Host: m1
                  Master_User: repl
                  Master_Port: 3307
                Connect_Retry: 60
              Master_Log_File: 3307-binlog.000002
          Read_Master_Log_Pos: 76364214
               Relay_Log_File: M2-relay-bin.000005
                Relay_Log_Pos: 4
        Relay_Master_Log_File: 3307-binlog.000002
             Slave_IO_Running: No
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 76364214
              Relay_Log_Space: 154
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 0
                  Master_UUID: afeab8d6-b871-11e7-9b2a-005056b643b3
             Master_Info_File: /data/mysql/3307/data/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)

　　本地relaylog 已经全部执行完毕，此时记录最新的relay log 信息：

[root@M2 data]# more relay-log.info
7
./M2-relay-bin.000005
4
3307-binlog.000002
76364214
0
0
1
 
0
0
1

　　上面这个信息很重要，说明了从库执行到主库的000002 的binlog的76364214 这个位置，我们下面将主库的binlog 拷贝过来模拟relaylog，并从这个位置开始恢复

5. 拷贝binlog 到目标端，并模拟成relay log

拷贝前先关闭从库，并修改cnf (skip-slave-start)让slave 不会重启后自动开始复制

[root@M2 data]# ll
total 185248
-rw-r----- 1 root  root       461 Oct 24 17:14 3307-binlog.000001
-rw-r----- 1 root  root  76364609 Oct 24 17:14 3307-binlog.000002
-rw-r----- 1 root  root       203 Oct 24 17:14 3307-binlog.000003
-rw-r----- 1 root  root       419 Oct 24 17:14 3307-binlog.000004
-rw-r----- 1 root  root       164 Oct 24 17:14 3307-binlog.index
-rw-r----- 1 mysql mysql       56 Oct 24 15:08 auto.cnf
-rw-r----- 1 mysql mysql     4720 Oct 24 17:14 ib_buffer_pool
-rw-r----- 1 mysql mysql 12582912 Oct 24 17:14 ibdata1
-rw-r----- 1 mysql mysql 50331648 Oct 24 17:14 ib_logfile0
-rw-r----- 1 mysql mysql 50331648 Oct 24 17:11 ib_logfile1
-rw-r----- 1 mysql mysql      177 Oct 24 17:14 M2-relay-bin.000005
-rw-r----- 1 mysql mysql       22 Oct 24 17:11 M2-relay-bin.index
-rw-r----- 1 mysql mysql      122 Oct 24 17:14 master.info
drwxr-x--- 2 mysql mysql     4096 Oct 24 15:07 mysql
-rw------- 1 root  root         0 Oct 24 15:08 nohup.out
drwxr-x--- 2 mysql mysql     4096 Oct 24 15:07 performance_schema
-rw-r----- 1 mysql mysql       68 Oct 24 17:14 relay-log.info
drwxr-x--- 2 mysql mysql     4096 Oct 24 15:07 restore
drwxr-x--- 2 mysql mysql     4096 Oct 24 16:47 sbtest
drwxr-x--- 2 mysql mysql    12288 Oct 24 15:07 sys
-rw-r----- 1 mysql mysql       24 Oct 24 15:07 xtrabackup_binlog_pos_innodb
-rw-r----- 1 mysql mysql      577 Oct 24 15:07 xtrabackup_info

　改名为relay log

[root@M2 data]# cp 3307-binlog.000001 relay.000001
[root@M2 data]# cp 3307-binlog.000002 relay.000002
[root@M2 data]# cp 3307-binlog.000003 relay.000003
[root@M2 data]# cp 3307-binlog.000004 relay.000004

改权限属性

[root@M2 data]# chown mysql.mysql -R *

　修改relay log index 文件，让系统能识别

[root@M2 data]# cat M2-relay-bin.index
	./relay.000001
	./relay.000002
	./relay.000003
	./relay.000004

　修改relay log info 文件，告诉系统从哪个位置开始复制

[root@M2 data]# cat relay-log.info
	7
	./relay.000002
	76364214
	3307-binlog.000002
	
	0
	0
	1
 
	0
	0
	1

　最后开起sql_thread 进程开始快速恢复

start slave sql_thread

　6. 检查数据是否一致

slave:

oot@localhost:mysql3307.sock [sbtest]>select count(1) from sbtest4;
+----------+
| count(1) |
+----------+
|   100000 |
+----------+
1 row in set (0.05 sec)
 
root@localhost:mysql3307.sock [sbtest]>select count(1) from sbtest3;
+----------+
| count(1) |
+----------+
|   100000 |
+----------+
1 row in set (0.05 sec)

　可以看到slave 已经把缺失的数据都全部恢复了。