mysql主从不同步处理过程分享

背景  8月7日15：58收到报障数据库出现不同步：数据库共四台，分别为10.255.70.11，10.255.70.12，10.255.70.13，10.255.70.14（ip为虚拟ip）

数据库结构为：

故障时不同步现为：（1）70.11和70.13之间主主不同步 ，（2）70.11和70.12之间主从不同步，（3）70.11和70.14之间主从是同步的

（1）由于my.cnf文件中有slave-skip-errors=all配置，所以在出现不同步错误时跳过，检查同步参数Slave_IO_Running: Yes/Slave_SQL_Running: Yes均为yes，实际数据是不同步的

（2）70.11和70.12之间不同步，同步参数为Slave_IO_Running: NO/Slave_SQL_Running: Yes，报错1062，截图如下：

在70.12上操作如下：

mysql>stop slave ;

mysql>SET GLOBAL SQL_SLAVE_SKIP_COUNTER =  

mysql>slave start

查看数据同步参数：发现已经同步

至此70.11和70.12的主从不同步问题已经解决，需要注意的是：跳过1062这个事务之后，虽然以后主从数据库是同步的，但是不同步期间的数据将不再重新同步，即：不同步期间从库的数据不完整

现在只存在70.11和70.13之间的主主数据库不同步，在此期间发现70.11的库db1数据丢失了，因为70.11和70.12，70.14的数据是同步的，所以这三个库全部数据丢失

解决办法思路如下：

（1）先将现有业务切至70.13上，将危害减少到最小

（2）首先恢复70.11的数据：查看数据库最近一次备份是当天凌晨3点半，3点半到下午15：58的数据通过binlog日志进行恢复

（3）备份70.11的数据，导入70.12/13/14的数据库，并作主主/主从同步

具体操作：

恢复8月7日凌晨3点半之前的数据：

[root@]# gunzip db1_20190807.sql.gz
[root@]# mysql -h 10.255.70.11 -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is
Server version: 5.7.-log MySQL Community Server (GPL)

Copyright (c) , , Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
|db1 |
+--------------------+
11 rows in set (0.00 sec)

mysql> use db1;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed, 1 warning
mysql> source db1.sql.gz;

通过binlog日志恢复凌晨3：30~~16：30之间的数据：

（1）将时间段内的binlog日志转化为mysql可识别的语句

[root@]# mysqlbinlog  --no-defaults  --database=db1 --start-datetime='2019-08-07 03:30:00' --stop-datetime='2019-08-17 16:30:00' mysql-bin. >temp20190807.sql
[root@]# ll temp20190807.sql
-rw-r--r--  root root  8月    : temp20190807.sql

（2）删除binlog日志中的drop语句

[root@]# sed -i -e '/DROP/d' temp20190807.sql

（3）数据恢复

[root@]# mysql -h 10.255.70.11 -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is
Server version: 5.7.-log MySQL Community Server (GPL)

Copyright (c) , , Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| db1 |
+--------------------+
 rows in set (0.00 sec)

mysql> use db1;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed,  warning
mysql> source temp20190807.sql;

至此：70.11的数据恢复完全

接下来进行70.12和13，14的数据同步：

（1）备份70.11的数据库

[root@ ~]# mysqldump -h 10.255.70.11 -uroot  -p  -R ottdb1 | gzip > /tmp/db120190807.sql.gz

（2）将70.11的备份数据copy给70.12/13/14

[root@ tmp]# scp db120190807.sql.gz 10.255.70.12:/tmp/
The authenticity of host '10.255.70.12 (10.255.70.12)' can't be established.
RSA key fingerprint is aa:::d3:aa:a8:af:a3:a9:c9:6e::6b::7f:b3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.255.70.12' (RSA) to the list of known hosts.
root@10.255.70.12's password:
ottdb120190807.sql.gz        

[root@ tmp]# scp db120190807.sql.gz 10.255.70.13:/tmp/
The authenticity of host '10.255.70.13 (10.255.70.13)' can't be established.
RSA key fingerprint is aa:::d3:aa:a8:af:a3:a9:c9:6e::6b::7f:b3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.255.70.13' (RSA) to the list of known hosts.
root@10.255.70.13's password:
ottdb120190807.sql.gz   

[root@ tmp]# scp db120190807.sql.gz 10.255.70.14:/tmp/
The authenticity of host '10.255.70.14 (10.255.70.14)' can't be established.
RSA key fingerprint is aa:::d3:aa:a8:af:a3:a9:c9:6e::6b::7f:b3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.255.70.14' (RSA) to the list of known hosts.
root@10.255.70.14's password:
ottdb120190807.sql.gz                              

（3）恢复70.12/13/14数据，以70.12为例，13和14也进行如下操作：

[root@]# mysql -h 10.255.70.12 -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is
Server version: 5.7.-log MySQL Community Server (GPL)

Copyright (c) , , Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| db1 |
+--------------------+
 rows in set (0.00 sec)

mysql> use db1;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed,  warning
mysql> source db120180807.sql;

（4）记录70.11和70.13的file和position

[root@]# mysql -h 10.255.70.11 -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is
Server version: 5.7.-log MySQL Community Server (GPL)

Copyright (c) , , Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show master status;
+------------------------+-----------+--------------+------------------+-------------------+
| File                   | Position  | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------------+-----------+--------------+------------------+-------------------+
| mysql-bin.          |    db1       | mysql            |                   |
+------------------------+-----------+--------------+------------------+-------------------+
 row in set (0.00 sec)

mysql>

[root@]# mysql -h 10.255.70.13 -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is
Server version: 5.7.-log MySQL Community Server (GPL)

Copyright (c) , , Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show master status;
+------------------------+-----------+--------------+------------------+-------------------+
| File                   | Position  | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------------+-----------+--------------+------------------+-------------------+
| mysql-bin.          |    db1       | mysql            |                   |
+------------------------+-----------+--------------+------------------+-------------------+
 row in set (0.00 sec)

mysql>

（5）实现70.11和70.13之间的主主同步，具体步骤如下：

在70.11上执行：

mysql> change master to master_host='10.255.70.13',master_user='root',master_password='123456',master_log_file='mysql-bin.000055',master_log_pos=622032567;

在70.13上执行：

mysql> change master to master_host='10.255.70.11',master_user='root',master_password='',master_log_file='mysql-bin.000048',master_log_pos=;

数据库同步完成，查看数据库同步情况

[root@]# mysql -h 10.255.70.11 -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is
Server version: 5.7.-log MySQL Community Server (GPL)

Copyright (c) , , Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show slave status\G;
*************************** . row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.255.70.13
                  Master_User: root
                  Master_Port:
                Connect_Retry:
              Master_Log_File: mysql-bin.
          Read_Master_Log_Pos:
               Relay_Log_File: mysq-relay-bin.
                Relay_Log_Pos:
        Relay_Master_Log_File: mysql-bin.
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: db1
          Replicate_Ignore_DB: mysql

接下来进行70.11和70.12和14的主从同步：

在70.12和70.14上执行：

mysql> change master to master_host='10.255.70.11',master_user='root',master_password='',master_log_file='mysql-bin.000048',master_log_pos=;

查看数据库同步情况：

[root@]# mysql -h 10.255.70.12 -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is
Server version: 5.7.-log MySQL Community Server (GPL)

Copyright (c) , , Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show slave status\G;
*************************** . row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.255.70.11
                  Master_User: root
                  Master_Port:
                Connect_Retry:
              Master_Log_File: mysql-bin.
          Read_Master_Log_Pos:
               Relay_Log_File: mysq-relay-bin.
                Relay_Log_Pos:
        Relay_Master_Log_File: mysql-bin.
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: db1
          Replicate_Ignore_DB: mysql

数据库恢复完成，修改业务地址到VIP地址，此次故障解决

数据同步过程中报错处理：

Last_IO_Errno: 1236
                Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file'
处理方式：

flush logs;
show master status;

记下File, Position

重新进行同步

Got fatal error 1236 from master when reading data from binary log