MySQL基于ROW格式的数据恢复
大家都知道MySQL Binlog 有三种格式,分别是Statement、Row、Mixd。Statement记录了用户执行的原始SQL,而Row则是记录了行的修改情况,在MySQL 5.6以上的版本默认是Mixd格式,但为了保证复制数据的完整性,建议生产环境都使用Row格式,就前面所说的Row记录的是行数据的修改情况,而不是原始SQL。那么线上或者测试环境误操删除或者更新几条数据后,又想恢复,那怎么办呢?下面演示基于Binlog格式为Row的误操后数据恢复,那么怎么把Binlog解析出来生成反向的原始SQL呢?下面我们一起来学习。
下面我们使用 binlog-rollback.pl 对数据进行恢复演示。(这脚本的作者不知道是谁,Github上也没找到这个脚本,所以无法标明出处),脚本是用Perl语言写的,非常好用的一个脚本,当然你也可以用Shell或者Python脚本来实现,下面是脚本的代码:
#!/usr/lib/perl -w use strict;
use warnings; use Class::Struct;
use Getopt::Long qw(:config no_ignore_case); # GetOption
# register handler system signals
use sigtrap 'handler', \&sig_int, 'normal-signals'; # catch signal
sub sig_int(){
my ($signals) = @_;
print STDERR "# Caught SIG$signals.\n";
exit ;
} my %opt;
my $srcfile;
my $host = '127.0.0.1';
my $port = ;
my ($user,$pwd);
my ($MYSQL, $MYSQLBINLOG, $ROLLBACK_DML);
my $outfile = '/dev/null';
my (%do_dbs,%do_tbs); # tbname=>tbcol, tbcol: @n=>colname,type
my %tbcol_pos; my $SPLITER_COL = ',';
my $SQLTYPE_IST = 'INSERT';
my $SQLTYPE_UPD = 'UPDATE';
my $SQLTYPE_DEL = 'DELETE';
my $SQLAREA_WHERE = 'WHERE';
my $SQLAREA_SET = 'SET'; my $PRE_FUNCT = '========================== '; # =========================================================
# 基于row模式的binlog,生成DML(insert/update/delete)的rollback语句
# 通过mysqlbinlog -v 解析binlog生成可读的sql文件
# 提取需要处理的有效sql
# "### "开头的行.如果输入的start-position位于某个event group中间,则会导致"无法识别event"错误
#
# 将INSERT/UPDATE/DELETE 的sql反转,并且1个完整sql只能占1行
# INSERT: INSERT INTO => DELETE FROM, SET => WHERE
# UPDATE: WHERE => SET, SET => WHERE
# DELETE: DELETE FROM => INSERT INTO, WHERE => SET
# 用列名替换位置@{1,2,3}
# 通过desc table获得列顺序及对应的列名
# 特殊列类型value做特别处理
# 逆序
#
# 注意:
# 表结构与现在的表结构必须相同[谨记]
# 由于row模式是幂等的,并且恢复是一次性,所以只提取sql,不提取BEGIN/COMMIT
# 只能对INSERT/UPDATE/DELETE进行处理
# ========================================================
sub main{ # get input option
&get_options(); #
&init_tbcol(); #
&do_binlog_rollback();
} &main(); # ----------------------------------------------------------------------------------------
# Func : get options and set option flag
# ----------------------------------------------------------------------------------------
sub get_options{
#Get options info
GetOptions(\%opt,
'help', # OUT : print help info
'f|srcfile=s', # IN : binlog file
'o|outfile=s', # out : output sql file
'h|host=s', # IN : host
'u|user=s', # IN : user
'p|password=s', # IN : password
'P|port=i', # IN : port
'start-datetime=s', # IN : start datetime
'stop-datetime=s', # IN : stop datetime
'start-position=i', # IN : start position
'stop-position=i', # IN : stop position
'd|database=s', # IN : database, split comma
'T|table=s', # IN : table, split comma
'i|ignore', # IN : ignore binlog check ddl and so on
'debug', # IN : print debug information
) or print_usage(); if (!scalar(%opt)) {
&print_usage();
} # Handle for options
if ($opt{'f'}){
$srcfile = $opt{'f'};
}else{
&merror("please input binlog file");
} $opt{'h'} and $host = $opt{'h'};
$opt{'u'} and $user = $opt{'u'};
$opt{'p'} and $pwd = $opt{'p'};
$opt{'P'} and $port = $opt{'P'};
if ($opt{'o'}) {
$outfile = $opt{'o'};
# 清空 outfile
`echo '' > $outfile`;
} #
$MYSQL = qq{mysql -h$host -u$user -p'$pwd' -P$port};
&mdebug("get_options::MYSQL\n\t$MYSQL"); # 提取binlog,不需要显示列定义信息,用-v,而不用-vv
$MYSQLBINLOG = qq{mysqlbinlog -v};
$MYSQLBINLOG .= " --start-position=".$opt{'start-position'} if $opt{'start-position'};
$MYSQLBINLOG .= " --stop-position=".$opt{'stop-position'} if $opt{'stop-postion'};
$MYSQLBINLOG .= " --start-datetime='".$opt{'start-datetime'}."'" if $opt{'start-datetime'};
$MYSQLBINLOG .= " --stop-datetime='$opt{'stop-datetime'}'" if $opt{'stop-datetime'};
$MYSQLBINLOG .= " $srcfile";
&mdebug("get_options::MYSQLBINLOG\n\t$MYSQLBINLOG"); # 检查binlog中是否含有 ddl sql: CREATE|ALTER|DROP|RENAME
&check_binlog() unless ($opt{'i'}); # 不使用mysqlbinlog过滤,USE dbname;方式可能会漏掉某些sql,所以不在mysqlbinlog过滤
# 指定数据库
if ($opt{'d'}){
my @dbs = split(/,/,$opt{'d'});
foreach my $db (@dbs){
$do_dbs{$db}=;
}
} # 指定表
if ($opt{'T'}){
my @tbs = split(/,/,$opt{'T'});
foreach my $tb (@tbs){
$do_tbs{$tb}=;
}
} # 提取有效DML SQL
$ROLLBACK_DML = $MYSQLBINLOG." | grep '^### '";
# 去掉注释: '### ' -> ''
# 删除首尾空格
$ROLLBACK_DML .= " | sed 's/###\\s*//g;s/\\s*\$//g'";
&mdebug("rollback dml\n\t$ROLLBACK_DML"); # 检查内容是否为空
my $cmd = "$ROLLBACK_DML | wc -l";
&mdebug("check contain dml sql\n\t$cmd");
my $size = `$cmd`;
chomp($size);
unless ($size >){
&merror("binlog DML is empty:$ROLLBACK_DML");
}; } # ----------------------------------------------------------------------------------------
# Func : check binlog contain DDL
# ----------------------------------------------------------------------------------------
sub check_binlog{
&mdebug("$PRE_FUNCT check_binlog");
my $cmd = "$MYSQLBINLOG ";
$cmd .= " | grep -E -i '^(CREATE|ALTER|DROP|RENAME)' ";
&mdebug("check binlog has DDL cmd\n\t$cmd");
my $ddlcnt = `$cmd`;
chomp($ddlcnt); my $ddlnum = `$cmd | wc -l`;
chomp($ddlnum);
my $res = ;
if ($ddlnum>){
# 在ddl sql前面加上前缀<DDL>
$ddlcnt = `echo '$ddlcnt' | sed 's/^/<DDL>/g'`;
&merror("binlog contain $ddlnum DDL:$MYSQLBINLOG. ddl sql:\n$ddlcnt");
} return $res;
} # ----------------------------------------------------------------------------------------
# Func : init all table column order
# if input --database --table params, only get set table column order
# ----------------------------------------------------------------------------------------
sub init_tbcol{
&mdebug("$PRE_FUNCT init_tbcol");
# 提取DML语句
my $cmd .= "$ROLLBACK_DML | grep -E '^(INSERT|UPDATE|DELETE)'";
# 提取表名,并去重
#$cmd .= " | awk '{if (\$1 ~ \"^UPDATE\") {print \$2}else {print \$3}}' | uniq ";
$cmd .= " | awk '{if (\$1 ~ \"^UPDATE\") {print \$2}else {print \$3}}' | sort | uniq ";
&mdebug("get table name cmd\n\t$cmd");
open ALLTABLE, "$cmd | " or die "can't open file:$cmd\n"; while (my $tbname = <ALLTABLE>){
chomp($tbname);
#if (exists $tbcol_pos{$tbname}){
# next;
#}
&init_one_tbcol($tbname) unless (&ignore_tb($tbname)); }
close ALLTABLE or die "can't close file:$cmd\n"; # init tb col
foreach my $tb (keys %tbcol_pos){
&mdebug("tbname->$tb");
my %colpos = %{$tbcol_pos{$tb}};
foreach my $pos (keys %colpos){
my $col = $colpos{$pos};
my ($cname,$ctype) = split(/$SPLITER_COL/, $col);
&mdebug("\tpos->$pos,cname->$cname,ctype->$ctype");
}
}
}; # ----------------------------------------------------------------------------------------
# Func : init one table column order
# ----------------------------------------------------------------------------------------
sub init_one_tbcol{
my $tbname = shift;
&mdebug("$PRE_FUNCT init_one_tbcol");
# 获取表结构及列顺序
my $cmd = $MYSQL." --skip-column-names --silent -e 'desc $tbname'";
# 提取列名,并拼接
$cmd .= " | awk -F\'\\t\' \'{print NR\"$SPLITER_COL`\"\$1\"`$SPLITER_COL\"\$2}'";
&mdebug("get table column infor cmd\n\t$cmd");
open TBCOL,"$cmd | " or die "can't open desc $tbname;"; my %colpos;
while (my $line = <TBCOL>){
chomp($line);
my ($pos,$col,$coltype) = split(/$SPLITER_COL/,$line);
&mdebug("linesss=$line\n\t\tpos=$pos\n\t\tcol=$col\n\t\ttype=$coltype");
$colpos{$pos} = $col.$SPLITER_COL.$coltype;
}
close TBCOL or die "can't colse desc $tbname"; $tbcol_pos{$tbname} = \%colpos;
} # ----------------------------------------------------------------------------------------
# Func : rollback sql: INSERT/UPDATE/DELETE
# ----------------------------------------------------------------------------------------
sub do_binlog_rollback{
my $binlogfile = "$ROLLBACK_DML ";
&mdebug("$PRE_FUNCT do_binlog_rollback"); # INSERT|UPDATE|DELETE
my $sqltype;
# WHERE|SET
my $sqlarea; my ($tbname, $sqlstr) = ('', '');
my ($notignore, $isareabegin) = (,); # output sql file
open SQLFILE, ">> $outfile" or die "Can't open sql file:$outfile"; # binlog file
open BINLOG, "$binlogfile |" or die "Can't open file: $binlogfile";
while (my $line = <BINLOG>){
chomp($line);
if ($line =~ /^(INSERT|UPDATE|DELETE)/){
# export sql
if ($sqlstr ne ''){
$sqlstr .= ";\n";
print SQLFILE $sqlstr;
&mdebug("export sql\n\t".$sqlstr);
$sqlstr = '';
} if ($line =~ /^INSERT/){
$sqltype = $SQLTYPE_IST;
$tbname = `echo '$line' | awk '{print \$3}'`;
chomp($tbname);
$sqlstr = qq{DELETE FROM $tbname};
}elsif ($line =~ /^UPDATE/){
$sqltype = $SQLTYPE_UPD;
$tbname = `echo '$line' | awk '{print \$2}'`;
chomp($tbname);
$sqlstr = qq{UPDATE $tbname};
}elsif ($line =~ /^DELETE/){
$sqltype = $SQLTYPE_DEL;
$tbname = `echo '$line' | awk '{print \$3}'`;
chomp($tbname);
$sqlstr = qq{INSERT INTO $tbname};
} # check ignore table
if(&ignore_tb($tbname)){
$notignore = ;
&mdebug("<BINLOG>#IGNORE#:line:".$line);
$sqlstr = '';
}else{
$notignore = ;
&mdebug("<BINLOG>#DO#:line:".$line);
}
}else {
if($notignore){
&merror("can't get tbname") unless (defined($tbname));
if ($line =~ /^WHERE/){
$sqlarea = $SQLAREA_WHERE;
$sqlstr .= qq{ SET};
$isareabegin = ;
}elsif ($line =~ /^SET/){
$sqlarea = $SQLAREA_SET;
$sqlstr .= qq{ WHERE};
$isareabegin = ;
}elsif ($line =~ /^\@/){
$sqlstr .= &deal_col_value($tbname, $sqltype, $sqlarea, $isareabegin, $line);
$isareabegin = ;
}else{
&mdebug("::unknown sql:".$line);
}
}
}
}
# export last sql
if ($sqlstr ne ''){
$sqlstr .= ";\n";
print SQLFILE $sqlstr;
&mdebug("export sql\n\t".$sqlstr);
} close BINLOG or die "Can't close binlog file: $binlogfile"; close SQLFILE or die "Can't close out sql file: $outfile"; # 逆序
# 1!G: 只有第一行不执行G, 将hold space中的内容append回到pattern space
# h: 将pattern space 拷贝到hold space
# $!d: 除最后一行都删除
my $invert = "sed -i '1!G;h;\$!d' $outfile";
my $res = `$invert`;
&mdebug("inverter order sqlfile :$invert");
} # ----------------------------------------------------------------------------------------
# Func : transfer column pos to name
# deal column value
#
# &deal_col_value($tbname, $sqltype, $sqlarea, $isareabegin, $line);
# ----------------------------------------------------------------------------------------
sub deal_col_value($$$$$){
my ($tbname, $sqltype, $sqlarea, $isareabegin, $line) = @_;
&mdebug("$PRE_FUNCT deal_col_value");
&mdebug("input:tbname->$tbname,type->$sqltype,area->$sqlarea,areabegin->$isareabegin,line->$line");
my @vals = split(/=/, $line);
my $pos = substr($vals[],);
my $valstartpos = length($pos)+;
my $val = substr($line,$valstartpos);
my %tbcol = %{$tbcol_pos{$tbname}};
my ($cname,$ctype) = split(/$SPLITER_COL/,$tbcol{$pos});
&merror("can't get $tbname column $cname type") unless (defined($cname) || defined($ctype));
&mdebug("column infor:cname->$cname,type->$ctype"); # join str
my $joinstr;
if ($isareabegin){
$joinstr = ' ';
}else{
# WHERE 被替换为 SET, 使用 , 连接
if ($sqlarea eq $SQLAREA_WHERE){
$joinstr = ', ';
# SET 被替换为 WHERE 使用 AND 连接
}elsif ($sqlarea eq $SQLAREA_SET){
$joinstr = ' AND ';
}else{
&merror("!!!!!!The scripts error");
}
} #
my $newline = $joinstr; # NULL value
if (($val eq 'NULL') && ($sqlarea eq $SQLAREA_SET)){
$newline .= qq{ $cname IS NULL};
}else{
# timestamp: record seconds
if ($ctype eq 'timestamp'){
$newline .= qq{$cname=from_unixtime($val)};
# datetime: @n=yyyy-mm-dd hh::ii::ss
}elsif ($ctype eq 'datetime'){
$newline .= qq{$cname='$val'};
}else{
$newline .= qq{$cname=$val};
}
}
&mdebug("\told>$line\n\tnew>$newline"); return $newline;
} # ----------------------------------------------------------------------------------------
# Func : check is ignore table
# params: IN table full name # format:`dbname`.`tbname`
# RETURN:
# 0 not ignore
# 1 ignore
# ----------------------------------------------------------------------------------------
sub ignore_tb($){
my $fullname = shift;
# 删除`
$fullname =~ s/`//g;
my ($dbname,$tbname) = split(/\./,$fullname);
my $res = ; # 指定了数据库
if ($opt{'d'}){
# 与指定库相同
if ($do_dbs{$dbname}){
# 指定表
if ($opt{'T'}){
# 与指定表不同
unless ($do_tbs{$tbname}){
$res = ;
}
}
# 与指定库不同
}else{
$res = ;
}
}
#&mdebug("Table check ignore:$fullname->$res");
return $res;
} # ----------------------------------------------------------------------------------------
# Func : print debug msg
# ----------------------------------------------------------------------------------------
sub mdebug{
my (@msg) = @_;
print "<DEBUG>@msg\n" if ($opt{'debug'});
} # ----------------------------------------------------------------------------------------
# Func : print error msg and exit
# ----------------------------------------------------------------------------------------
sub merror{
my (@msg) = @_;
print "<Error>:@msg\n";
&print_usage();
exit();
} # ----------------------------------------------------------------------------------------
# Func : print usage
# ----------------------------------------------------------------------------------------
sub print_usage{
print <<EOF;
==========================================================================================
Command line options :
--help # OUT : print help info
-f, --srcfile # IN : binlog file. [required]
-o, --outfile # OUT : output sql file. [required]
-h, --host # IN : host. default '127.0.0.1'
-u, --user # IN : user. [required]
-p, --password # IN : password. [required]
-P, --port # IN : port. default '3306'
--start-datetime # IN : start datetime
--stop-datetime # IN : stop datetime
--start-position # IN : start position
--stop-position # IN : stop position
-d, --database # IN : database, split comma
-T, --table # IN : table, split comma. [required] set -d
-i, --ignore # IN : ignore binlog check contain DDL(CREATE|ALTER|DROP|RENAME)
--debug # IN : print debug information Sample :
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -u 'user' -p 'pwd'
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -u 'user' -p 'pwd' -i
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -u 'user' -p 'pwd' --debug
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -h '192.168.1.2' -u 'user' -p 'pwd' -P
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -u 'user' -p 'pwd' --start-position=
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -u 'user' -p 'pwd' --start-position= --stop-position=
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -u 'user' -p 'pwd' -d 'db1,db2'
shell> perl binlog-rollback.pl -f 'mysql-bin.0000*' -o '/tmp/t.sql' -u 'user' -p 'pwd' -d 'db1,db2' -T 'tb1,tb2'
==========================================================================================
EOF
exit;
} ;
这脚本含有注释以及使用说明,所以使用起来还是比较简单的,如果你会Perl语言,相信也很容易看懂代码。binlog-rollback.pl的使用参数如下:
[root@localhost mysql3306]# perl binlog-rollback.pl
==========================================================================================
Command line options :
--help # OUT : print help info
-f, --srcfile # IN : binlog file. [required]
-o, --outfile # OUT : output sql file. [required]
-h, --host # IN : host. default '127.0.0.1'
-u, --user # IN : user. [required]
-p, --password # IN : password. [required]
-P, --port # IN : port. default ''
--start-datetime # IN : start datetime
--stop-datetime # IN : stop datetime
--start-position # IN : start position
--stop-position # IN : stop position
-d, --database # IN : database, split comma
-T, --table # IN : table, split comma. [required] set -d
-i, --ignore # IN : ignore binlog check contain DDL(CREATE|ALTER|DROP|RENAME)
--debug # IN : print debug information Sample :
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -u 'user' -p 'pwd'
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -u 'user' -p 'pwd' -i
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -u 'user' -p 'pwd' --debug
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -h '192.168.1.2' -u 'user' -p 'pwd' -P
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -u 'user' -p 'pwd' --start-position=
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -u 'user' -p 'pwd' --start-position= --stop-position=
shell> perl binlog-rollback.pl -f 'mysql-bin.000001' -o '/tmp/t.sql' -u 'user' -p 'pwd' -d 'db1,db2'
shell> perl binlog-rollback.pl -f 'mysql-bin.0000*' -o '/tmp/t.sql' -u 'user' -p 'pwd' -d 'db1,db2' -T 'tb1,tb2'
==========================================================================================
[root@localhost mysql3306]#
下面主要演示对一个表的增、删、修(Insert/Delete/Update)操作,基于Binlog为Row格式的反向解析。
细心看脚本的朋友都能看到这个脚本需要提供一个连接MySQL的用户,主要是为了获取表结构。下面我们测试一个普通用户并给予SELECT权限即可,默认是host是127.0.0.1,这个可以修改脚本,我这里按脚本默认的:
<Test>[(none)]> GRANT SELECT ON *.* TO 'recovery'@'127.0.0.1' identified by '';
Query OK, 0 rows affected (0.08 sec) <Test>[(none)]> flush privileges;
Query OK, 0 rows affected (0.04 sec) <Test>[(none)]>
往xuanzhi库的表tb1里插入2行数据,记得binlog格式要为ROW:
<Test>[xuanzhi]> show global variables like 'binlog_format';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| binlog_format | ROW |
+---------------+-------+
1 row in set (0.00 sec) <Test>[xuanzhi]> insert into xuanzhi.tb1 select 1,'aa';
Query OK, 1 row affected (0.01 sec)
Records: 1 Duplicates: 0 Warnings: 0 <Test>[xuanzhi]> insert into xuanzhi.tb1 select 2,'cc';
Query OK, 1 row affected (0.01 sec)
Records: 1 Duplicates: 0 Warnings: 0 <Test>[xuanzhi]> select * from xuanzhi.tb1;
+------+------+
| id | name |
+------+------+
| 1 | aa |
| 2 | cc |
+------+------+
2 rows in set (0.00 sec) <Test>[xuanzhi]>
为了看到运行脚本在不指定库看到的效果,我这里再往test库的user表插入两行数据:
<Test>[xuanzhi]> insert into test.user select 1,'user1',20;
Query OK, 1 row affected (0.03 sec)
Records: 1 Duplicates: 0 Warnings: 0 <Test>[xuanzhi]> insert into test.user select 2,'user2',30;
Query OK, 1 row affected (0.01 sec)
Records: 1 Duplicates: 0 Warnings: 0 <Test>[xuanzhi]>
查看此时的此时处于那个binlog:
<Test>[xuanzhi]> show master status;
+----------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+----------------------+----------+--------------+------------------+-------------------+
| localhost-bin.000023 | 936 | | | |
+----------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec) <Test>[xuanzhi]>
下面运行脚本 binlog-rollback.pl ,不指定任何库和表的情况下,这时表把binlog里所有DML操作都生成反向的SQL(最新的DML会生成在输入文件的最前面):
[root@localhost mysql3306]# perl binlog-rollback.pl -f 'localhost-bin.000023' -o '/data/t.sql' -u 'recovery' -p ''
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
[root@localhost mysql3306]#
我们查看输出的文件:/data/t.sql
[root@localhost mysql3306]# cat /data/t.sql
DELETE FROM `test`.`user` WHERE `id`= AND `name`='user2' AND `age`=;
DELETE FROM `test`.`user` WHERE `id`= AND `name`='user1' AND `age`=;
DELETE FROM `xuanzhi`.`tb1` WHERE `id`= AND `name`='bb';
DELETE FROM `xuanzhi`.`tb1` WHERE `id`= AND `name`='aa';
可以看到,INSERT操作的反向操作就是DELETE,这里把所有库的DML操作都查出来了,在后面会演示找单个库或者表所产生的反向SQL。
下面模拟运维人员、开发人员或者DBA误操删除数据,分别在不同的库删除一条记录:
<Test>[xuanzhi]> delete from xuanzhi.tb1 where id=2;
Query OK, 1 row affected (0.06 sec) <Test>[xuanzhi]> delete from test.user where id=1;
Query OK, 1 row affected (0.00 sec) <Test>[xuanzhi]>
这个时候发现自己删除错了,需要恢复,刚好这些数据不在最新的备份里,正常的恢复方法有两种:
一、是基于最新的完整备份+binlog进行数据恢复了,这时需要把备份导回去,还要找出Binlog DELETE前的pos位置,再进行binlog恢复,恢复完后再把记录恢复到误操的环境上。如果表很大,这时间要很久。
二、因为Binlog格式为ROW时,记录了行的修改,所以DELETE是可以看到所有列的值的,把binlog解析出来,找到被DELETE的记录,通过各种处理再恢复回去,但binlog不能基于一个库或表级别的解析,只能整个binlog解析再进行操作。
以上的方法都比较消耗时间,当然使用binlog-rollback.pl脚本有点类似第二种方法,但是binlog-rollback.pl可以指定库或表进行反向解析,还可以指定POS点,效率相当更高一些。
下面我们运行 binlog-rollback.pl 脚本,生成删除数据语句的反向SQL:
[root@localhost mysql3306]# perl binlog-rollback.pl -f 'localhost-bin.000023' -o '/data/t.sql' -u 'recovery' -p ''
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
[root@localhost mysql3306]#
再次查看输出文件:
[root@localhost mysql3306]# cat /data/t.sql
INSERT INTO `test`.`user` SET `id`=, `name`='user1', `age`=;
INSERT INTO `xuanzhi`.`tb1` SET `id`=, `name`='bb';
DELETE FROM `test`.`user` WHERE `id`= AND `name`='user2' AND `age`=;
DELETE FROM `test`.`user` WHERE `id`= AND `name`='user1' AND `age`=;
DELETE FROM `xuanzhi`.`tb1` WHERE `id`= AND `name`='bb';
DELETE FROM `xuanzhi`.`tb1` WHERE `id`= AND `name`='aa'; [root@localhost mysql3306]#
刚刚DELETE的2条记录已经生成了反向INSERT语句,这样恢复就简单多啦:
INSERT INTO `test`.`user` SET `id`=, `name`='user1', `age`=;
INSERT INTO `xuanzhi`.`tb1` SET `id`=, `name`='bb';
下面我们模拟修改数据的时候,误修改了,如下:
<Test>[xuanzhi]> select * from xuanzhi.tb1;
+------+------+
| id | name |
+------+------+
| 1 | aa |
+------+------+
1 row in set (0.00 sec) <Test>[xuanzhi]> select * from test.user;
+------+-------+------+
| id | name | age |
+------+-------+------+
| 2 | user2 | 30 |
+------+-------+------+
1 row in set (0.00 sec) <Test>[xuanzhi]> update xuanzhi.tb1 set name = 'MySQL' where id=1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0 <Test>[xuanzhi]> update test.user set age = 20 where id = 2;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0 <Test>[xuanzhi]>
这个时候发现修改错数据了,需要还原,同样可以使用脚本binlog-rollback.pl 进行对所在Binlog的DML生成反向的SQL,进行恢复:
[root@localhost mysql3306]# perl binlog-rollback.pl -f 'localhost-bin.000023' -o '/data/t.sql' -u 'recovery' -p ''
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
[root@localhost mysql3306]#
再查看输出文件:
[root@localhost mysql3306]# cat /data/t.sql
UPDATE `test`.`user` SET `id`=, `name`='user2', `age`= WHERE `id`= AND `name`='user2' AND `age`=;
UPDATE `xuanzhi`.`tb1` SET `id`=, `name`='aa' WHERE `id`= AND `name`='MySQL';
INSERT INTO `test`.`user` SET `id`=, `name`='user1', `age`=;
INSERT INTO `xuanzhi`.`tb1` SET `id`=, `name`='bb';
DELETE FROM `test`.`user` WHERE `id`= AND `name`='user2' AND `age`=;
DELETE FROM `test`.`user` WHERE `id`= AND `name`='user1' AND `age`=;
DELETE FROM `xuanzhi`.`tb1` WHERE `id`= AND `name`='bb';
DELETE FROM `xuanzhi`.`tb1` WHERE `id`= AND `name`='aa'; [root@localhost mysql3306]#
可以看到生成了反向的UPDATE语句:
UPDATE `test`.`user` SET `id`=, `name`='user2', `age`= WHERE `id`= AND `name`='user2' AND `age`=;
UPDATE `xuanzhi`.`tb1` SET `id`=, `name`='aa' WHERE `id`= AND `name`='MySQL';
下面进行指定库的反向解析,参数为(-d)
[root@localhost mysql3306]# perl binlog-rollback.pl -f 'localhost-bin.000023' -o '/data/t.sql' -u 'recovery' -p '' -d 'xuanzhi'
mysql: [Warning] Using a password on the command line interface can be insecure.
[root@localhost mysql3306]# cat /data/t.sql
UPDATE `xuanzhi`.`tb1` SET `id`=, `name`='aa' WHERE `id`= AND `name`='MySQL';
INSERT INTO `xuanzhi`.`tb1` SET `id`=, `name`='bb';
DELETE FROM `xuanzhi`.`tb1` WHERE `id`= AND `name`='bb';
DELETE FROM `xuanzhi`.`tb1` WHERE `id`= AND `name`='aa'; [root@localhost mysql3306]#
可以看到输入的文件只含xuanzhi库的所有DML的反向SQL。
下面进行指定库下某个表的反向解析,参数为:-T (为了看到效果在xuanzhi库下的tb2表删除一些记录):
<Test>[xuanzhi]> select * from tb2;
+------+------+
| id | name |
+------+------+
| 1 | aa |
| 2 | bb |
| 3 | cc |
+------+------+
3 rows in set (0.04 sec) <Test>[xuanzhi]> delete from xuanzhi.tb2 where id <2;
Query OK, 1 row affected (0.02 sec) <Test>[xuanzhi]>
这个时候应该如果只指定xuanzhi库,那么tb1和tb2的DML操作的反向操作都会记录下来:
[root@localhost mysql3306]# perl binlog-rollback.pl -f 'localhost-bin.000023' -o '/data/t.sql' -u 'recovery' -p '' -d 'xuanzhi'
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
[root@localhost mysql3306]# cat /data/t.sql
INSERT INTO `xuanzhi`.`tb2` SET `id`=, `name`='aa';
UPDATE `xuanzhi`.`tb1` SET `id`=, `name`='aa' WHERE `id`= AND `name`='MySQL';
INSERT INTO `xuanzhi`.`tb1` SET `id`=, `name`='bb';
DELETE FROM `xuanzhi`.`tb1` WHERE `id`= AND `name`='bb';
DELETE FROM `xuanzhi`.`tb1` WHERE `id`= AND `name`='aa'; [root@localhost mysql3306]#
指定单个表tb2:
[root@localhost mysql3306]# perl binlog-rollback.pl -f 'localhost-bin.000023' -o '/data/t.sql' -u 'recovery' -p '' -d 'xuanzhi' -T 'tb2'
mysql: [Warning] Using a password on the command line interface can be insecure.
[root@localhost mysql3306]# cat /data/t.sql
INSERT INTO `xuanzhi`.`tb2` SET `id`=, `name`='aa'; [root@localhost mysql3306]#
因为上面删除了一条tb2的数据,所有这个文件就对应生成一条tb2的INSERT记录
下面进行POS点生成反向SQL:(--start-position= --stop-position=)
# at
# :: server id end_log_pos CRC32 0xb67ef6ba Query thread_id= exec_time= error_code=
SET TIMESTAMP=/*!*/;
BEGIN
/*!*/;
# at
# :: server id end_log_pos CRC32 0x219a127c Table_map: `test`.`user` mapped to number
# at
# :: server id end_log_pos CRC32 0xf5e0d39e Update_rows: table id flags: STMT_END_F BINLOG '
K+TdVhPqlBMAMwAAAJMGAAAAAEoAAAAAAAEABHRlc3QABHVzZXIAAwP+AwL+Hgd8Epoh
K+TdVh/qlBMAQgAAANUGAAAAAEoAAAAAAAEAAgAD///4AgAAAAV1c2VyMh4AAAD4AgAAAAV1c2Vy
MhQAAACe0+D1
'/*!*/;
### UPDATE `test`.`user`
### WHERE
### @=
### @='user2'
### @=
### SET
### @=
### @='user2'
### @=
# at
# :: server id end_log_pos CRC32 0x1e62cb77 Xid =
COMMIT/*!*/;
# at
# :: server id end_log_pos CRC32 0x04dfe1f0 Query thread_id= exec_time= error_code=
SET TIMESTAMP=/*!*/;
BEGIN
/*!*/;
# at
# :: server id end_log_pos CRC32 0x897ae6bd Table_map: `xuanzhi`.`tb2` mapped to number
# at
# :: server id end_log_pos CRC32 0xea61aff0 Delete_rows: table id flags: STMT_END_F BINLOG '
QOfdVhPqlBMANAAAAHMHAAAAAEYAAAAAAAEAB3h1YW56aGkAA3RiMgACA/4C/goDveZ6iQ==
QOfdViDqlBMAKwAAAJ4HAAAAAEYAAAAAAAEAAgAC//wBAAAAAmFh8K9h6g==
'/*!*/;
### DELETE FROM `xuanzhi`.`tb2`
### WHERE
### @=
### @='aa'
# at
# :: server id end_log_pos CRC32 0x49e1ce9c Xid =
COMMIT/*!*/;
从上面的binlog可以看到开始的--start-position=1557 结束的--stop-position=1981,这一段binlog里做了UPDATE `test`.`user` ... 和 DELETE FROM `xuanzhi`.`tb2` ... 的操作,那么用binlog-rollback.pl应该会生成一个UPDATE和一个INSERT语句
[root@localhost mysql3306]# perl binlog-rollback.pl -f 'localhost-bin.000023' -o '/data/t.sql' -u 'recovery' -p '123456' --start-position=1557 --stop-position=1981
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
[root@localhost mysql3306]# cat /data/t.sql
INSERT INTO `xuanzhi`.`tb2` SET `id`=1, `name`='aa';
UPDATE `test`.`user` SET `id`=2, `name`='user2', `age`=30 WHERE `id`=2 AND `name`='user2' AND `age`=20; [root@localhost mysql3306]#
更多的测试,就看同学们了,有测试不当的地方请告诉我,大家一起学习。
总结: 一、感谢那些有分享精神的大神们,让我们学到了更多的东西,但开源的脚本需要多测试。
二、误操的情况,时有发生,所以我们要做好备份,做好一些数据恢复的测试。
三、该脚本在处理比较在的binlog时,会经常出现些小问题
作者:陆炫志 出处:xuanzhi的博客 http://www.cnblogs.com/xuanzhi201111 您的支持是对博主最大的鼓励,感谢您的认真阅读。本文版权归作者所有,欢迎转载,但请保留该声明。 |
MySQL基于ROW格式的数据恢复的更多相关文章
- 浅析MySQL基于ROW格式的二进制日志
上文分析的二进制日志实际上是基于STATEMENT格式的,下面我们来看看基于ROW格式的二进制日志,毕竟,两者对应的binlog事件类型也不一样,同时,很多童鞋反映基于ROW格式的二进制日志无法查到原 ...
- my42_Mysql基于ROW格式的主从同步
模拟主从update事务,从库跳过部分update事务后,再次开始同步的现象 主库 mysql> select * from dbamngdb.isNodeOK; +----+--------- ...
- MySQL Binlog Mixed模式记录成Row格式
背景: 一个简单的主从结构,主的binlog format是Mixed模式,在执行一条简单的导入语句时,通过mysqlbinlog导出发现记录的Binlog全部变成了Row的格式(明明设置的是Mixe ...
- my15_ mysql binlog格式从mixed修改为row格式
由于主库繁忙,就在从库上修改binlog格式 1. 从库切日志mysql> flush logs;Query OK, 0 rows affected (0.00 sec) mysql> f ...
- 基于GTID环境的数据恢复
下面说一个线上环境的数据恢复案例,线上环境误操作的事情,随时都有可能发生的,这时候运维DBA或者DBA会想,没人会傻到把数据库干掉的吧?又或者没有会闲得蛋痛删除了几条数据又想恢复这么无聊吧?适适这样的 ...
- Mysql基于GTID主从复制
Mysql5.6基于GTID全局事务的复制 什么是GTID? GTID(Global Transaction Identifiers)是全局事务标识 当使用GTIDS时,在主上提交的每一个事务都会 ...
- MySQL二进制日志格式对复制的影响
复制的分类 基于SQL语句的复制 - SBR 主库二进制日志格式使用STATEMENT 在MySQL 5.1之前仅存在SBR模式, 又称之为逻辑复制. 主库记录CUD操作的SQL语句, 从库会读取并重 ...
- MySQL基于binlog主从复制
MySQL复制介绍 默认情况 下复制是异步进行的,从库也不需要一直连接到主库来同步数据 MySQL复制的数据粒度可以是主实例上所有的数据库,也可以是指定的一个或多个数据库 ,也可以是一个数据库里的指定 ...
- Mysql基于GTID复制模式-运维小结 (完整篇)
先来看mysql5.6主从同步操作时遇到的一个报错:mysql> change master to master_host='192.168.10.59',master_user='repli' ...
随机推荐
- java.lang.RuntimeException: Cannot create a secure XMLInputFactory 异常处理
背景:本人在近日的webservice CXF接口的编写调试过程中,发现一个很奇怪的现象--我在本地对接口进行往返请求时完全没有问题,不管是main方法直接调用还是业务场景里调用都无问题,在其他同 ...
- JS 如何将 HTML 页面导出为多页 PDF
参考链接:https://blog.csdn.net/pwc1996/article/details/70141383
- [转载]Tensorflow 的reduce_sum()函数的axis,keep_dim这些参数到底是什么意思?
转载链接:https://www.zhihu.com/question/51325408/answer/125426642来源:知乎 这个问题无外乎有三个难点: 什么是sum 什么是reduce 什么 ...
- 新年第一个目标一张表盘串讲所有canves的知识点
我们的目标 首先是canves的坐标系统,基于浏览器的左上角为原点,x,y轴为正方向的坐标系统. 首先初始化,打标签 <canvas id="canvas" height=& ...
- Linux查看本机IP:curl cip.cc
curl http://members.3322.org/dyndns/getip curl ip.6655.com/ip.aspx curl ifconfig.me curl icanhazip.c ...
- 基于React 的前端UI开发框架 及与Electron 的结合 https://cxjs.io/
1.cxjs 基于React 的前端UI开发框架 https://cxjs.io/ coreu http://coreui.io/ 2.antd-admin ...
- ASP.NET WEBAPI 使用Swagger生成API文档
一.安装 新建一个没有身份验证的mvc项目 - SwaggerMvc5Demo,然后添加一个名为Remote(自定义)且包含基础读写(不想手写)的ApiController 开源地址:https: ...
- ubuntu html5开发工具brackets
Brackets 是一款使用 HTML,CSS,JavaScript 创建的开源的针对 Web 开发的编辑器.实时预览,快速编辑,跨平台,可扩展,开源,让 Brackets 成为一款非常优秀的编辑器. ...
- Razor视图基本语法
<!--Razor C#--> @for (int i = 0; i < 10; i++) { <baobao>good</baobao> } < ...
- java子类数组的引用转换成超类数组的引用
public class Person { } public class Student extends Person{ private String name; public Student(Str ...