Online DDL:

一.FIC之前的添加删除二级索引:
  • 1.首先创建临时表,定义目标新表的选项和索引
  • 2.逐行拷贝数据到临时表
  • 3.插入行时更新索引信息
  • 4.数据全部被拷贝到新表后,删除旧表,rename新表

为保证数据一致性,旧表复制数据到新表的全程只允许读,不允许改

二.5.1 5.5FIC新特性 fast index creation
  • 1.创建二级索引给表加上共享S锁
  • 2.创建过程不需要重建表
  • 3.删除二级索引只需要更新内部视图,并标记索引的空间可用,去除数据库元数据的关于该索引的定义

依然只能读,不允许改,但加快了修改索引的速度

因为聚簇索引的特性,主键索引修改依然需要通过旧方式copy table过程

三.5.6之后
  • 1.之前FIC只对二级索引创建删除有效,5.6之后将此特性扩展到添加列、删除列,修改列类型、列重命名、设置默认值

Online DDL选项

索引的copy与inplace创建

一.ALGORITHM=COPY
  • 1.新建带索引(主键索引)的临时表
  • 2.将原表锁定,禁止DML操作,只允许select查询
  • 3.将原表数据拷贝到临时表
  • 4.禁止读写,进行rename,升级字典锁
  • 5.完成创建索引操作

需要记录undo和redo,效率不如inplace,短期占用buffer pool,影响性能

二.ALGORITHM=INPLACE
  • 1.创建索引(二级索引,主键+普通字段)数据字典
  • 2.加表共享锁S,禁止DML,允许select查询
  • 3.读取聚簇索引,构造新的索引项,排序并插入新索引
  • 5.等待打开当前表的所有只读事务提交
  • 6.创建索引结束

避免重建表带来的IO和CPU消耗,保证DDL期间的性能和并发

三.ALGORITHM=DEFAULT
  • 1.alter table后面什么都不加的时候,默认是这个方式
  • 2.如果old_alter_table为OFF,默认就是inplace方式
  • 3.inplace不支持则进行copy
四.lock选项

控制是否锁表,根据不同的DDL操作类型表现不同,mysql原则是尽量不锁表,但是修改主键这样的昂贵操作不得不锁表

  • 1.LOCK=NONE,允许DDL期间并发读写涉及的表,显式指定时,当不支持对该表的继续写入,则alter语句失败,是ALGORITHM=COPY的默认lock级别
  • 2.LOCK=SHARED,DDL期间表上的写操作会被阻塞,但是不影响select
  • 3.LOCK=DEFAULT,让mysql自己判断lock模式,原则是尽量不锁表
  • 4.LOCK=EXCLUSIVE,DDL期间该表不可用,堵塞任何读写请求,使用场景:
    • 最短时间内完成
    • 短时间表不可用能刚接受

注意:

  • 1.任何模式下,online DDL开始之前都需要一个短时间的排它锁X来准备环境
  • 2.当alter命令发出后,会首先等待该表上的其他操作完成
  • 3.alter命令之后的其他请求会出现等待MDL锁
  • 4.alter完成之前,其他DDL也会被阻塞一小段时间?

https://www.cnblogs.com/zengkefu/p/5674945.html


ddl实现细节

Prepare阶段
  • 1.创建frm文件
  • 2.持有exclusive-mdl锁,禁止读写,速度非常快
  • 3.根据alter类型,确定执行方式,检查存储引擎是否支持inplace,不支持则使用copy---copy,online-rebuild:重新组织表,online-norebuild:改数据字典即可
  • 4.更新数据字典的内存对象
  • 5.分配row_log对象记录增量,记录DDL期间数据修改的日志,如果日志量超过参数 innodb_online_alter_log_max_size 的上限,则DDL失败
  • 6.生成临时ibd文件(rebuild类型)
ddl执行阶段
  • 1.降级exclusvie-mdl锁,允许读写
  • 2.扫描原表的聚簇索引每条记录
  • 3.遍历新表的聚簇索引和二级索引,逐一处理
  • 4.根据记录构造对应的索引项
  • 5.将构造索引项插入sort_buffer块进行排序
  • 6.将sort_buffer块中的排序结果插入新的索引
  • 7.处理ddl执行过程中产生的增量-row-log,rebuild类型需要
commit阶段
  • 1.升级到exclusive-mdl索引,禁止读写
  • 2.应用最后row_log中产生的日志
  • 3.更新innodb的数据字典
  • 4.提交事务-刷数据的redo日志
  • 5.修改统计信息
  • 6.rename临时ibd文件,frm文件
  • 7.变更完成

http://blog.itpub.net/22664653/viewspace-2056953


DDL SQL组合使用

1.全部变更写到一条sql里面
alter table t1 add index idx1(c1),add unique index idx2(c2),change c4_old_name c4_new_name integer unsigned;

copy方式下,效率最高,只需要拷贝1次

2.每个变更一条sql
alter table t1 add index idx1(c1);
alter table t1 add unique index idx2(c2);
alter table t1 change c4_old_name c4_new_name integer unsigned not null;

测试

1.显式开启一个查询事务,默认DDL并不好?理论与实际不一样
  • session1
mysql session 1 >use db1;
Database changed
mysql session 1 >begin;
Query OK, 0 rows affected (0.00 sec) mysql session 1 >select * from t1;
+------+------+
| id | name |
+------+------+
| 1 | yzw |
+------+------+
1 row in set (0.00 sec) mysql session 1 >

开始一个事务,执行简单查询

  • session2
mysql session 2 >use db1
Database changed
mysql session 2 >alter table t1 add index idx_id(id);
Query OK, 0 rows affected (9 min 47.09 sec)
Records: 0 Duplicates: 0 Warnings: 0

被阻塞9分47.09秒

  • session3
mysql session 3 >use db1;
Database changed
mysql session 3 >select * from t1;
+------+------+
| id | name |
+------+------+
| 1 | yzw |
+------+------+
1 row in set (9 min 37.60 sec)

被阻塞9分37.60秒

  • session4
mysql session 4 >mysql session 4 >show processlist;
+----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 432908 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 432806 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 0 | | NULL |
| 67 | root | localhost | db1 | Query | 23 | Waiting for table metadata lock | alter table t1 add index idx_id(id) |
| 68 | root | localhost | db1 | Query | 14 | Waiting for table metadata lock | select * from t1 |
| 69 | root | localhost | NULL | Query | 0 | starting | show processlist |
| 70 | root | localhost | db1 | Sleep | 36 | | NULL |
+----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-------------------------------------+
7 rows in set (0.00 sec)

session2的alter和session3的select被阻塞

都在等待MDL锁

alter

2.第二个加了算法,时间更长,dml并发更好?理论与实际不一样
  • session1
mysql session 1 >begin;
Query OK, 0 rows affected (0.00 sec) mysql session 1 >select * from t1;
+------+------+
| id | name |
+------+------+
| 1 | yzw |
+------+------+
1 row in set (0.00 sec)

开启一个事务

  • session2
mysql session 2 >use db1
Database changed
mysql session 2 >alter table t1 add index idx_id(id), algorithm=inplace,lock=none;
Query OK, 0 rows affected (9 min 34.48 sec)
Records: 0 Duplicates: 0 Warnings: 0

显示指定inplace方式增加索引

  • session3
mysql session 3 >use db1;
Database changed
mysql session 3 >select * from t1;
+------+------+
| id | name |
+------+------+
| 1 | yzw |
+------+------+
1 row in set (9 min 22.13 sec)

查询被阻塞

  • session4
mysql session 4 >show processlist;
+----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 500684 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 500582 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 1 | | NULL |
| 71 | root | localhost | db1 | Sleep | 45 | | NULL |
| 72 | root | localhost | db1 | Query | 20 | Waiting for table metadata lock | alter table t1 add index idx_id(id), algorithm=inplace,lock=none |
| 73 | root | localhost | db1 | Query | 7 | Waiting for table metadata lock | select * from t1 |
| 74 | root | localhost | NULL | Query | 0 | starting | show processlist |
+----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------+
7 rows in set (0.00 sec)

等待MDL锁

3.原有copy方式DDL
1.构造测试数据
sysbench --test=insert.lua \
--oltp-table-size=5000000 --mysql-table-engine=innodb --mysql-user=root --mysql-password=123456 \
--mysql-port=3306 --mysql-host=localhost --mysql-db=test --max-requests=0 --time=60 \
--oltp-tables-count=2 --report-interval=10 --threads=2 prepare
2.使用原有copy方式进行ddl
mysql session 1 >select version();
+------------+
| version() |
+------------+
| 5.7.16-log |
+------------+
1 row in set (0.00 sec) mysql session 1 >show create table sbtest1
-> ;
+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| sbtest1 | CREATE TABLE `sbtest1` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`k` int(10) unsigned NOT NULL DEFAULT '0',
`c` char(120) NOT NULL DEFAULT '',
`pad` char(60) NOT NULL DEFAULT '',
PRIMARY KEY (`id`),
KEY `k_1` (`k`)
) ENGINE=InnoDB AUTO_INCREMENT=5000001 DEFAULT CHARSET=utf8 MAX_ROWS=1000000 |
+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec) mysql session 1 >show variables like '%old_alter_table%';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| old_alter_table | OFF |
+-----------------+-------+
1 row in set (0.01 sec) mysql session 1 >set global old_alter_table=on;
Query OK, 0 rows affected (0.00 sec) mysql session 1 >show variables like 'old_alter_table';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| old_alter_table | OFF |
+-----------------+-------+
1 row in set (0.00 sec) mysql session 1 >set old_alter_table=1;
Query OK, 0 rows affected (0.00 sec) mysql session 1 >show variables like 'old_alter_table';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| old_alter_table | ON |
+-----------------+-------+
1 row in set (0.00 sec) mysql session 1 >show index from sbtest1;
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| sbtest1 | 0 | PRIMARY | 1 | id | A | 4804854 | NULL | NULL | | BTREE | | |
| sbtest1 | 1 | k_1 | 1 | k | A | 883154 | NULL | NULL | | BTREE | | |
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
2 rows in set (0.00 sec)
2.删除增加索引

会触发copy操作,并在数据目录产生临时frm和ibd文件

其他DML会被阻塞,请求MDL锁

  • session1
mysql session 1 >alter table sbtest1 drop index k_1;
Query OK, 5000000 rows affected (55.42 sec)
Records: 5000000 Duplicates: 0 Warnings: 0 mysql session 1 >alter table sbtest1 add index k_1(k);
Query OK, 5000000 rows affected (1 min 20.26 sec)
Records: 5000000 Duplicates: 0 Warnings: 0
  • session2
mysql session 2 >update sbtest1 set k=2496416 where id=1;
Query OK, 1 row affected (14.39 sec)
Rows matched: 1 Changed: 1 Warnings: 0 mysql session 2 >update sbtest1 set k=2496417 where id=1;
Query OK, 1 row affected (1 min 12.48 sec)
Rows matched: 1 Changed: 1 Warnings: 0
  • session3
+-----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-----------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-----------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 507514 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 507412 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 0 | | NULL |
| 716 | root | localhost | test | Query | 46 | copy to tmp table | alter table sbtest1 drop index k_1 |
| 834 | root | localhost | test | Query | 5 | Waiting for table metadata lock | update sbtest1 set k=2496416 where id=1 |
| 835 | root | localhost | NULL | Sleep | 35 | | NULL |
| 836 | root | localhost | db1 | Query | 0 | starting | show processlist |
+-----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-----------------------------------------+ mysql session 3 >mysql session 3 >show processlist;
+-----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-----------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-----------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 507617 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 507515 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 1 | | NULL |
| 716 | root | localhost | test | Query | 27 | copy to tmp table | alter table sbtest1 add index k_1(k) |
| 834 | root | localhost | test | Query | 20 | Waiting for table metadata lock | update sbtest1 set k=2496417 where id=1 |
| 835 | root | localhost | NULL | Sleep | 138 | | NULL |
| 836 | root | localhost | db1 | Query | 0 | starting | show processlist |
+-----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-----------------------------------------+
7 rows in set (0.00 sec)
  • session4
[root@master test]# pwd
/mysqlData/data/test
[root@master test]# ls -ltr
total 2564148
-rw-r-----. 1 mysql mysql 61 Mar 13 10:45 db.opt
-rw-r-----. 1 mysql mysql 8632 Mar 13 11:25 sbtest2.frm
-rw-r-----. 1 mysql mysql 1224736768 Mar 13 11:26 sbtest2.ibd
-rw-r-----. 1 mysql mysql 8632 Mar 13 11:33 sbtest1.frm
-rw-r-----. 1 mysql mysql 1149239296 Mar 13 11:34 sbtest1.ibd
-rw-r-----. 1 mysql mysql 8632 Mar 13 11:35 #sql-663e_2cc.frm
-rw-r-----. 1 mysql mysql 251658240 Mar 13 11:35 #sql-663e_2cc.ibd
4.新特性online DDL
1.改成online DDL方式

并不影响DML

  • session1
mysql session 1 >set old_alter_table=0;
Query OK, 0 rows affected (0.00 sec) mysql session 1 >show variables like '%old_alter_table%';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| old_alter_table | OFF |
+-----------------+-------+
1 row in set (0.01 sec) mysql session 1 >alter table sbtest1 drop index k_1;
Query OK, 0 rows affected (0.07 sec)
Records: 0 Duplicates: 0 Warnings: 0
# 索引瞬间被删掉了 mysql session 1 >alter table sbtest1 add index k_1(k);
Query OK, 0 rows affected (9.45 sec)
Records: 0 Duplicates: 0 Warnings: 0
  • session2
mysql session 2 >update sbtest1 set k=2496417 where id=1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
# 添加二级索引的时候,也不会阻塞DML
  • session3
mysql session 3 >show processlist;
+------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 508423 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 508321 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 1 | | NULL |
| 716 | root | localhost | test | Query | 1 | altering table | alter table sbtest1 add index k_1(k) |
| 1328 | root | localhost | test | Sleep | 0 | | NULL |
| 1329 | root | localhost | db1 | Query | 0 | starting | show processlist |
+------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------+
6 rows in set (0.00 sec)
5.大事务之前的online DDL

alter之前有大事务执行,ddl会被阻塞,且后面的所有其他所有请求都会被阻塞

操作DDL之前最好还是先检查一下是否有大事务在进行中

  • session1
mysql session 1 >select * from sbtest1 where c='long select before alter';
Empty set (2.49 sec)
  • session2
mysql session 2 >alter table sbtest1 add index k_1(k);
Query OK, 0 rows affected (11.46 sec)
Records: 0 Duplicates: 0 Warnings: 0
  • session3
mysql session 3 >select * from sbtest1 where c='long select after alter execution but not complete';
Empty set (3.69 sec)
  • session4
mysql session 4 >show processlist;
+------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 508885 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 508783 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 1 | | NULL |
| 1328 | root | localhost | test | Query | 1 | Waiting for table metadata lock | alter table sbtest1 add index k_1(k) |
| 1329 | root | localhost | test | Query | 1 | Waiting for table metadata lock | select * from sbtest1 where c='long select after alter execution but not complete' |
| 1480 | root | localhost | test | Query | 2 | Sending data | select * from sbtest1 where c='long select before alter' |
| 1619 | root | localhost | NULL | Query | 0 | starting | show processlist |
+------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------------------------+
7 rows in set (0.00 sec)
6.添加删除列,不加inplace算法

允许ddl期间进行并发DML操作,并产生临时文件

  • session1,添加删除列
mysql session 1 >ALTER TABLE `sbtest2` \
-> ADD COLUMN `f_new_col1` int(11) NULL DEFAULT 0, \
-> ADD COLUMN `f_new_col2` varchar(32) NULL DEFAULT '' AFTER `f_new_col1`;
Query OK, 0 rows affected (51.50 sec)
Records: 0 Duplicates: 0 Warnings: 0 mysql session 1 >alter table sbtest2 drop column f_new_col1,\
-> drop column f_new_col2;
Query OK, 0 rows affected (48.64 sec)
Records: 0 Duplicates: 0 Warnings: 0
  • session2,进行update
mysql session 2 >update sbtest2 set c="update when add colomun ddl start" where c='33333';
Query OK, 0 rows affected (25.67 sec)
Rows matched: 0 Changed: 0 Warnings: 0 mysql session 2 >update sbtest2 set c="update when add colomun ddl start" where c='33333';
Query OK, 0 rows affected (6.15 sec)
Rows matched: 0 Changed: 0 Warnings: 0
  • session3,查询
mysql session 3 >select * from sbtest2 where c='select when add colomun ddl start';
Empty set (23.74 sec) mysql session 3 >select * from sbtest2 where c='select when add colomun ddl start';
Empty set (2.24 sec)
  • session4
mysql session 4 >show processlist;
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 523165 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 523063 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 0 | | NULL |
| 11000 | root | localhost | NULL | Query | 0 | starting | show processlist |
| 11021 | root | localhost | test | Query | 9 | altering table | ALTER TABLE `sbtest2`
ADD COLUMN `f_new_col1` int(11) NULL DEFAULT 0,
ADD COLUMN `f_new_col2` varc |
| 11110 | root | localhost | test | Query | 6 | updating | update sbtest2 set c="update when add colomun ddl start" where c='33333' |
| 11113 | root | localhost | test | Query | 5 | Sending data | select * from sbtest2 where c='select when add colomun ddl start' |
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
7 rows in set (0.00 sec) mysql session 4 >show processlist;
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 523338 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 523236 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 1 | | NULL |
| 11000 | root | localhost | NULL | Query | 0 | starting | show processlist |
| 11021 | root | localhost | test | Query | 22 | altering table | alter table sbtest2 drop column f_new_col1,
drop column f_new_col2 |
| 11110 | root | localhost | test | Query | 6 | updating | update sbtest2 set c="update when add colomun ddl start" where c='33333' |
| 11113 | root | localhost | test | Query | 2 | Sending data | select * from sbtest2 where c='select when add colomun ddl start' |
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------------------------------------------+
7 rows in set (0.00 sec)
  • session5,产生临时文件
[root@master test]# ls -ltr
total 4010036
-rw-r-----. 1 mysql mysql 61 Mar 13 10:45 db.opt
-rw-r-----. 1 mysql mysql 8632 Mar 13 11:56 sbtest1.frm
-rw-r-----. 1 mysql mysql 1275068416 Mar 13 11:57 sbtest1.ibd
-rw-r-----. 1 mysql mysql 8716 Mar 13 15:54 sbtest2.frm
-rw-r-----. 1 mysql mysql 1442840576 Mar 13 15:55 sbtest2.ibd
-rw-r-----. 1 mysql mysql 8632 Mar 13 15:57 #sql-663e_2b0d.frm
-rw-r-----. 1 mysql mysql 1388314624 Mar 13 15:58 #sql-ib87-1396920660.ibd
  • 删除的时候产生的临时文件比ibd更大
[root@master test]# ls -ltrh
total 3.9G
-rw-r-----. 1 mysql mysql 61 Mar 13 10:45 db.opt
-rw-r-----. 1 mysql mysql 8.5K Mar 13 11:56 sbtest1.frm
-rw-r-----. 1 mysql mysql 1.2G Mar 13 11:57 sbtest1.ibd
-rw-r-----. 1 mysql mysql 8.5K Mar 13 16:10 sbtest2.frm
-rw-r-----. 1 mysql mysql 1.3G Mar 13 16:11 sbtest2.ibd
-rw-r-----. 1 mysql mysql 8.6K Mar 13 16:12 #sql-663e_2b0d.frm
-rw-r-----. 1 mysql mysql 1.4G Mar 13 16:13 #sql-ib90-1396920666.ibd
7.修改列,加inplace算法(5.7)
  • 1).创建f_new_col2的时候长度是32
mysql session 1 >ALTER TABLE `sbtest2` \
-> ADD COLUMN `f_new_col1` int(11) NULL DEFAULT 0, \
-> ADD COLUMN `f_new_col2` varchar(32) NULL DEFAULT '' AFTER `f_new_col1`;
Query OK, 0 rows affected (1 min 4.21 sec)
Records: 0 Duplicates: 0 Warnings: 0
  • 2).更改f_new_col2长度为50
mysql session 1 >ALTER TABLE sbtest2 \
-> CHANGE f_new_col2 f_new_col2 varchar(50) NULL DEFAULT '', algorithm=inplace ;
Query OK, 0 rows affected (0.09 sec)
Records: 0 Duplicates: 0 Warnings: 0
  • 3).可以发现长度已经被更改
mysql session 1 >show create table sbtest2;
+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| sbtest2 | CREATE TABLE `sbtest2` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`k` int(10) unsigned NOT NULL DEFAULT '0',
`c` char(120) NOT NULL DEFAULT '',
`pad` char(60) NOT NULL DEFAULT '',
`f_new_col1` int(11) DEFAULT '0',
`f_new_col2` varchar(50) DEFAULT '',
PRIMARY KEY (`id`),
KEY `k_2` (`k`)
) ENGINE=InnoDB AUTO_INCREMENT=5000001 DEFAULT CHARSET=utf8 MAX_ROWS=1000000 |
+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
  • 4).再更改更长长度时不支持inplace,但是改成长度是50又支持,50改短或长都不支持?
mysql session 1 >ALTER TABLE sbtest2 \
-> CHANGE f_new_col2 f_new_col2 varchar(150) NULL DEFAULT '', algorithm=inplace ;
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY.
mysql session 1 >ALTER TABLE sbtest2 \
-> CHANGE f_new_col2 f_new_col2 varchar(32) NULL DEFAULT '', algorithm=inplace ;
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY.
mysql session 1 >ALTER TABLE sbtest2 \
-> CHANGE f_new_col2 f_new_col2 varchar(50) NULL DEFAULT '', algorithm=inplace ;
Query OK, 0 rows affected (0.01 sec)
Records: 0 Duplicates: 0 Warnings: 0 mysql session 1 >ALTER TABLE sbtest2 \
-> CHANGE f_new_col2 f_new_col2 varchar(32) NULL DEFAULT '', algorithm=inplace ;
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY.
mysql session 1 >ALTER TABLE sbtest2 \
-> CHANGE f_new_col2 f_new_col2 varchar(88) NULL DEFAULT '', algorithm=inplace ;
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY.

修改列类型应该要rebuild整张表

8.继续上面的修改列长度(5.7)

ddl堵塞DML,但是不阻塞select

  • 1).session1
mysql session 1 >ALTER TABLE sbtest2 \
-> CHANGE f_new_col2 f_new_col2 varchar(200) NULL DEFAULT '';
Query OK, 5000000 rows affected (1 min 28.96 sec)
Records: 5000000 Duplicates: 0 Warnings: 0
  • 2).session2
mysql session 2 >update sbtest2 set c="update when add colomun ddl start" where c='33333';
Query OK, 0 rows affected (1 min 34.01 sec)
Rows matched: 0 Changed: 0 Warnings: 0
  • 3).session3
mysql session 3 >select * from sbtest2 where c='select when add colomun ddl start';
Empty set (2.82 sec)
  • 4).session4
mysql session 4 >show processlist;
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------------------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 524786 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 524684 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 1 | | NULL |
| 11000 | root | localhost | NULL | Query | 0 | starting | show processlist |
| 11021 | root | localhost | test | Query | 3 | copy to tmp table | ALTER TABLE sbtest2
CHANGE f_new_col2 f_new_col2 varchar(200) NULL DEFAULT '' |
| 12186 | root | localhost | test | Query | 2 | Waiting for table metadata lock | update sbtest2 set c="update when add colomun ddl start" where c='33333' |
| 12189 | root | localhost | test | Query | 1 | Sending data | select * from sbtest2 where c='select when add colomun ddl start' |
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------------------------------------------------+
7 rows in set (0.01 sec)
  • 5).session5,copy肯定是产生临时文件
[root@master test]# ls -ltrh
total 3.2G
-rw-r-----. 1 mysql mysql 61 Mar 13 10:45 db.opt
-rw-r-----. 1 mysql mysql 8.5K Mar 13 11:56 sbtest1.frm
-rw-r-----. 1 mysql mysql 1.2G Mar 13 11:57 sbtest1.ibd
-rw-r-----. 1 mysql mysql 1.4G Mar 13 16:13 sbtest2.ibd
-rw-r-----. 1 mysql mysql 8.6K Mar 13 16:15 sbtest2.frm
-rw-r-----. 1 mysql mysql 8.6K Mar 13 16:21 #sql-663e_2b0d.frm
-rw-r-----. 1 mysql mysql 648M Mar 13 16:22 #sql-663e_2b0d.ibd

https://www.cnblogs.com/zengkefu/p/5674945.html

https://dev.mysql.com/doc/refman/5.6/en/upgrading-from-previous-series.html


OAK DDL

5.6之前使用第三方工具进行ddl

1.oak_online_alter_table前提条件
  • 1.单列唯一索引,联合索引和联合主键索引不可以,触发mysqlbug
  • 2.没有外键foreign key
  • 3.没有定义触发器,有的话也一棍打死
2.oak ddl流程
  • 1.确认表符合oak_online_alter_table的执行条件
  • 2.如果有触发器,备份触发器,删除触发器
  • 3.预估oak命令执行时间,表有1亿,1秒执行3000条,大概执行12个小时,计划时间执行
  • 4.进行新旧表数据校验,如果DDL改变了表字段类型,可能导致表数据变化
  • 5.表切换rename
  • 6.删除原表触发器
  • 7.删除表
3.check list功能
./run_bat IP onlineddl_check port dbname tabname
4.检查外键和触发器SQL
  • 1.information_schema.key_column_usage
  • 2.select * from information_schema.key_column_usage where table_schema=dbname and table_name=tabname and referenced_table_name is not null
  • 3.select * from information_schema.key_column_usage where referenced_table_schema=dbname and referenced_table_name=tabname;
5.安装openark-kit工具包
wget -O /etc/yum.repos.d/oak.repo http://download.opensuse.org/repositories/home:LenzGr/CentOS_7/home:LenzGr.repo
install openark-kit
6.操作过程
  • 1).构造测试数据
sysbench --test=insert.lua \
--oltp-table-size=500000 --mysql-table-engine=innodb --mysql-user=root --mysql-password=123456 \
--mysql-port=3306 --mysql-host=localhost --mysql-db=test --max-requests=0 --time=60 \
--oltp-tables-count=2 --report-interval=10 --threads=2 prepare
  • 2).检查是否有触发器
select trigger_schema,trigger_name,event_object_schema,event_object_table from information_schema.triggers where event_object_schema='test';
  • 3).执行online DDL
oak-online-alter-table -u root --ask-pass -S /usr/local/mysql/mysql.sock -d test -t sbtest1 -g new_sbtest1 -a "add last_update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,add key last_update_time(last_update_time)" --sleep=300 --skip-delete-pass

执行ddl的时候,依然可以进行ddl操作

root@master 22:34:  [test]> insert into sbtest1 values(10000000,10000,'AAAAA','BBBBB');
Query OK, 1 row affected (0.01 sec)
  • 4).执行校验
select sum(crc32(concat(ifnull(id,'NULL'),ifnull(k,'NULL')))) as sum \
from sbtest1 \
union all \
select sum(crc32(concat(ifnull(id,'NULL'),ifnull(k,'NULL')))) as sum \
from new_sbtest1; # select count(*) from sbtest1 union all select count(*) from new_sbtest1;
root@master 22:34: [test]> select count(*) from sbtest1 union all select count(*) from new_sbtest1;
+----------+
| count(*) |
+----------+
| 500001 |
| 291001 |
+----------+
2 rows in set (0.19 sec)
# desc sbtest1;
desc new_sbtest1;
  • 5).因为在原表增加了触发器,即使ddl完成之后,新的DML依然会插入到新表
root@master 22:34:  [test]> select trigger_schema,trigger_name,event_object_schema,event_object_table from information_schema.triggers where event_object_schema='test';
+----------------+----------------+---------------------+--------------------+
| trigger_schema | trigger_name | event_object_schema | event_object_table |
+----------------+----------------+---------------------+--------------------+
| test | sbtest1_AI_oak | test | sbtest1 |
| test | sbtest1_AU_oak | test | sbtest1 |
| test | sbtest1_AD_oak | test | sbtest1 |
+----------------+----------------+---------------------+--------------------+
3 rows in set (0.00 sec)
root@master 22:38: [test]> select count(*) from sbtest1 union all select count(*) from new_sbtest1;
+----------+
| count(*) |
+----------+
| 500001 |
| 500001 |
+----------+
2 rows in set (0.16 sec) root@master 22:38: [test]> insert into sbtest1 values(10000001,10000,'AAAAA','BBBBB');
Query OK, 1 row affected (0.00 sec) root@master 22:39: [test]> select count(*) from sbtest1 union all select count(*) from new_sbtest1;
+----------+
| count(*) |
+----------+
| 500002 |
| 500002 |
+----------+
2 rows in set (0.16 sec)
  • 6).表切换,更改数据字典
use test;
set names utf8;
rename table sbtest1 to old_sbtest1,new_sbtest1 to sbtest1;
  • 7).删除触发器和原表

drop trigger sbtest1_AI_oak;
drop trigger sbtest1_AU_oak;
drop trigger sbtest1_AD_oak;
drop table old_sbtest1; root@master 22:40: [test]> show tables;
+----------------+
| Tables_in_test |
+----------------+
| old_sbtest1 |
| sbtest1 |
| sbtest2 |
+----------------+
3 rows in set (0.00 sec) root@master 22:41: [test]> select trigger_schema,trigger_name,event_object_schema,event_object_table from information_schema.triggers where event_object_schema='test';
+----------------+----------------+---------------------+--------------------+
| trigger_schema | trigger_name | event_object_schema | event_object_table |
+----------------+----------------+---------------------+--------------------+
| test | sbtest1_AI_oak | test | old_sbtest1 |
| test | sbtest1_AU_oak | test | old_sbtest1 |
| test | sbtest1_AD_oak | test | old_sbtest1 |
+----------------+----------------+---------------------+--------------------+
3 rows in set (0.01 sec) root@master 22:41: [test]> drop trigger sbtest1_AI_oak;
Query OK, 0 rows affected (0.00 sec) root@master 22:41: [test]> drop trigger sbtest1_AU_oak;
Query OK, 0 rows affected (0.01 sec) root@master 22:41: [test]> drop trigger sbtest1_AD_oak;
Query OK, 0 rows affected (0.01 sec) root@master 22:41: [test]> drop table old_sbtest1;
Query OK, 0 rows affected (0.23 sec) root@master 22:41: [test]> show tables;
+----------------+
| Tables_in_test |
+----------------+
| sbtest1 |
| sbtest2 |
+----------------+
2 rows in set (0.00 sec)
7.用oak做Online DDL,如果rename那一步操作错了?在从库上rename了怎么恢复?(包含旧表删除和trigger删除).
  • 1)第一种情况加索引的Online DDL
  • 2)第二种情况加字段的Online DDL

发现rename错表了?

再做一次drop ddl

不是从主库进行ddl,选择从库ddl,从库rename错了,想恢复?

http://www.cnblogs.com/xuanzhi201111/p/4549250.html

http://www.cnblogs.com/xinysu/p/6732646.html#_lab2_4_3

mysql小白系列_13 Online DDL的更多相关文章

  1. mysql小白系列_10 mysql主从复制原理

    1.如何解决主从复制延迟的问题? (1)主从复制原理 http://www.cnblogs.com/jenvid/p/8410922.html 1.salve执行slave start,salve服务 ...

  2. mysql小白系列_07 锁与事务

    1.MySQL参数autocommit生产环境设1还是0?为什么? 2.MySQL参数tx_isolation生产环境上大多数是设什么值,为什么? 3.与MySQL锁相关的有哪些因素? 1.MySQL ...

  3. mysql小白系列_14 线上故障分析与排错

    1.重现故障5---线上执行update报错,并处理.(表结构和UPDATE语句自己构造,请给出详细步骤) 1)update故障出现ERROR 1206 (HY000): The total numb ...

  4. mysql小白系列_12 sysbench

    压测 1.查看机器负载load top - load average 1m 5m 15m cat /proc/loadavg 与CPU个数有关,1个load表示1个CPUcat /proc/cpuin ...

  5. mysql小白系列_11 MHA

    一.MHA是什么?能干什么的 (1)以Perl语言写的一套Mysql故障切换方案,一个脚本管理工具 (2)保障数据库的高可用性 (3)修复多个slave之间的差异日志,最终使所有的slave保持数据一 ...

  6. mysql小白系列_09 mysql性能优化关键点

    一 服务器参数调优,有哪些关键点? 1. 应用访问优化 优化方法 性能提升效果 优化成本 说明 减少数据访问能不访问就不访问-减少磁盘IO 1~1000 低 缓存服务器缓存mysql数据,Redis. ...

  7. mysql小白系列_08 zabbix添加自定义监控项items和触发器

    监控mysql存活 1.配置agent自定义参数 vi /usr/local/zabbix/etc/zabbix_agentd.conf Include=/usr/local/zabbix/etc/z ...

  8. mysql小白系列_08 zabbix3.2.6概念及部署

    一 zabbix功能简介 1.zabbix三大监控组件 zabbix server web gui database zabbix_server zabbix proxy agent client 2 ...

  9. mysql小白系列_06 备份与恢复 - mydumper与myloader

    http://blog.csdn.net/leshami/article/details/46815553 https://www.cnblogs.com/zhoujinyi/p/3423641.ht ...

随机推荐

  1. fseek 在以字符串模式打开的文件中工作不正常 [MSDN]

    For streams opened in text mode, fseek and _fseeki64 have limited use, because carriage return-linef ...

  2. GIT分布式版本控制

    1.1Git简介 linus 用C语言编写 2005年诞生 分布式版本管理系统 速度快,适合大规模,跨地区多人协同开发 Git不仅是一款开源的分布式版本控制系统,而且有其独特的功能特性,例如大多数的分 ...

  3. Libra教程之:Transaction的生命周期

    文章目录 Transaction的生命周期 提交一个Transaction 交易入链的详细过程 接收Transaction 和其他Validators共享这个Transaction 区块Proposi ...

  4. 计算5的n次幂html代码

    <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8&quo ...

  5. Linux下必知必会文件和目录

    转载于:https://blog.51cto.com/xiyuxingxia/2372712

  6. iOS9.2.1 App从AppStore上下载闪退问题

    首先这是小编的第一篇文章,我是一名做iOS开发的小白,出于爱好会更新发表些相关的技术文章,偶尔也会发些视频.恳请大家不要去嘲笑一个努力的人,要是做的不好请多多评论,反正我也不改. 好了!敲黑板!!说正 ...

  7. Linux下文件完整性监控工具Tripwire详解

    Tripwire 是目前最为著名的Unix下文件系统完整性检查的软件工具,这一软件采用的技术核心就是对每个要监控的文件产生一个数字签名,保留下来.当文件现在的数字签名与保留的数字签名不一致时,那么现在 ...

  8. 涉及secureCRT中文显示的一些设置

    1.secureCRT中文显示乱码: 如果你的linux本身是显示着中文的,可进行如下设置: 选项->会话选项 外观->字符编码改为UTF-8,确定即可 2.secureCRT中文横向显示 ...

  9. Excel 函数 常见错误

    excel公式出错的值. 常见错误值类型: #N/A #VALUE! #NAME? #REF! #NUM! #NULL! #N/A 找不到指定内容,比如查找根本不存在的内容,便显示该错误值 比如在使用 ...

  10. 一步一步教你PowerBI数据分析:制作客户RFM数据分析

    客户分析就是根据客户信息数据来分析客户特征,评估客户价值,从而为客户制订相应的营销策略与资源配置.通过合理.系统的客户分析,企业可以知道不同的客户有着什么样的需求,分析客户消费特征与商务效益的关系,使 ...