一、ICP( Index_Condition_Pushdown)

对 where 中过滤条件的处理,根据索引使用情况分成了三种:(何登成)index key, index filter, table filter

如果WHERE条件可以使用索引,MySQL 会把这部分过滤操作放到存储引擎层,存储引擎通过索引过滤,把满足的行从表中读取出。ICP能减少Server层访问存储引擎的次数和引擎层访问基表的次数。

  • session级别设置:set optimizer_switch="index_condition_pushdown=on
  • 对于InnoDB表,ICP只适用于辅助索引

  • 当使用ICP优化时,执行计划的Extra列显示Using index condition提示

  • 不支持主建索引的ICP(对于Innodb的聚集索引,完整的记录已经被读取到Innodb Buffer,此时使用ICP并不能降低IO操作)

  • 当 SQL 使用覆盖索引时但只检索部分数据时,ICP 无法使用

  • ICP的加速效果取决于在存储引擎内通过ICP筛选掉的数据的比例

index_condition_pushdown会大大减少行锁的个数,如select for update, 因为行锁是在引擎层的

例如:

现在的索引

show index from sm_performance_all;
+--------------------+------------+-------------------------------+--------------+----------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------------------+------------+-------------------------------+--------------+----------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| sm_performance_all | 0 | PRIMARY | 1 | id | A | 40527 | NULL | NULL | | BTREE | | |
| sm_performance_all | 1 | FK_a9t29a4b2af1vfny1j2minc1x | 1 | company_id | A | 316 | NULL | NULL | YES | BTREE | | |
| sm_performance_all | 1 | FK_n3ng4a5qju19fw8qy4uskp4g1 | 1 | bill_id | A | 21532 | NULL | NULL | YES | BTREE | | |
| sm_performance_all | 1 | FK_eb13u3xwslt9t7wwuycg7vha6 | 1 | car_id | A | 16794 | NULL | NULL | YES | BTREE | | |
| sm_performance_all | 1 | FK_2bfhskvklf6mdk557tc3yy3y1 | 1 | commission_entity_id | A | 177 | NULL | NULL | YES | BTREE | | |
| sm_performance_all | 1 | FK_6fr5ib5iyjyu155dncmc48cwr | 1 | member_card_id | A | 34 | NULL | NULL | YES | BTREE | | |
| sm_performance_all | 1 | FK_93p22vcog266wa82i44a6m18b | 1 | user_id | A | 483 | NULL | NULL | YES | BTREE | | |
| sm_performance_all | 1 | FK_p6nc7l6ewnkcpm2y4o3wct81r | 1 | member_card_bill_id | A | 4 | NULL | NULL | YES | BTREE | | |
| sm_performance_all | 1 | billId_userId_memberCarBillId | 1 | bill_id | A | 24194 | NULL | NULL | YES | BTREE | | |
| sm_performance_all | 1 | billId_userId_memberCarBillId | 2 | user_id | A | 25688 | NULL | NULL | YES | BTREE | | |
| sm_performance_all | 1 | billId_userId_memberCarBillId | 3 | member_card_bill_id | A | 25946 | NULL | NULL | YES | BTREE | | |
+--------------------+------------+-------------------------------+--------------+----------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
11 rows in set (0.00 sec)

现在的语句执行情况

 explain select * from sm_performance_all p where p.date_created>'2018-01-01'  and p.date_created< '2018-02-01' and p.type=0;
+----+-------------+-------+------------+------+---------------+------+---------+------+-------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+------+---------+------+-------+----------+-------------+
| 1 | SIMPLE | p | NULL | ALL | NULL | NULL | NULL | NULL | 40527 | 1.11 | Using where |
+----+-------------+-------+------------+------+---------------+------+---------+------+-------+----------+-------------+
1 row in set, 1 warning (0.00 sec)
添加索引后
ALTER TABLE sm_performance_all add index date_created_type(date_created, type );
 explain select * from sm_performance_all p where p.date_created>'2018-01-01'  and p.date_created< '2018-02-01' and p.type=0;
+----+-------------+-------+------------+-------+-------------------+-------------------+---------+------+------+----------+-----------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+-------------------+-------------------+---------+------+------+----------+-----------------------+
| 1 | SIMPLE | p | NULL | range | date_created_type | date_created_type | 6 | NULL | 1 | 10.00 | Using index condition |
+----+-------------+-------+------------+-------+-------------------+-------------------+---------+------+------+----------+-----------------------+
1 row in set, 1 warning (0.00 sec)

二、MRR(Multi-Range Read )

随机 IO 转化为顺序 IO 以降低查询过程中 IO 开销的一种手段,这对IO-bound类型的SQL语句性能带来极大的提升。

MRR can be used for InnoDB and MyISAM tables for index range scans and equi-join operations.

  1. A portion of the index tuples are accumulated in a buffer.

  2. The tuples in the buffer are sorted by their data row ID.

  3. Data rows are accessed according to the sorted index tuple sequence.

上述的SQL语句需要根据辅助索引date_created_type进行查询,但是由于要求得到的是表中所有的列,因此需要回表进行读取。而这里就可能伴随着大量的随机I/O。这个过程如下图所示:

而MRR的优化在于,并不是每次通过辅助索引就回表去取记录,而是将其rowid给缓存起来,然后对rowid进行排序后,再去访问记录,这样就能将随机I/O转化为顺序I/O,从而大幅地提升性能。这个过程如下所示:

然而,在MySQL当前版本中,基于成本的算法过于保守,导致大部分情况下优化器都不会选择MRR特性。为了确保优化器使用mrr特性,请执行下面的SQL语句:

set optimizer_switch='mrr=on,mrr_cost_based=off';

读取全部字段时

 explain select * from sm_performance_all p where p.date_created>'2018-01-01'  and p.date_created< '2018-02-01' and p.type=0;
+----+-------------+-------+------------+-------+-------------------+-------------------+---------+------+------+----------+----------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+-------------------+-------------------+---------+------+------+----------+----------------------------------+
| 1 | SIMPLE | p | NULL | range | date_created_type | date_created_type | 6 | NULL | 1 | 10.00 | Using index condition; Using MRR |
+----+-------------+-------+------------+-------+-------------------+-------------------+---------+------+------+----------+----------------------------------+
1 row in set, 1 warning (0.00 sec)

只读取部分字段时:

读取外键

explain select car_id from sm_performance_all p where p.date_created>'2018-01-01'  and p.date_created< '2018-02-01' and p.type=0;
+----+-------------+-------+------------+-------+-------------------+-------------------+---------+------+------+----------+----------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+-------------------+-------------------+---------+------+------+----------+----------------------------------+
| 1 | SIMPLE | p | NULL | range | date_created_type | date_created_type | 6 | NULL | 1 | 10.00 | Using index condition; Using MRR |
+----+-------------+-------+------------+-------+-------------------+-------------------+---------+------+------+----------+----------------------------------+
1 row in set, 1 warning (0.00 sec)

读取主键
explain select id from sm_performance_all p where p.date_created>'2018-01-01' and p.date_created< '2018-02-01' and p.type=0;
+----+-------------+-------+------------+-------+-------------------+-------------------+---------+------+------+----------+--------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+-------------------+-------------------+---------+------+------+----------+--------------------------+
| 1 | SIMPLE | p | NULL | range | date_created_type | date_created_type | 6 | NULL | 1 | 10.00 | Using where; Using index |
+----+-------------+-------+------------+-------+-------------------+-------------------+---------+------+------+----------+--------------------------+
1 row in set, 1 warning (0.00 sec)

For MRR, a storage engine uses the value of the read_rnd_buffer_size system variable as a guideline for how much memory it can allocate for its buffer.

默认256KB

 show GLOBAL VARIABLES like '%buffer_size';
+-------------------------+----------+
| Variable_name | Value |
+-------------------------+----------+
| bulk_insert_buffer_size | 8388608 |
| innodb_log_buffer_size | 16777216 |
| innodb_sort_buffer_size | 1048576 |
| join_buffer_size | 262144 |
| key_buffer_size | 8388608 |
| myisam_sort_buffer_size | 8388608 |
| preload_buffer_size | 32768 |
| read_buffer_size | 131072 |
| read_rnd_buffer_size | 262144 |
| sort_buffer_size | 262144 |
+-------------------------+----------+
10 rows in set (0.00 sec)

三、表连接实现方式

3.1 Nested Loop Join

将驱动表/外部表的结果集作为循环基础数据,然后循环该结果集,每次获取一条数据作为下一个表的过滤条件查询数据,然后合并结果,获取结果集返回给客户端。Nested-Loop一次只将一行传入内层循环, 所以外层循环(的结果集)有多少行, 内存循环便要执行多少次,效率非常差。

 EXPLAIN SELECT * from sm_performance_all  p LEFT JOIN sm_bill b ON p.bill_id > b.car_id where p.company_id>1024;
+----+-------------+-------+------------+-------+------------------------------+------------------------------+---------+------+--------+----------+------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+------------------------------+------------------------------+---------+------+--------+----------+------------------------------------------------+
| 1 | SIMPLE | p | NULL | range | FK_a9t29a4b2af1vfny1j2minc1x | FK_a9t29a4b2af1vfny1j2minc1x | 9 | NULL | 20263 | 100.00 | Using index condition; Using MRR |
| 1 | SIMPLE | b | NULL | ALL | car_id_idx | NULL | NULL | NULL | 738383 | 100.00 | Range checked for each record (index map: 0x2) |
+----+-------------+-------+------------+-------+------------------------------+------------------------------+---------+------+--------+----------+------------------------------------------------+
2 rows in set, 1 warning (0.00 sec)


3.2 Block Nested-Loop Join

将外层循环的行/结果集存入join buffer, 内层循环的每一行与整个buffer中的记录做比较,从而减少内层循环的次数。主要用于当被join的表上无索引。

CREATE TABLE t1 (a int PRIMARY KEY, b int);
CREATE TABLE t2 (a int PRIMARY KEY, b int);
INSERT INTO t1 VALUES (1,2), (2,1), (3,2), (4,3), (5,6), (6,5), (7,8), (8,7), (9,10);
INSERT INTO t2 VALUES (3,0), (4,1), (6,4), (7,5); EXPLAIN
SELECT * FROM t1 LEFT JOIN t2 ON t1.a = t2.a WHERE t2.b <= t1.a AND t1.a <= t1.b; +----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+----------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+----------------------------------------------------+
| 1 | SIMPLE | t1 | NULL | ALL | PRIMARY | NULL | NULL | NULL | 9 | 33.33 | Using where |
| 1 | SIMPLE | t2 | NULL | ALL | PRIMARY | NULL | NULL | NULL | 4 | 25.00 | Using where; Using join buffer (Block Nested Loop) |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+----------------------------------------------------+
2 rows in set, 1 warning (0.00 sec)


3.3 Batched Key Access

当被join的表能够使用索引时,就先好顺序,然后再去检索被join的表。对这些行按照索引字段进行排序,因此减少了随机IO。如果被Join的表上没有索引,则使用老版本的BNL策略。

参考:

mysql reference :  multi-range read

insideMysql :  Mysql join算法与调优

关于Mysql 的 ICP、MRR、BKA等特性的更多相关文章

  1. MySQL Index--BNL/ICP/MRR/BKA

    MySQL关联查询算法: BNL(Block Nested-Loop) ICP(Index Condition Pushdown) MRR(Multi-Range Read) BKA(Batched ...

  2. MySQL--BNL/ICP/MRR/BKA

    #======================================================##MySQL关联查询算法:BNL(Block Nested-Loop)ICP(Index ...

  3. MySQL · 特性分析 · 优化器 MRR & BKA【转】

    MySQL · 特性分析 · 优化器 MRR & BKA 上一篇文章咱们对 ICP 进行了一次全面的分析,本篇文章小编继续为大家分析优化器的另外两个选项: MRR & batched_ ...

  4. MySQL 8.0.2复制新特性(翻译)

    译者:知数堂星耀队 MySQL 8.0.2复制新特性 MySQL 8 正在变得原来越好,而且这也在我们MySQL复制研发团队引起了一阵热潮.我们一直致力于全面提升MySQL复制,通过引入新的和一些有趣 ...

  5. PostgreSQL 和 MySQL 在用途、好处、特性和特点上的异同

    PostgreSQL 和 MySQL 在用途.好处.特性和特点上的异同. PostgreSQL 和 MySQL 是将数据组织成表的关系数据库.这些表可以根据每个表共有的数据链接或关联.关系数据库使您的 ...

  6. 【mysql】关于ICP、MRR、BKA等特性

    一.Index Condition Pushdown(ICP) Index Condition Pushdown (ICP)是mysql使用索引从表中检索行数据的一种优化方式,从mysql5.6开始支 ...

  7. MySQL中有关icp mrr和bka的特性

    文辉考我的问题,有关这三个的特性,如果在面试过程中,个人见解可以答以下 icp MyQL数据库会在取出索引的同时,判断是否进行WHERE条件过滤,也就是把WHERE的部分过滤操作放在存储引擎层,在某些 ...

  8. ICP、MRR、BKA等特性

    一.Index Condition Pushdown(ICP) Index Condition Pushdown (ICP)是 mysql 使用索引从表中检索行数据的一种优化方式,从mysql5.6开 ...

  9. MRR,BKA,ICP相关

    MRR Multi-Range Read,多范围读,5.6以上版本开始支持 工作原理&优化效果: 将查询到的辅助索引结果放在一个缓冲(read_rnd_buffer_size = 4M)中 将 ...

随机推荐

  1. HAVANA 团队简介

    在Ensembl 下载的gtf 文件中,会有一部分来源自 HAVANA havana 的全称叫做 human  and vertebrate analysis and annotation, 是sag ...

  2. python 迭代器模式

    迭代器模式 迭代器模式(Iterator Pattern)是 Java 和 .Net 编程环境中非常常用的设计模式.这种模式用于顺序访问集合对象的元素,不需要知道集合对象的底层表示. 迭代器模式属于行 ...

  3. 04单例模式Singleton

    一.什么是单例模式 单例模式是一种对象创建型模式,使用单例模式, 可以保证为一个类只生成唯一的实例对象.也就是说, 在整个程序空间中,该类只存在一个实例对象. 其实,GoF对单例模式的定义是:保证一个 ...

  4. 网络编程 -- RPC实现原理 -- Netty -- 迭代版本V1 -- 入门应用

    网络编程 -- RPC实现原理 -- 目录 啦啦啦 V1——Netty入门应用 Class : NIOServerBootStrap package lime.pri.limeNio.netty.ne ...

  5. 除了/etc/init.d/加启动脚本 或者在/etc/rc.local中加启动命令,还可以通过crontab来完成服务器重启后自动启动服务的操作

    @reboot /bin/sh /opt/soft/percona/bin/mysqld_safe --defaults-file=/mnt/perconadata/my.cnf --basedir= ...

  6. 转载->CPU的内部架构和工作原理

    CPU的内部架构和工作原理 本片博客转自:http://www.cnblogs.com/onepixel/p/8724526.html  感谢博主分享! 内部架构 CPU 的根本任务就是执行指令,对计 ...

  7. ALTER SEQUENCE 导致 REPLICAT 延时

    1.查看OGG线程状态 GGSCI (klcoredb-node1) 46> info all Program Status Group Lag at Chkpt Time Since Chkp ...

  8. Docker集群管理portainer的使用

    1.Slave主机docker需要开启2375端口 ubuntu: sudo vim /etc/default/docker DOCKER_OPTS="-H=unix:///var/run/ ...

  9. 使用原生Java代码生成可执行Jar包

    最近想做一个功能,就是把我们编译后的字节码及其资源文件打包成一个可执行的jar包,在装有jre的机器上双击就能运行. 首先是我们需要选择哪些字节码和文件需要打包到文件中,这个我们用JFileChoos ...

  10. 10.5Djang admin 管理工具

    2018-10-5 17:30:57 Django admin 管理工具  参考连接: https://www.cnblogs.com/yuanchenqi/articles/8323452.html ...