union all 优化案例
遇到个子查询嵌套 UNION ALL 的SQL语句很慢，谓词过滤条件不能内推进去，需要优化这段 UNION ALL这块的内容。
UNION ALL 慢SQL：
SELECT * FROM ((SELECT
jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001 AS jfw03001,
jfw03.jfw03002 AS jfw03002,
jfw03.jfw03003 AS jfw03003,
jfw03.jfw03004 AS jfw03004,
jfw03.jfw03005 AS jfw03005,
jfw03.jfw03006 AS jfw03006,
jfw03.jfw03007 AS jfw03007,
jfw03.remark AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM emc jfw03
LEFT JOIN emc old03
ON jfw03.emc_person_id =
old03.emc_person_id
WHERE jfw03.jfw03001 = old03.jfw03001
AND jfw03.handle_mark = '2'
AND old03.handle_mark IN ('1', '3'))
UNION ALL (SELECT jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001,
jfw03.jfw03002,
jfw03.jfw03003,
jfw03.jfw03004,
jfw03.jfw03005,
jfw03.jfw03006,
jfw03.jfw03007,
jfw03.remark,
NULL AS oldId,
NULL AS jfw03001pre,
NULL AS jfw03002pre,
NULL AS jfw03003pre,
NULL AS jfw03004pre,
NULL AS jfw03005pre,
NULL AS jfw03006pre,
NULL AS jfw03007pre,
NULL AS remarkpre
FROM emc jfw03
WHERE jfw03.handle_mark = '2'
AND NOT EXISTS
(SELECT 1
FROM emc old03
WHERE jfw03.emc_person_id =
old03.emc_person_id
AND jfw03.jfw03001 = old03.jfw03001
AND old03.handle_mark IN ('1', '3')))
UNION ALL (SELECT NULL AS newid,
old03.emc_person_id AS bizPersonOid,
NULL AS jfw03001,
NULL AS jfw03002,
NULL AS jfw03003,
NULL AS jfw03004,
NULL AS jfw03005,
NULL AS jfw03006,
NULL AS jfw03007,
NULL AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM emc old03
WHERE old03.handle_mark IN ('1', '3')
AND NOT EXISTS
(SELECT 1
FROM emc jfw03
WHERE jfw03.emc_person_id =
old03.emc_person_id
AND jfw03.jfw03001 = old03.jfw03001
AND jfw03.handle_mark = '2'))) jfw03;
执行计划：
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------
Append (cost=0.69..7253737.08 rows=49711 width=274) (actual time=0.252..60906.535 rows=3291407 loops=1)
-> Nested Loop (cost=0.69..2753734.17 rows=49709 width=274) (actual time=0.251..16535.465 rows=49558 loops=1)
-> Seq Scan on emc jfw03 (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.010..5103.042 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 8885720
-> Index Scan using idx_1_2_3_emc on emc old03 (cost=0.69..2.12 rows=1 width=139) (actual time=0.010..0.010 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
-> Nested Loop Anti Join (cost=0.69..1864694.18 rows=1 width=399) (actual time=0.032..16606.236 rows=1065808 loops=1)
-> Seq Scan on emc jfw03_1 (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.013..5171.111 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 8885720
-> Index Only Scan using idx_1_2_3_emc on emc old03_1 (cost=0.69..1.33 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
Heap Fetches: 0
-> Nested Loop Anti Join (cost=0.69..2634563.07 rows=1 width=399) (actual time=0.033..27638.050 rows=2176041 loops=1)
-> Seq Scan on emc old03_2 (cost=0.00..347258.58 rows=2205308 width=139) (actual time=0.014..5510.884 rows=2225077 loops=1)
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 7774923
-> Index Only Scan using idx_1_2_3_emc on emc jfw03_2 (cost=0.69..1.03 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
Heap Fetches: 0
Planning Time: 0.318 ms
Execution Time: 60987.916 ms
(27 行记录)
加个索引看看速度
CREATE INDEX idx_emc_1 ON emc ( handle_mark );
加索引后执行计划
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------
Append (cost=21059.67..7011218.65 rows=49969 width=278) (actual time=115.683..46578.675 rows=3291407 loops=1)
-> Nested Loop (cost=21059.67..2655195.15 rows=49967 width=278) (actual time=115.682..11939.294 rows=49558 loops=1)
-> Bitmap Heap Scan on emc jfw03 (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=115.458..417.906 rows=1114280 loops=1)
Recheck Cond: (handle_mark = '2'::bpchar)
Heap Blocks: exact=221151
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..20778.90 rows=1120329 width=0) (actual time=92.887..92.887 rows=1114280 loops=1)
Index Cond: (handle_mark = '2'::bpchar)
-> Index Scan using idx_1_2_3_emc on emc old03 (cost=0.69..2.13 rows=1 width=141) (actual time=0.010..0.010 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
-> Nested Loop Anti Join (cost=21059.67..1766155.15 rows=1 width=401) (actual time=127.721..11690.213 rows=1065808 loops=1)
-> Bitmap Heap Scan on emc jfw03_1 (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=127.640..423.128 rows=1114280 loops=1)
Recheck Cond: (handle_mark = '2'::bpchar)
Heap Blocks: exact=221151
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..20778.90 rows=1120329 width=0) (actual time=105.117..105.117 rows=1114280 loops=1)
Index Cond: (handle_mark = '2'::bpchar)
-> Index Only Scan using idx_1_2_3_emc on emc old03_1 (cost=0.69..1.34 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
Heap Fetches: 0
-> Nested Loop Anti Join (cost=39323.15..2589118.81 rows=1 width=401) (actual time=201.187..22828.908 rows=2176041 loops=1)
-> Bitmap Heap Scan on emc old03_2 (cost=39322.47..289365.69 rows=2222658 width=141) (actual time=201.135..603.170 rows=2225077 loops=1)
Recheck Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
Heap Blocks: exact=222258
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..38766.81 rows=2222658 width=0) (actual time=179.770..179.770 rows=2225077 loops=1)
Index Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
-> Index Only Scan using idx_1_2_3_emc on emc jfw03_2 (cost=0.69..1.02 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
Heap Fetches: 0
Planning Time: 0.373 ms
Execution Time: 46666.296 ms
(33 行记录)
快了14秒，没达到目的效果，无解，只能改写SQL，可以改 FULL JOIN。
等价改写：
SELECT * FROM (
with x as (SELECT * FROM emc where handle_mark IN ('1', '3','2'))
SELECT
jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001 AS jfw03001,
jfw03.jfw03002 AS jfw03002,
jfw03.jfw03003 AS jfw03003,
jfw03.jfw03004 AS jfw03004,
jfw03.jfw03005 AS jfw03005,
jfw03.jfw03006 AS jfw03006,
jfw03.jfw03007 AS jfw03007,
jfw03.remark AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM (SELECT * FROM x WHERE handle_mark IN ('2')) jfw03
full join (SELECT * FROM x WHERE handle_mark IN ('1', '3')) old03
ON jfw03.emc_person_id = old03.emc_person_id AND jfw03.jfw03001 = old03.jfw03001);
等价改写计划：
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------
Merge Full Join (cost=10000479890.42..10000480406.20 rows=33430 width=1844) (actual time=9433.530..10954.254 rows=3291407 loops=1)
Merge Cond: ((x.emc_person_id = x_1.emc_person_id) AND (x.jfw03001 = x_1.jfw03001))
CTE x
-> Bitmap Heap Scan on emc (cost=57545.45..325771.51 rows=3342986 width=145) (actual time=268.766..721.294 rows=3339357 loops=1)
Recheck Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
Heap Blocks: exact=222260
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..56709.71 rows=3342986 width=0) (actual time=246.952..246.953 rows=3339357 loops=1)
Index Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
-> Sort (cost=76389.65..76431.43 rows=16715 width=924) (actual time=3818.370..4021.726 rows=1114280 loops=1)
Sort Key: x.emc_person_id, x.jfw03001
Sort Method: quicksort Memory: 345133kB
-> CTE Scan on x (cost=0.00..75217.18 rows=16715 width=924) (actual time=268.780..1391.594 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 2225077
-> Sort (cost=77729.26..77812.83 rows=33430 width=924) (actual time=5615.144..5949.562 rows=2225599 loops=1)
Sort Key: x_1.emc_person_id, x_1.jfw03001
Sort Method: quicksort Memory: 676056kB
-> CTE Scan on x x_1 (cost=0.00..75217.18 rows=33430 width=924) (actual time=0.007..360.353 rows=2225077 loops=1)
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1114280
Planning Time: 0.138 ms
Execution Time: 11101.829 ms
(22 行记录)
改写后11s就能跑出结果，符合性能优化目标。
这条改写SQL留个坑，不完全等价，完全等价的没放出来，有兴趣的同学可以思考下，自己做做实验。
union all 优化案例的更多相关文章
- Hive优化案例
1.Hadoop计算框架的特点 数据量大不是问题,数据倾斜是个问题. jobs数比较多的作业效率相对比较低,比如即使有几百万的表,如果多次关联多次汇总,产生十几个jobs,耗时很长.原因是map re ...
- 数据库优化案例——————某市中心医院HIS系统
记得在自己学习数据库知识的时候特别喜欢看案例,因为优化的手段是容易掌握的,但是整体的优化思想是很难学会的.这也是为什么自己特别喜欢看案例,今天也开始分享自己做的优化案例. 最近一直很忙,博客产出也少的 ...
- mysql优化案例
MySQL优化案例 Mysql5.1大表分区效率测试 Mysql5.1大表分区效率测试MySQL | add at 2009-03-27 12:29:31 by PConline | view:60, ...
- SQL 优化案例 1
create or replace procedure SP_GET_NEWEST_CAPTCHA( v_ACCOUNT_ID in VARCHAR2, --接收短信的手机号 v_Tail_num i ...
- 老李案例分享：Weblogic性能优化案例
老李案例分享:Weblogic性能优化案例 POPTEST的测试技术交流qq群:450192312 网站应用首页大小在130K左右,在之前的测试过程中,其百用户并发的平均响应能力在6.5秒,性能优化后 ...
- 数据库优化案例——————某知名零售企业ERP系统
写在前面 记得在自己学习数据库知识的时候特别喜欢看案例,因为优化的手段是容易掌握的,但是整体的优化思想是很难学会的.这也是为什么自己特别喜欢看案例,今天也分享自己做的优化案例. 之前分享过OA系统.H ...
- Spark集群之yarn提交作业优化案例
Spark集群之yarn提交作业优化案例 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.启动Hadoop集群 1>.自定义批量管理脚本 [yinzhengjie@s101 ...
- MySQL参数优化案例
环境介绍 优化层级与指导思想 优化过程 最小化安装情况下的性能表现 优化innodb_buffer_pool_size 优化innodb_log_files_in_group&innodb_l ...
- MySQL的索引单表优化案例分析
建表 建立本次优化案例中所需的数据库及数据表 CREATE DATABASE db0206; USE db0206; CREATE TABLE `db0206`.`article`( `id` INT ...
- SQL 优化案例
create or replace procedure SP_GET_NEWEST_CAPTCHA( v_ACCOUNT_ID in VARCHAR2, --接收短信的手机号 v_Tail_num i ...
随机推荐
- MAC使用XQuartz调用图形界面
DBA经常遇到需要调用图形的操作,通常Windows用户习惯使用Xmanager这类软件,MAC用户习惯使用XQuartz,之前版本系统会自带,现在需要自行下载. 比如在 https://www.xq ...
- 小知识：Flex ASM特性对集群资源显示的影响
有客户咨询,认为19c RAC集群资源状态和11g RAC大不一样,比如在他们的19c集群,也是只部署2节点,却显示3个资源状态,其中第三个还是offline状态,担心是否有影响. 实际上这和Flex ...
- Oracle配置DCD避免会话被防火墙强制断开
今天有客户反馈应用测试连接Oracle数据库的会话半小时左右未做操作就会被中断,报错ORA-3113,询问数据库是否有这样的超时设置,能否取消掉这个限制? 登上环境查看监听日志发现连接的IP和数据库I ...
- 蓝鲸单机离线部署：app_mgr组件安装失败解决
之前在腾讯蓝鲸智云-单机离线部署测试中,遇到了几个安装问题,本文记录下3.2 app_mgr组件安装失败 的解决过程,因为这个问题卡了很久(可能也是因为笔者对python相关知识和蓝鲸产品不够熟悉), ...
- IntelliJ IDEA 在电脑上安装多个JDK 切换的方法
在电脑上来回切换多个版本的JDK进行开发,方法很简单: 1.下载jdk 下载的时候不要下载安装包,而是下载zip包,这样直接解压就可,不污染注册表,难引起其他问题 2.解压后 把JDK配置到IDEA里 ...
- Kafka的ACK机制
Kafka的ack机制,指的是producer的消息发送确认机制,这直接影响到Kafka集群的吞吐量和消息可靠性.而吞吐量和可靠性就像硬币的两面,两者不可兼得,只能平衡. ACK有3个可选值,分别是1 ...
- logging --- Python 的日志记录工具
logging --- Python 的日志记录工具 源代码: Lib/logging/__init__.py Important 此页面仅包含 API 参考信息.教程信息和更多高级用法的讨论,请参阅 ...
- 微信小程序云开发项目-个人待办事项-03【主页】模块开发
上一篇: 微信小程序云开发项目-个人待办事项-02今日模块开发 https://blog.csdn.net/IndexMan/article/details/124497893 模块开发步骤 本篇介绍 ...
- 如何用Apipost校验响应结果
数据校验的意义 我们可以通过 json-schema 预先定义接口的数据返回格式,当接口完成后,我们可以通过匹配 实际响应结果 和 预先定义的接口格式 ,来发现接口问题.如下图: 数据校验的设置 我们 ...
- 类型注解Callable
from collections.abc import Callable """ Callable[[ParamType1, ParamType2], ReturnTyp ...