hive高阶1--sql和hive语句执行顺序、explain查看执行计划、group by生成MR
hive语句执行顺序
msyql语句执行顺序
代码写的顺序：
select ... from... where.... group by... having... order by..
或者
from ... select ...
代码的执行顺序：
from... where...group by... having.... select ... order by...
hive 语句执行顺序
大致顺序
from... where.... select...group by... having ... order by...
explain查看执行计划
hive语句和mysql都可以通过explain查看执行计划，这样就可以查看执行顺序，比如代码
explain
select city,ad_type,device,sum(cnt) as cnt
from tb_pmp_raw_log_basic_analysis
where day = '2016-05-28' and type = 0 and media = 'sohu' and (deal_id = '' or deal_id = '-' or deal_id is NULL)
group by city,ad_type,device显示执行计划如下
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: tb_pmp_raw_log_basic_analysis
Statistics: Num rows: 8195357 Data size: 580058024 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (((deal_id = '') or (deal_id = '-')) or deal_id is null) (type: boolean)
Statistics: Num rows: 8195357 Data size: 580058024 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: city (type: string), ad_type (type: string), device (type: string), cnt (type: bigint)
outputColumnNames: city, ad_type, device, cnt
Statistics: Num rows: 8195357 Data size: 580058024 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: sum(cnt)
keys: city (type: string), ad_type (type: string), device (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 8195357 Data size: 580058024 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string)
sort order: +++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string)
Statistics: Num rows: 8195357 Data size: 580058024 Basic stats: COMPLETE Column stats: NONE
value expressions: _col3 (type: bigint)
Reduce Operator Tree:
Group By Operator
aggregations: sum(VALUE._col0)
keys: KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 4097678 Data size: 290028976 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 4097678 Data size: 290028976 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 4097678 Data size: 290028976 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: -1具体介绍如下
**stage1的map阶段**
TableScan：from加载表，描述中有行数和大小等
Filter Operator：where过滤条件筛选数据，描述有具体筛选条件和行数、大小等
Select Operator：筛选列，描述中有列名、类型，输出类型、大小等。
Group By Operator：分组，描述了分组后需要计算的函数，keys描述用于分组的列，outputColumnNames为输出的列名，可以看出列默认使用固定的别名_col0，以及其他信息
Reduce Output Operator：map端本地的reduce，进行本地的计算，然后按列映射到对应的reduce
**stage1的reduce阶段Reduce Operator Tree**
Group By Operator：总体分组，并按函数计算。map计算后的结果在reduce端的合并。描述类似。mode: mergepartial是说合并map的计算结果。map端是hash映射分组
Select Operator：最后过滤列用于输出结果
File Output Operator：输出结果到临时文件中，描述介绍了压缩格式、输出文件格式。
stage0第二阶段没有，这里可以实现limit 100的操作。
总结
1，每个stage都是一个独立的MR，复杂的hql语句可以产生多个stage，可以通过执行计划的描述，看看具体步骤是什么。
2，执行计划有时预测数据量，不是真实运行，可能不准确
group by的MR
hive语句最好写子查询嵌套，这样分阶段的导入数据，可以逐步减少数据量。但可能会浪费时间。所以需要设计好。
group by本身也是一种数据筛选，可以大量减少数据，尤其用于去重等方面，功效显著。但group by产生MR有时不可控，不知道在哪个阶段更好。尤其，map端本地的reduce减少数据有很大作用。
尤其，hadoop的MR不患寡而患不均。数据倾斜将是MR计算的最大瓶颈。hive中可以设置分区、桶、distribute by等来控制分配数据给Reduce。
那么，group by生成MR是否可以优化呢?
下面两端代码，可以对比一下，
代码1
explain
select advertiser_id,crt_id,ad_place_id,channel,ad_type,rtb_type,media,count(1) as cnt
from (
select
split(all,'\\\\|~\\\\|')[41] as advertiser_id,
split(all,'\\\\|~\\\\|')[11] as crt_id,
split(all,'\\\\|~\\\\|')[8] as ad_place_id,
split(all,'\\\\|~\\\\|')[34] as channel,
split(all,'\\\\|~\\\\|')[42] as ad_type,
split(all,'\\\\|~\\\\|')[43] as rtb_type,
split(split(all,'\\\\|~\\\\|')[5],'/')[1] as media
from tb_pmp_raw_log_bid_tmp tb
) a
group by advertiser_id,crt_id,ad_place_id,channel,ad_type,rtb_type,media;代码2
explain
select
split(all,'\\\\|~\\\\|')[41] as advertiser_id,
split(all,'\\\\|~\\\\|')[11] as crt_id,
split(all,'\\\\|~\\\\|')[8] as ad_place_id,
split(all,'\\\\|~\\\\|')[34] as channel,
split(all,'\\\\|~\\\\|')[42] as ad_type,
split(all,'\\\\|~\\\\|')[43] as rtb_type,
split(split(all,'\\\\|~\\\\|')[5],'/')[1] as media
from tb_pmp_raw_log_bid_tmp tb
group by split(all,'\\\\|~\\\\|')[41],split(all,'\\\\|~\\\\|')[11],split(all,'\\\\|~\\\\|')[8],split(all,'\\\\|~\\\\|')[34],split(all,'\\\\|~\\\\|')[42],split(all,'\\\\|~\\\\|')[43],split(split(all,'\\\\|~\\\\|')[5],'/')[1]先进行子查询，然后group by，还是直接group by，两种那个好一点，
我个人测试后认为，数据量小，第一种会好一点，如果数据量大，可能第二种会好。至于数据量多大。TB级以下的都是小数据。
两个执行计划对比如下，可以看出基本执行的步骤的数据分析量差不多。
group by一定要用，但内外，先后执行顺序效果差不多。
代码1
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: tb
Statistics: Num rows: 1126576783 Data size: 112657678336 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: split(all, '\\|~\\|')[41] (type: string), split(all, '\\|~\\|')[11] (type: string), split(all, '\\|~\\|')[8] (type: string), split(all, '\\|~\\|')[34] (type: string), split(all, '\\|~\\|')[42] (type: string), split(all, '\\|~\\|')[43] (type: string), split(split(all, '\\|~\\|')[5], '/')[1] (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 1126576783 Data size: 112657678336 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
keys: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
Statistics: Num rows: 1126576783 Data size: 112657678336 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
sort order: +++++++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
Statistics: Num rows: 1126576783 Data size: 112657678336 Basic stats: COMPLETE Column stats: NONE
value expressions: _col7 (type: bigint)
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0)
keys: KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string), KEY._col4 (type: string), KEY._col5 (type: string), KEY._col6 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
Statistics: Num rows: 563288391 Data size: 56328839118 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string), _col7 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
Statistics: Num rows: 563288391 Data size: 56328839118 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 563288391 Data size: 56328839118 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: -1代码2
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: tb
Statistics: Num rows: 1126576783 Data size: 112657678336 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: all (type: string)
outputColumnNames: all
Statistics: Num rows: 1126576783 Data size: 112657678336 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: split(all, '\\|~\\|')[41] (type: string), split(all, '\\|~\\|')[11] (type: string), split(all, '\\|~\\|')[8] (type: string), split(all, '\\|~\\|')[34] (type: string), split(all, '\\|~\\|')[42] (type: string), split(all, '\\|~\\|')[43] (type: string), split(split(all, '\\|~\\|')[5], '/')[1] (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 1126576783 Data size: 112657678336 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
sort order: +++++++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
Statistics: Num rows: 1126576783 Data size: 112657678336 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string), KEY._col4 (type: string), KEY._col5 (type: string), KEY._col6 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 563288391 Data size: 56328839118 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 563288391 Data size: 56328839118 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 563288391 Data size: 56328839118 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: -1
hive高阶1--sql和hive语句执行顺序、explain查看执行计划、group by生成MR的更多相关文章
- Hive 高阶应用开发示例(一)
Hive的一些常用的高阶开发 内容 1.开窗函数 2.行转列,列转行,多行转一行,一行转多行 3.分组: 增强型group 4.排序 5.关联 本次的内容: 内容1 和内容2,采用 ...
- SQL复杂查询语句-SELECT * FROM cs WHERE score>70 GROUP BY s_id HAVING COUNT(*)>1
如果同时存在where,group by,的时候的执行顺序应该是这样的: 1,首先where后面添加条件把数据进行了过滤,返回一个结果集 2,然后group by将上面返回的结果集进行分组,返回一个结 ...
- MySQL--运行机制,SQL执行顺序,Explain
MySQL的运行机制是什么? 首先客户端先要发送用户信息去服务器端进行授权认证,当输入正确密码之后可以连接到数据库了,当连接服务器端成功之后就可以正常的执行 SQL 命令了,MySQL 服务器拿到 ...
- Hive高阶聚合函数 GROUPING SETS、Cube、Rollup
-- GROUPING SETS作为GROUP BY的子句,允许开发人员在GROUP BY语句后面指定多个统计选项,可以简单理解为多条group by语句通过union all把查询结果聚合起来结合起 ...
- SQL语句的执行顺序 1>优先执行，然后依数字排序
1>…From 表 2>…Where 条件 3>…Group by 列 4>…Having 筛选条件 ...
- sql执行效率,explain 查询执行效率
1.对查询进行优化,应尽量避免全表扫描,首先应考虑在 where 及 order by 涉及的列上建立索引. 2.应尽量避免在 where 子句中对字段进行 null 值判断,否则将导致引擎放弃使用索 ...
- SQL逻辑查询语句执行顺序
阅读目录 一 SELECT语句关键字的定义顺序 二 SELECT语句关键字的执行顺序 三 准备表和数据 四 准备SQL逻辑查询测试语句 五 执行顺序分析 一 SELECT语句关键字的定义顺序 SELE ...
- SQL执行顺序和coalesce以及case when的用法
1.mysql的执行顺序 from on join where group by having select distinct union //UNION 操作符用于合并两个或多个 SELECT ...
- SQL 中的语法顺序与执行顺序
FROM : HOME SQL 是一种声明式语言 SQL 语言是为计算机声明了一个你想从原始数据中获得什么样的结果的一个范例,而不是告诉计算机如何能够得到结果. SQL 语言声明的是结果集的属性,计算 ...
随机推荐
- Spring-Cloud（三）Eureka注册中心实现高可用
前言: spring-cloud为基础的微服务架构,所有的微服务都需要注册到注册中心,如果这个注册中心阻塞或者崩了,那么整个系统都无法继续正常提供服务,所以,这里就需要对注册中心进行集群,换言之,高可 ...
- 拥抱开源，Office 365开发迎来新时代
前言 作为全球最大的开放源代码托管平台,Github在上周迎来了它的十岁生日.自从2008年正式上线以来,Github上面汇聚了数以千万计的开发人员和各种项目,它几乎成为了开源的代名词和风向标,各大软 ...
- ASP.NET Core 如何在运行Docker容器时指定容器外部端口
前面我写了一系列关于持续集成的文章,最终构建出来的镜像运行之后,应该会发现每次构建运行之后端口都变了,这对于我们来说是十分不方便的,所以我们可以通过修改docker compose的配置文件来完成我们 ...
- [LeetCode] Maximum Length of Pair Chain 链对的最大长度
You are given n pairs of numbers. In every pair, the first number is always smaller than the second ...
- cd
从当前目录切换到目标目录 cd [目标目录] 切换到用户主目录 cd cd~ 切换到根目录 cd / 切换到上级目录 cd .. cd ../ cd ..// 切换到上级目录的父目录 ...
- [USACO14DEC]后卫马克Guard Mark
题目描述 FJ将飞盘抛向身高为H(1 <= H <= 1,000,000,000)的Mark,但是Mark 被N(2 <= N <= 20)头牛包围.牛们可以叠成一个牛塔,如果 ...
- ●BZOJ 3640 JC的小苹果
题链: http://www.lydsy.com/JudgeOnline/problem.php?id=3640题解: 期望dp,高斯消元 设dp[i][h]在i位置且血量为h这个状态的期望经过次数. ...
- 【BZOJ3631】【JLOI2014】松鼠的新家
原题传送门 题意:给你一棵树,然后有一个遍历顺序,你需要补全这个遍历顺序,然后输出这个遍历顺序中每个点的出现次数. 解题思路:本来想找树剖的题,结果发现了一题可以直接写lca的.... 做法1:非常简 ...
- [BZOJ]4200: [Noi2015]小园丁与老司机
Time Limit: 20 Sec Memory Limit: 512 MBSec Special Judge Description 小园丁 Mr. S 负责看管一片田野,田野可以看作一个二维 ...
- HDU5339——Untitled
Problem Description There is an integer a and n integers b1,…,bn. After selecting some numbers from ...