Hive新功能 Cube, Rollup介绍
说明:Hive之cube、rollup,还有窗口函数,在传统关系型数据(Oracle、sqlserver)中都是有的,用法都很相似。
GROUPING SETS
GROUPING SETS作为GROUP BY的子句,允许开发人员在GROUP BY语句后面指定多个统计选项,可以简单理解为多条group by语句通过union all把查询结果聚合起来结合起来,下面是几个实例可以帮助我们了解,
以acorn_3g.test_xinyan_reg为例:
- [dp@YZSJHL19-87 xjob]$ hive -e "use acorn_3g;desc test_xinyan_reg;"
- user_id bigint None
- device_id int None 手机,平板
- os_id int None 操作系统类型
- app_id int None 手机app_id
- client_version string None 客户端版本
- from_id int None 四级渠道
几个demo帮助大家了解:
grouping sets语句 | 等价hive语句 |
---|---|
select device_id,os_id,app_id,count(user_id) from test_xinyan_reg group by device_id,os_id,app_id grouping sets((device_id)) | SELECT device_id,null,null,count(user_id) FROM test_xinyan_reg group by device_id |
select device_id,os_id,app_id,count(user_id) from test_xinyan_reg group by device_id,os_id,app_id grouping sets((device_id,os_id)) | SELECT device_id,os_id,null,count(user_id) FROM test_xinyan_reg group by device_id,os_id |
select device_id,os_id,app_id,count(user_id) from test_xinyan_reg group by device_id,os_id,app_id grouping sets((device_id,os_id),(device_id)) | SELECT device_id,os_id,null,count(user_id) FROM test_xinyan_reg group by device_id,os_id UNION ALL SELECT device_id,null,null,count(user_id) FROM test_xinyan_reg group by device_id |
select device_id,os_id,app_id,count(user_id) from test_xinyan_reg group by device_id,os_id,app_id grouping sets((device_id),(os_id),(device_id,os_id),()) | SELECT device_id,null,null,count(user_id) FROM test_xinyan_reg group by device_id UNION ALL SELECT null,os_id,null,count(user_id) FROM test_xinyan_reg group by os_id UNION ALL SELECT device_id,os_id,null,count(user_id) FROM test_xinyan_reg group by device_id,os_id UNION ALL SELECT null,null,null,count(user_id) FROM test_xinyan_reg |
CUBE函数
cube简称数据魔方,可以实现hive多个任意维度的查询,cube(a,b,c)则首先会对(a,b,c)进行group by,然后依次是(a,b),(a,c),(a),(b,c),(b),(c),最后在对全表进行group by,他会统计所选列中值的所有组合的聚合
- select device_id,os_id,app_id,client_version,from_id,count(user_id)
- from test_xinyan_reg
- group by device_id,os_id,app_id,client_version,from_id with cube;
手工实现需要写的hql语句(写个程序自己生成的,手写累死):
- SELECT device_id,null,null,null,null ,count(user_id) FROM test_xinyan_reg group by device_id
- UNION ALL
- SELECT null,os_id,null,null,null ,count(user_id) FROM test_xinyan_reg group by os_id
- UNION ALL
- SELECT device_id,os_id,null,null,null ,count(user_id) FROM test_xinyan_reg group by device_id,os_id
- UNION ALL
- SELECT null,null,app_id,null,null ,count(user_id) FROM test_xinyan_reg group by app_id
- UNION ALL
- SELECT device_id,null,app_id,null,null ,count(user_id) FROM test_xinyan_reg group by device_id,app_id
- UNION ALL
- SELECT null,os_id,app_id,null,null ,count(user_id) FROM test_xinyan_reg group by os_id,app_id
- UNION ALL
- SELECT device_id,os_id,app_id,null,null ,count(user_id) FROM test_xinyan_reg group by device_id,os_id,app_id
- UNION ALL
- SELECT null,null,null,client_version,null ,count(user_id) FROM test_xinyan_reg group by client_version
- UNION ALL
- SELECT device_id,null,null,client_version,null ,count(user_id) FROM test_xinyan_reg group by device_id,client_version
- UNION ALL
- SELECT null,os_id,null,client_version,null ,count(user_id) FROM test_xinyan_reg group by os_id,client_version
- UNION ALL
- SELECT device_id,os_id,null,client_version,null ,count(user_id) FROM test_xinyan_reg group by device_id,os_id,client_version
- UNION ALL
- SELECT null,null,app_id,client_version,null ,count(user_id) FROM test_xinyan_reg group by app_id,client_version
- UNION ALL
- SELECT device_id,null,app_id,client_version,null ,count(user_id) FROM test_xinyan_reg group by device_id,app_id,client_version
- UNION ALL
- SELECT null,os_id,app_id,client_version,null ,count(user_id) FROM test_xinyan_reg group by os_id,app_id,client_version
- UNION ALL
- SELECT device_id,os_id,app_id,client_version,null ,count(user_id) FROM test_xinyan_reg group by device_id,os_id,app_id,client_version
- UNION ALL
- SELECT null,null,null,null,from_id ,count(user_id) FROM test_xinyan_reg group by from_id
- UNION ALL
- SELECT device_id,null,null,null,from_id ,count(user_id) FROM test_xinyan_reg group by device_id,from_id
- UNION ALL
- SELECT null,os_id,null,null,from_id ,count(user_id) FROM test_xinyan_reg group by os_id,from_id
- UNION ALL
- SELECT device_id,os_id,null,null,from_id ,count(user_id) FROM test_xinyan_reg group by device_id,os_id,from_id
- UNION ALL
- SELECT null,null,app_id,null,from_id ,count(user_id) FROM test_xinyan_reg group by app_id,from_id
- UNION ALL
- SELECT device_id,null,app_id,null,from_id ,count(user_id) FROM test_xinyan_reg group by device_id,app_id,from_id
- UNION ALL
- SELECT null,os_id,app_id,null,from_id ,count(user_id) FROM test_xinyan_reg group by os_id,app_id,from_id
- UNION ALL
- SELECT device_id,os_id,app_id,null,from_id ,count(user_id) FROM test_xinyan_reg group by device_id,os_id,app_id,from_id
- UNION ALL
- SELECT null,null,null,client_version,from_id ,count(user_id) FROM test_xinyan_reg group by client_version,from_id
- UNION ALL
- SELECT device_id,null,null,client_version,from_id ,count(user_id) FROM test_xinyan_reg group by device_id,client_version,from_id
- UNION ALL
- SELECT null,os_id,null,client_version,from_id ,count(user_id) FROM test_xinyan_reg group by os_id,client_version,from_id
- UNION ALL
- SELECT device_id,os_id,null,client_version,from_id ,count(user_id) FROM test_xinyan_reg group by device_id,os_id,client_version,from_id
- UNION ALL
- SELECT null,null,app_id,client_version,from_id ,count(user_id) FROM test_xinyan_reg group by app_id,client_version,from_id
- UNION ALL
- SELECT device_id,null,app_id,client_version,from_id ,count(user_id) FROM test_xinyan_reg group by device_id,app_id,client_version,from_id
- UNION ALL
- SELECT null,os_id,app_id,client_version,from_id ,count(user_id) FROM test_xinyan_reg group by os_id,app_id,client_version,from_id
- UNION ALL
- SELECT device_id,os_id,app_id,client_version,from_id ,count(user_id) FROM test_xinyan_reg group by device_id,os_id,app_id,client_version,from_id
- UNION ALL
- SELECT null,null,null,null,null ,count(user_id) FROM test_xinyan_reg
看着很蛋疼是不是,体会到cube的强大了吗!(低版本hive可以通过union all方式解决,算是没有办法的办法)
ROLL UP函数
rollup可以实现从右到做递减多级的统计,显示统计某一层次结构的聚合。
- select device_id,os_id,app_id,client_version,from_id,count(user_id)
- from test_xinyan_reg
- group by device_id,os_id,app_id,client_version,from_id with rollup;
等价以下sql语句:
- select device_id,os_id,app_id,client_version,from_id,count(user_id)
- from test_xinyan_reg
- group by device_id,os_id,app_id,client_version,from_id
- grouping sets ((device_id,os_id,app_id,client_version,from_id),(device_id,os_id,app_id,client_version),(device_id,os_id,app_id),(device_id,os_id),(device_id),());
Grouping_ID函数
当我们没有统计某一列时,它的值显示为null,这可能与列本身就有null值冲突,这就需要一种方法区分是没有统计还是值本来就是null。(写一个排列组合的算法,就马上理解了,grouping_id其实就是所统计各列二进制和)
直接拿官方文档一个例子,O(∩_∩)O哈哈~
Column1 (key) | Column2 (value) |
---|---|
1 | NULL |
1 | 1 |
2 | 2 |
3 | 3 |
3 | NULL |
4 | 5 |
hql统计:
- SELECT key, value, GROUPING__ID, count(*) from T1 GROUP BY key, value WITH ROLLUP
统计结果如下:
NULL | NULL | 0 00 | 6 |
1 | NULL | 1 10 | 2 |
1 | NULL | 3 11 | 1 |
1 | 1 | 3 11 | 1 |
2 | NULL | 1 10 | 1 |
2 | 2 | 3 11 | 1 |
3 | NULL | 1 10 | 2 |
3 | NULL | 3 11 | 1 |
3 | 3 | 3 11 | 1 |
4 | NULL | 1 10 | 1 |
4 | 5 | 3 11 | 1 |
GROUPING__ID转变为二进制,如果对应位上有值为null,说明这列本身值就是null。(通过类DataFilterNull.py 扫描,可以筛选过滤掉列中null、“”统计结果),
窗口函数
hive窗口函数,感觉大部分都是在模仿oracle,有对oracle熟悉的,应该看下就知道怎么用。
具体参见:http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.0.2/ds_Hive/language_manual/ptf-window.html
主要围绕..over( partitoin by ..) ..
3g业务求新增激活时候,有的一部手机,可能注册多个渠道,这时候就要按时间顺序求第一个:
- select f.udid,f.from_id,f.ins_date
- from (select /* +MAPJOIN(u) */ u.device_id as udid ,g.device_id as gdid,u.from_id,u.ins_date,row_number() over (partition by u.device_id order by u.ins_date asc) as row_number
- from user_device_info u
- left outer join (select device_id from 3g_device_id where log_date<'2013-07-25') g on ( u.device_id = g.device_id )
- where u.log_date='2013-07-25' and u.device_id is not null and u.device_id <> '') f
- where f.gdid is null and row_number=1
参考资料
apache hive窗口函数官方介绍:http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.0.2/ds_Hive/language_manual/ptf-window.html
apache hive官方:cube、rollup函数介绍:https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation,+Cube,+Grouping+and+Rollup
oracle窗口函数介绍:http://www.blogjava.net/pengpenglin/archive/2012/04/12/211334.html
Hive新功能 Cube, Rollup介绍的更多相关文章
- hive新功能cube和rollup
1.cube简称数据魔方,可以实现hive多个任意维度的查询,cube(a,b,c)则首先会对(a,b,c)进行group by,然后依次是(a,b),(a,c),(a),(b,c),(b),(c), ...
- Android 9 新功能 及 API 介绍(提供了实用的模块化的功能支持,包括 人工智能)
Android 9(API 级别 28)为用户和开发者引入了众多新特性和新功能. 本文重点介绍面向开发者的新功能. 要了解新 API,请阅读 API 差异报告或访问 Android API 参考. ...
- hive 之 Cube, Rollup介绍
1. GROUPING SETS GROUPING SETS作为GROUP BY的子句,允许开发人员在GROUP BY语句后面指定多个统维度,可以简单理解为多条group by语句通过union al ...
- geotrellis使用(二十)geotrellis1.0版本新功能及变化介绍
目录 前言 变化情况介绍 总结 一.前言 之前版本是0.9或者0.10.1.0.10.2,最近发现更新成为1.0.0-2077839.1.0应该也能称之为正式版了吧.发现其中有很多变化, ...
- hive新特性reflect函数介绍
reflect函数可以支持在sql中调用java中的自带函数,秒杀一切udf函数. 使用案例1:所有记录执行相同的java内置函数 hive中建一张表test_udf:column1(int),col ...
- nopCommerce 3.2新功能
NopCommerce版本3.20,上周被释放,对于那些你谁还不熟悉新版本或刚经过脱脂发行说明我们的新功能的详细介绍. 在nopCommerce 3.20新功能的工作往往需要某些设置或语言资源的快速修 ...
- TFS 2015新功能之一,当前迭代查询标记
TFS 2015发布在即,有幸作为MVP提前获得了TFS的RTM版本,下面就TFS 2015的新功能做一些介绍: TFS 2015新功能之一,当前迭代查询标记 在TFS的查询中,可以将" ...
- MFC界面库BCGControlBar v30.1新功能详解:Dialogs和Forms
亲爱的BCGSoft用户,我们非常高兴地宣布BCGControlBar Professional for MFC和BCGSuite for MFC v30.1正式发布!此版本包含themed find ...
- Apache Flink 1.9.0版本新功能介绍
摘要:Apache Flink是一个面向分布式数据流处理和批量数据处理的开源计算平台,它能够基于同一个Flink运行时,提供支持流处理和批处理两种类型应用的功能.目前,Apache Flink 1.9 ...
随机推荐
- jquery chosen 插件 动态设置+更新选项值
我要在表单里使用一个select下拉菜单(是不是multiple无所谓),所以选择了jquery chosen这个插件.现在有一个需求,需要根据表单的另一个域来动态加载该select元素的选项. 1 ...
- sql语句中处理金额,把分换算成元
问题,sql语句中直接将金额/100返回的结果会有多个小数位. as value from account as acc left join conCategory as cate on acc.ca ...
- linux yum安装lsof命令
[root@ITC-MCC ~]# yum install lsof[USM] permission denied^C[root@ITC-MCC ~]# [root@ITC-MCC ~]# [root ...
- dva subscription的使用方法
import { routerRedux } from 'dva/router' export default { namespace: 'notice', state: { notices:[], ...
- epoll模型边沿触发
body, table{font-family: 微软雅黑; font-size: 10pt} table{border-collapse: collapse; border: solid gray; ...
- SSH使用主机名访问
比如说A电脑已经和B电脑实现了ssh免密码登陆!但是A电脑通过 ssh B电脑的主机名称 不行! 解决办法: 01.修改A电脑中的hosts文件 vim /etc/hosts 02.进入编辑界面 ...
- 通过VNC连接远程服务器,然后登陆服务器上的虚拟机,出现键盘输入问题的解决方法
前几天由于要在服务器上装一个虚拟机,然后就选择了vmware workstation,装好之后,进入虚拟机中的centOS系统,发现键盘上的Cpas Lock键不起作用,按下之后还是输入小写,而且按住 ...
- 重写ajax方法实现特定情况下跳转登录页面
jQuery(function($){ // 备份jquery的ajax方法 var _ajax=$.ajax; // 重写ajax方法, $.ajax=function(opt){ var _suc ...
- Jmeter-Logic Controllers(逻辑控制器)
Critical Section Controller(临界区控制器) 参考:http://www.cnblogs.com/yanzhe/p/7729984.html ForEach Controll ...
- LightOJ - 1205:Palindromic Numbers (数位DP&回文串)
A palindromic number or numeral palindrome is a 'symmetrical' number like 16461 that remains the sam ...