Hive函数：LAG,LEAD,FIRST_VALUE,LAST

参考自大数据田地：http://lxw1234.com/archives/2015/04/190.htm

测试数据准备：

create external table test_data (

cookieid string,

createtime string,  --页面访问时间

url string       --被访问页面

) ROW FORMAT DELIMITED

FIELDS TERMINATED BY ','

stored as textfile location '/user/jc_rc_ftp/test_data';

 select * from test_data l;

+-------------+----------------------+---------+--+

| l.cookieid  |     l.createtime     |  l.url  |

+-------------+----------------------+---------+--+

| cookie1     | 2015-04-10 10:00:02  | url2    |

| cookie1     | 2015-04-10 10:00:00  | url1    |

| cookie1     | 2015-04-10 10:03:04  | 1url3   |

| cookie1     | 2015-04-10 10:50:05  | url6    |

| cookie1     | 2015-04-10 11:00:00  | url7    |

| cookie1     | 2015-04-10 10:10:00  | url4    |

| cookie1     | 2015-04-10 10:50:01  | url5    |

| cookie2     | 2015-04-10 10:00:02  | url22   |

| cookie2     | 2015-04-10 10:00:00  | url11   |

| cookie2     | 2015-04-10 10:03:04  | 1url33  |

| cookie2     | 2015-04-10 10:50:05  | url66   |

| cookie2     | 2015-04-10 11:00:00  | url77   |

| cookie2     | 2015-04-10 10:10:00  | url44   |

| cookie2     | 2015-04-10 10:50:01  | url55   |

+-------------+----------------------+---------+--+

LAG
LAG(col,n,DEFAULT) 用于统计窗口内往上第n行值

第一个参数为列名，第二个参数为往上第n行（可选，默认为1），第三个参数为默认值（当往上第n行为NULL时候，取默认值，如不指定，则为NULL）

SELECT cookieid,

createtime,

url,

ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,

LAG(createtime,1,'1970-01-01 00:00:00') OVER(PARTITION BY cookieid ORDER BY createtime) AS last_1_time,

LAG(createtime,2) OVER(PARTITION BY cookieid ORDER BY createtime) AS last_2_time

FROM test_data;

+-----------+----------------------+---------+-----+----------------------+----------------------+--+

| cookieid  |      createtime      |   url   | rn  |     last_1_time      |     last_2_time      |

+-----------+----------------------+---------+-----+----------------------+----------------------+--+

| cookie1   | 2015-04-10 10:00:00  | url1    | 1   | 1970-01-01 00:00:00  | NULL                 |

| cookie1   | 2015-04-10 10:00:02  | url2    | 2   | 2015-04-10 10:00:00  | NULL                 |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | 3   | 2015-04-10 10:00:02  | 2015-04-10 10:00:00  |

| cookie1   | 2015-04-10 10:10:00  | url4    | 4   | 2015-04-10 10:03:04  | 2015-04-10 10:00:02  |

| cookie1   | 2015-04-10 10:50:01  | url5    | 5   | 2015-04-10 10:10:00  | 2015-04-10 10:03:04  |

| cookie1   | 2015-04-10 10:50:05  | url6    | 6   | 2015-04-10 10:50:01  | 2015-04-10 10:10:00  |

| cookie1   | 2015-04-10 11:00:00  | url7    | 7   | 2015-04-10 10:50:05  | 2015-04-10 10:50:01  |

| cookie2   | 2015-04-10 10:00:00  | url11   | 1   | 1970-01-01 00:00:00  | NULL                 |

| cookie2   | 2015-04-10 10:00:02  | url22   | 2   | 2015-04-10 10:00:00  | NULL                 |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | 3   | 2015-04-10 10:00:02  | 2015-04-10 10:00:00  |

| cookie2   | 2015-04-10 10:10:00  | url44   | 4   | 2015-04-10 10:03:04  | 2015-04-10 10:00:02  |

| cookie2   | 2015-04-10 10:50:01  | url55   | 5   | 2015-04-10 10:10:00  | 2015-04-10 10:03:04  |

| cookie2   | 2015-04-10 10:50:05  | url66   | 6   | 2015-04-10 10:50:01  | 2015-04-10 10:10:00  |

| cookie2   | 2015-04-10 11:00:00  | url77   | 7   | 2015-04-10 10:50:05  | 2015-04-10 10:50:01  |

+-----------+----------------------+---------+-----+----------------------+----------------------+--+

LEAD

与LAG相反
LEAD(col,n,DEFAULT) 用于统计窗口内往下第n行值
第一个参数为列名，第二个参数为往下第n行（可选，默认为1），第三个参数为默认值（当往下第n行为NULL时候，取默认值，如不指定，则为NULL）

SELECT cookieid,

createtime,

url,

ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,

LEAD(createtime,1,'1970-01-01 00:00:00') OVER(PARTITION BY cookieid ORDER BY createtime) AS next_1_time,

LEAD(createtime,2) OVER(PARTITION BY cookieid ORDER BY createtime) AS next_2_time

FROM test_data;

+-----------+----------------------+---------+-----+----------------------+----------------------+--+

| cookieid  |      createtime      |   url   | rn  |     next_1_time      |     next_2_time      |

+-----------+----------------------+---------+-----+----------------------+----------------------+--+

| cookie1   | 2015-04-10 10:00:00  | url1    | 1   | 2015-04-10 10:00:02  | 2015-04-10 10:03:04  |

| cookie1   | 2015-04-10 10:00:02  | url2    | 2   | 2015-04-10 10:03:04  | 2015-04-10 10:10:00  |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | 3   | 2015-04-10 10:10:00  | 2015-04-10 10:50:01  |

| cookie1   | 2015-04-10 10:10:00  | url4    | 4   | 2015-04-10 10:50:01  | 2015-04-10 10:50:05  |

| cookie1   | 2015-04-10 10:50:01  | url5    | 5   | 2015-04-10 10:50:05  | 2015-04-10 11:00:00  |

| cookie1   | 2015-04-10 10:50:05  | url6    | 6   | 2015-04-10 11:00:00  | NULL                 |

| cookie1   | 2015-04-10 11:00:00  | url7    | 7   | 1970-01-01 00:00:00  | NULL                 |

| cookie2   | 2015-04-10 10:00:00  | url11   | 1   | 2015-04-10 10:00:02  | 2015-04-10 10:03:04  |

| cookie2   | 2015-04-10 10:00:02  | url22   | 2   | 2015-04-10 10:03:04  | 2015-04-10 10:10:00  |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | 3   | 2015-04-10 10:10:00  | 2015-04-10 10:50:01  |

| cookie2   | 2015-04-10 10:10:00  | url44   | 4   | 2015-04-10 10:50:01  | 2015-04-10 10:50:05  |

| cookie2   | 2015-04-10 10:50:01  | url55   | 5   | 2015-04-10 10:50:05  | 2015-04-10 11:00:00  |

| cookie2   | 2015-04-10 10:50:05  | url66   | 6   | 2015-04-10 11:00:00  | NULL                 |

| cookie2   | 2015-04-10 11:00:00  | url77   | 7   | 1970-01-01 00:00:00  | NULL                 |

+-----------+----------------------+---------+-----+----------------------+----------------------+--+

FIRST_VALUE

取分组内排序后，截止到当前行，第一个值

SELECT cookieid,

createtime,

url,

ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,

FIRST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime) AS first1

FROM test_data;

+-----------+----------------------+---------+-----+---------+--+

| cookieid  |      createtime      |   url   | rn  | first1  |

+-----------+----------------------+---------+-----+---------+--+

| cookie1   | 2015-04-10 10:00:00  | url1    | 1   | url1    |

| cookie1   | 2015-04-10 10:00:02  | url2    | 2   | url1    |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | 3   | url1    |

| cookie1   | 2015-04-10 10:10:00  | url4    | 4   | url1    |

| cookie1   | 2015-04-10 10:50:01  | url5    | 5   | url1    |

| cookie1   | 2015-04-10 10:50:05  | url6    | 6   | url1    |

| cookie1   | 2015-04-10 11:00:00  | url7    | 7   | url1    |

| cookie2   | 2015-04-10 10:00:00  | url11   | 1   | url11   |

| cookie2   | 2015-04-10 10:00:02  | url22   | 2   | url11   |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | 3   | url11   |

| cookie2   | 2015-04-10 10:10:00  | url44   | 4   | url11   |

| cookie2   | 2015-04-10 10:50:01  | url55   | 5   | url11   |

| cookie2   | 2015-04-10 10:50:05  | url66   | 6   | url11   |

| cookie2   | 2015-04-10 11:00:00  | url77   | 7   | url11   |

+-----------+----------------------+---------+-----+---------+--+

LAST_VALUE

取分组内排序后，截止到当前行，最后一个值

SELECT cookieid,

createtime,

url,

ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,

LAST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime) AS last1

FROM test_data;

+-----------+----------------------+---------+-----+---------+--+

| cookieid  |      createtime      |   url   | rn  |  last1  |

+-----------+----------------------+---------+-----+---------+--+

| cookie1   | 2015-04-10 10:00:00  | url1    | 1   | url1    |

| cookie1   | 2015-04-10 10:00:02  | url2    | 2   | url2    |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | 3   | 1url3   |

| cookie1   | 2015-04-10 10:10:00  | url4    | 4   | url4    |

| cookie1   | 2015-04-10 10:50:01  | url5    | 5   | url5    |

| cookie1   | 2015-04-10 10:50:05  | url6    | 6   | url6    |

| cookie1   | 2015-04-10 11:00:00  | url7    | 7   | url7    |

| cookie2   | 2015-04-10 10:00:00  | url11   | 1   | url11   |

| cookie2   | 2015-04-10 10:00:02  | url22   | 2   | url22   |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | 3   | 1url33  |

| cookie2   | 2015-04-10 10:10:00  | url44   | 4   | url44   |

| cookie2   | 2015-04-10 10:50:01  | url55   | 5   | url55   |

| cookie2   | 2015-04-10 10:50:05  | url66   | 6   | url66   |

| cookie2   | 2015-04-10 11:00:00  | url77   | 7   | url77   |

+-----------+----------------------+---------+-----+---------+--+

SELECT cookieid,

createtime,

url,

ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,

LAST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime DESC) AS last1

FROM test_data;

+-----------+----------------------+---------+-----+---------+--+

| cookieid  |      createtime      |   url   | rn  |  last1  |

+-----------+----------------------+---------+-----+---------+--+

| cookie1   | 2015-04-10 11:00:00  | url7    | 7   | url7    |

| cookie1   | 2015-04-10 10:50:05  | url6    | 6   | url6    |

| cookie1   | 2015-04-10 10:50:01  | url5    | 5   | url5    |

| cookie1   | 2015-04-10 10:10:00  | url4    | 4   | url4    |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | 3   | 1url3   |

| cookie1   | 2015-04-10 10:00:02  | url2    | 2   | url2    |

| cookie1   | 2015-04-10 10:00:00  | url1    | 1   | url1    |

| cookie2   | 2015-04-10 11:00:00  | url77   | 7   | url77   |

| cookie2   | 2015-04-10 10:50:05  | url66   | 6   | url66   |

| cookie2   | 2015-04-10 10:50:01  | url55   | 5   | url55   |

| cookie2   | 2015-04-10 10:10:00  | url44   | 4   | url44   |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | 3   | 1url33  |

| cookie2   | 2015-04-10 10:00:02  | url22   | 2   | url22   |

| cookie2   | 2015-04-10 10:00:00  | url11   | 1   | url11   |

+-----------+----------------------+---------+-----+---------+--+

如果不指定ORDER BY，则默认按照记录在文件中的偏移量进行排序，会出现错误的结果

SELECT cookieid,

createtime,

url,

FIRST_VALUE(url) OVER(PARTITION BY cookieid) AS first2

FROM test_data;

+-----------+----------------------+---------+---------+--+

| cookieid  |      createtime      |   url   | first2  |

+-----------+----------------------+---------+---------+--+

| cookie1   | 2015-04-10 10:00:02  | url2    | url2    |

| cookie1   | 2015-04-10 10:50:01  | url5    | url2    |

| cookie1   | 2015-04-10 10:10:00  | url4    | url2    |

| cookie1   | 2015-04-10 11:00:00  | url7    | url2    |

| cookie1   | 2015-04-10 10:50:05  | url6    | url2    |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | url2    |

| cookie1   | 2015-04-10 10:00:00  | url1    | url2    |

| cookie2   | 2015-04-10 10:50:01  | url55   | url55   |

| cookie2   | 2015-04-10 10:10:00  | url44   | url55   |

| cookie2   | 2015-04-10 11:00:00  | url77   | url55   |

| cookie2   | 2015-04-10 10:50:05  | url66   | url55   |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | url55   |

| cookie2   | 2015-04-10 10:00:00  | url11   | url55   |

| cookie2   | 2015-04-10 10:00:02  | url22   | url55   |

+-----------+----------------------+---------+---------+--+

SELECT cookieid,

createtime,

url,

LAST_VALUE(url) OVER(PARTITION BY cookieid) AS last2

FROM test_data;

+-----------+----------------------+---------+--------+--+

| cookieid  |      createtime      |   url   | last2  |

+-----------+----------------------+---------+--------+--+

| cookie1   | 2015-04-10 10:00:02  | url2    | url1   |

| cookie1   | 2015-04-10 10:50:01  | url5    | url1   |

| cookie1   | 2015-04-10 10:10:00  | url4    | url1   |

| cookie1   | 2015-04-10 11:00:00  | url7    | url1   |

| cookie1   | 2015-04-10 10:50:05  | url6    | url1   |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | url1   |

| cookie1   | 2015-04-10 10:00:00  | url1    | url1   |

| cookie2   | 2015-04-10 10:50:01  | url55   | url22  |

| cookie2   | 2015-04-10 10:10:00  | url44   | url22  |

| cookie2   | 2015-04-10 11:00:00  | url77   | url22  |

| cookie2   | 2015-04-10 10:50:05  | url66   | url22  |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | url22  |

| cookie2   | 2015-04-10 10:00:00  | url11   | url22  |

| cookie2   | 2015-04-10 10:00:02  | url22   | url22  |

+-----------+----------------------+---------+--------+--+

14 rows selected (78.058 seconds)

如果想要取分组内排序后最后一个值，则需要变通一下：

SELECT cookieid,

createtime,

url,

ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,

LAST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime) AS last1,

FIRST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime DESC) AS last2

FROM test_data

ORDER BY cookieid,createtime;

+-----------+----------------------+---------+-----+---------+--------+--+

| cookieid  |      createtime      |   url   | rn  |  last1  | last2  |

+-----------+----------------------+---------+-----+---------+--------+--+

| cookie1   | 2015-04-10 10:00:00  | url1    | 1   | url1    | url7   |

| cookie1   | 2015-04-10 10:00:02  | url2    | 2   | url2    | url7   |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | 3   | 1url3   | url7   |

| cookie1   | 2015-04-10 10:10:00  | url4    | 4   | url4    | url7   |

| cookie1   | 2015-04-10 10:50:01  | url5    | 5   | url5    | url7   |

| cookie1   | 2015-04-10 10:50:05  | url6    | 6   | url6    | url7   |

| cookie1   | 2015-04-10 11:00:00  | url7    | 7   | url7    | url7   |

| cookie2   | 2015-04-10 10:00:00  | url11   | 1   | url11   | url77  |

| cookie2   | 2015-04-10 10:00:02  | url22   | 2   | url22   | url77  |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | 3   | 1url33  | url77  |

| cookie2   | 2015-04-10 10:10:00  | url44   | 4   | url44   | url77  |

| cookie2   | 2015-04-10 10:50:01  | url55   | 5   | url55   | url77  |

| cookie2   | 2015-04-10 10:50:05  | url66   | 6   | url66   | url77  |

| cookie2   | 2015-04-10 11:00:00  | url77   | 7   | url77   | url77  |

+-----------+----------------------+---------+-----+---------+--------+--+

Hive函数：LAG,LEAD,FIRST_VALUE,LAST_VALUE的更多相关文章

pandas实现hive的lag和lead函数以及 first_value和last_value函数
lag和lead VS shift 该函数的格式如下: 第一个参数为列名, 第二个参数为往上第n行(可选,默认为1), 第三个参数为默认值(当往上第n行为NULL时候,取默认值,如不指定,则为NULL ...
Hive 窗口函数LEAD LAG FIRST_VALUE LAST_VALUE
窗口函数(window functions)对多行进行操作,并为查询中的每一行返回一个值. OVER()子句能将窗口函数与其他分析函数(analytical functions)和报告函数(repor ...
oracle listagg函数、lag函数、lead函数实例
Oracle大师Thomas Kyte在他的经典著作中,反复强调过一个实现需求方案选取顺序: “如果你可以使用一句SQL解决的需求,就使用一句SQL:如果不可以,就考虑PL/SQL是否可以:如果PL/ ...
hive函数参考手册
hive函数参考手册原文见:https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF 1.内置运算符1.1关系运算符运 ...
Hive函数以及自定义函数讲解（UDF）
Hive函数介绍HQL内嵌函数只有195个函数(包括操作符,使用命令show functions查看),基本能够胜任基本的hive开发,但是当有较为复杂的需求的时候,可能需要进行定制的HQL函数开发. ...
大数据入门第十一天——hive详解（三）hive函数
一.hive函数 1.内置运算符与内置函数函数分类: 查看函数信息: DESC FUNCTION concat; 常用的分析函数之rank() row_number(),参考:https://www ...
Hadoop生态圈-Hive函数
Hadoop生态圈-Hive函数作者:尹正杰版权声明:原创作品,谢绝转载!否则将追究法律责任.
Hive（四）hive函数与hive shell
一.hive函数 1.hive内置函数 (1)内容较多,见< Hive 官方文档> https://cwiki.apache.org/confluence/displ ...
Hive入门笔记---2.hive函数大全
Hive函数大全–完整版现在虽然有很多SQL ON Hadoop的解决方案,像Spark SQL.Impala.Presto等等,但就目前来看,在基于Hadoop的大数据分析平台.数据仓库中,Hiv ...

随机推荐

equals和hashCode详解
equals和hashCode详解 http://www.cnblogs.com/Qian123/p/5703507.html
javascript中函数的闭包自调用
话不多说, 直接上代码 // 定义一个变量outerParam, 然后使用一个闭包函数给该变量初始化var outerParam = (function testClosure(param) { // ...
页面内部DIV让点击外部DIV 事件不发生（阻止冒泡事件）
如标题的情况,经常发生,尤其是在一些弹出框上面之类的. <script> function zuzhimaopao(){ e.stopPropagation(); } </scrip ...
1833 深坑 TLE 求解
题目描述: 大家知道,给出正整数n,则1到n这n个数可以构成n!种排列,把这些排列按照从小到大的顺序(字典顺序)列出,如n=3时,列出1 2 3,1 3 2,2 1 3,2 3 1,3 1 2,3 2 ...
排序算法Java实现（选择排序）
算法描述:对于给定的一组记录,经过第一轮比较后得到最小的记录,然后将该记录与第一个记录的位置进行交换:接着对不包括第一个记录以外的其他记录进行第二轮比较,得到最小的记录并与第二个记录进行位置交换:重复 ...
国内可用的Internet时间同步服务器地址（NTP时间服务器）
不知道什么鬼我这系统自带的Internet时间同步服务器地址居然不可用,终端ping系统自带服务器两个居然都不通???难道时间服务器也和谐么? 好在阿里云提供了7个NTP时间服务器也就是Interne ...
SQL更新语句，Error Code: 1175. You are using safe update(在进行视图更新的时候遇到)
转发于:http://blog.csdn.net/qq_26684469/article/details/51105188?locationNum=5&fps=1 原来的SET SQL_SAF ...
pyrhon多进程操作初探
linux系统中提供了fork函数进行进程的创建,这个接口在函数返回上比较特殊,有两个返回值,一个是子进程返回值为0,一个是父进程返回值,值大于0,表是子进程的ID.如果小于0.则表示接口出错. py ...
用SpringBoot搭建简单电商项目 01
前几节呢,我们已经简单介绍了SpringBoot框架的使用,从这一节开始,我们尝试着使用SpringBoot框架来一步一步搭建一个简单电商项目.当然了,这不是真正的电商项目,你可以看成是一个CRUD案 ...
JDK1.8源码(六)——java.util.LinkedList 类
上一篇博客我们介绍了List集合的一种典型实现 ArrayList,我们知道 ArrayList 是由数组构成的,本篇博客我们介绍 List 集合的另一种典型实现 LinkedList,这是一个有链表 ...

Hive函数：LAG,LEAD,FIRST_VALUE,LAST_VALUE

LEAD

FIRST_VALUE

LAST_VALUE

Hive函数：LAG,LEAD,FIRST_VALUE,LAST_VALUE的更多相关文章

随机推荐

热门专题