Pig group用法举例
[hadoop@vm1 ~]$ cat orders.data
1 apple 30 x
2 apple 50 x
3 banana 30 y
4 pear 20 y
5 banana 10 y
[hadoop@vm1 ~]$
data = load '/orders.data' as (orderid:int, fruit:chararray, amount:int);
grpd = group data by fruit;
describe grpd;
grpd: {group: chararray,data: {(orderid: int,fruit: chararray,amount: int)}}
dump grpd;
(pear,{(4,pear,20)})
(apple,{(2,apple,50),(1,apple,30)})
(banana,{(5,banana,10),(3,banana,30)})
dump grpd;
(pear,{(4,pear,20)})
(apple,{(2,apple,50),(1,apple,30)})
(banana,{(5,banana,10),(3,banana,30)})
group data by $0+$1;
orders = load '/orders.data' as (orderid:int, fruit:chararray, amount:int, type:chararray);
grpd = group orders by (fruit, type);
describe grpd;
grpd: {group: (fruit: chararray,type: chararray),orders: {(orderid: int,fruit: chararray,amount: int,type: chararray)}}
dump grpd;
((pear,y),{(4,pear,20,y)})
((apple,x),{(2,apple,50,x),(1,apple,30,x)})
((banana,y),{(5,banana,10,y),(3,banana,30,y)})
sums = foreach grpd generate group, SUM(orders.amount);
dump sums;
((pear,y),20)
((apple,x),80)
((banana,y),40)
sums2 = foreach grpd generate group.$0, group.$1, SUM(orders.amount);
dump sums2;
(pear,y,20)
(apple,x,80)
(banana,y,40
grpd = group orders all;
describe grpd;
grpd: {group: chararray,orders: {(orderid: int,fruit: chararray,amount: int,type: chararray)}}
dump grpd;
(all,{(5,banana,10,y),(4,pear,20,y),(3,banana,30,y),(2,apple,50,x),(1,apple,30,x)})
A = LOAD 'data1' AS (owner:chararray,pet:chararray);
DUMP A;
(Alice,turtle)
(Alice,goldfish)
(Alice,cat)
(Bob,dog)
(Bob,cat)
B = LOAD 'data2' AS (friend1:chararray,friend2:chararray);
DUMP B;
(Cindy,Alice)
(Mark,Alice)
(Paul,Bob)
(Paul,Jane)
X = COGROUP A BY owner, B BY friend2;
DESCRIBE X;
X: {group: chararray,A: {owner: chararray,pet: chararray},B: {friend1: chararray,friend2: chararray}}
DUMP X;
(Alice,{(Alice,turtle),(Alice,goldfish),(Alice,cat)},{(Cindy,Alice),(Mark,Alice)})
(Bob,{(Bob,dog),(Bob,cat)},{(Paul,Bob)})
(Jane,{},{(Paul,Jane)})
A = LOAD 'input_data';
B = GROUP A BY $0 PARTITION BY org.apache.pig.test.utils.SimpleCustomPartitioner PARALLEL 2;
public class SimpleCustomPartitioner extends Partitioner <PigNullableWritable, Writable> {
//@Override
public int getPartition(PigNullableWritable key, Writable value, int numPartitions) {
if(key.getValueAsPigType() instanceof Integer) {
int ret = (((Integer)key.getValueAsPigType()).intValue() % numPartitions);
return ret;
}
else {
return (key.hashCode()) % numPartitions;
}
}
}
Pig group用法举例的更多相关文章
- Pig sample用法举例
some = sample data 0.1 遍历整个数据集,获取指定比例的行数的数据,获取的数据不确定,条数也不准确. 内部重写为filter data by random() <= ...
- Pig limit用法举例
lmt = limit data 10; 只获取指定条数的数据,不能保证每次得到的结果一致,先执行order再limit可以保证一致. 输入数据全部载入. 会触发reduce阶段 a ...
- Pig join用法举例
jnd = join a by f1, b by f2; join操作默认的是内连接,只有两边都匹配才会保留 需要用null补位的那边需要知道它的模式: 如果是左外连接,需要知道右边的数据集的 ...
- Pig foreach用法举例
foreach:一行一行的遍历数据,处理一行的数据,然后返回一个tuple. users = load '/users.data'; 1)别名引用 f = foreach users genera ...
- Pig load 用法举例
users = load '/users.data' using PigStorage() as (name:chararray, age:int, address:chararray); loa ...
- Pig distinct用法举例
dst = distinct data: DISTINCT只能对整个记录(整行)去重,不能在字段级别去重. 触发reduce阶段 data = load 'data'; distinct ...
- Pig store用法举例
store:将数据存储到HDFS等文件系统里 将数据保存到/data目录 store data into '/data'; 以逗号为分隔符 store data into '/data' usin ...
- Pig filter用法举例
filter:过滤数据,只有符合特定条件的数据才会被保留下来,然后进入下一个数据流. 1)等值比较 filter data by $0 == 1 filter data by $0 != 1 ...
- c++ stl容器set成员函数介绍及set集合插入,遍历等用法举例
c++ stl集合set介绍 c++ stl集合(Set)是一种包含已排序对象的关联容器.set/multiset会根据待定的排序准则,自动将元素排序.两者不同在于前者不允许元素重复,而后者允许. 1 ...
随机推荐
- (转)Windows下MySQL :GUI安装和使用(MySQL GUI tools)
原文:http://blog.csdn.net/dahunbi/article/details/52970815 MySQL GUI Tools是MySQL官方提供的图形化管理工具,功能很强大,值得推 ...
- PL/SQL DEVELOPER数字超长显示了科学计数法
问题: 最近在做项目中,ID使用了长整形,10进制数值大约长度17位,在pl/sql developer 上数值由科学计数法显示. 在查看时不是很方便,且数值进行了省略显示,不准确. 解决方法: 在t ...
- sublime text3在交互时解决input()函数无法使用的问题
1,打开sublime text3工具栏,依次点击View->Show Console菜单打开命令行, 2,在命令行里 输入代码 import urllib.request,o ...
- Css权重解析
Css权重解析 关于CSS权重,我们需要一套计算公式来去计算,这个就是 CSS Specificity,我们称为CSS 特性或称非凡性,它是一个衡量CSS值优先级的一个标准 具体规范入如下: spec ...
- ERROR:org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 解决方案
我尝试的过程如下 1. 时间没有同步 用date命令看一下每个机器 如果时间差距大 说明确实有问题 ** 配置时间服务器 ** 检查时区 $ d ...
- thinkphp 实现rabbitMq常驻进程消费队列
1,项目一级目录新建一个server文件 #!/usr/bin/env php <?php try { require __DIR__ . "/start.php"; } c ...
- xampp启动失败 Apache shutdown unexpectedly
我知道是端口被我的虚拟机占用,但是我的虚拟机也要同时打开,所以 我需要修改xampp的端口 1, 2修改二个配置文件 修改httpd.conf 修改httpd-ssl.conf 4,重启xampp
- Cacheable redis 宕机
使用Cacheable注解Redis方法时,如果Redis服务器挂了,就直接抛出异常了, java.net.ConnectException: Connection refused: connect ...
- 通过spark-sql快速读取hive中的数据
1 配置并启动 1.1 创建并配置hive-site.xml 在运行Spark SQL CLI中需要使用到Hive Metastore,故需要在Spark中添加其uris.具体方法是将HIVE_CON ...
- i.mx6 Android5.1.1 build解析
参考资料:理解 Android Build 系统 把总结放前面: 1. 常用编译命令 make clean 执行清理,等同于:rm -rf out/. make sdk 编译出 Android 的 S ...