Hadoop Combiners
In the last post and in the preceding one we saw how to write a MapReduce program for finding the top-n items of a data set. The difference between the two was that the first program (which we call basic) emitted to the reducers every single item read from input, while the second (which we call enhanced) made a partial computation and emitted only a subset of the input. The enhanced top-n optimizes network transmissions (the less the key-value pairs emitted, the less network is used for transmitting them from mapper to reducer) and reduces the number of keys shuffled and sorted; but this is obtained at the cost of rewriting of the mapper.
If we look at the code of the mapper of the enhanced top-n , we can see that it implements the idea behind the reducer: it uses a Map for making a partial count of the words and emits every word only once; looking at the reducer's code, we see that it implements the same idea. If we could execute the code of the reducer of the basic top-n after the mapper has run on every machine (with its subset of data), we would obtain exactly the same result than rewriting the mapper as in the enhanced. This is exactly what Hadoop combiners do: they're executed just after the mapper on every machine for improving performance. For telling Hadoop which class to use as a combiner, we can use the Job.setCombinerClass() method.
Caution: using the reducer as a combiner works only if the function we're computing is both commutative (a + b = b + a) and associative (a + (b + c) = (a + b) + c).
Let's make an example. Suppose we're analyzing the traffic of a website and we have an input file with the number of visits per day like this (YYYYMMDD value):
20140401 100
20140331 1000
20140330 1300
20140329 5100
20140328 1200
We want to find which is the day with the highest number of visits.
Let's say that we have two mappers; the first one receives the first three lines and the second receives the last two. If we write the mapper to emit every line, the reducer will evaluate something like this:
max(100, 1000, 1300, 5100, 1200) -> 5100
and the max is 5100.
If we use the reducer as a combiner, the reducer will evaluate something like this:
max( max(100, 1000, 1300), max(5100, 1200)) -> max( 1300, 5100) -> 5100
because each of the two mapper will evaluate locally the max function. In this case the result will be 5100 as well, since the function we're evaluating (the max function) is both commutative and associative.
Let's say that now we need to compute the average number of visits per day. If we write the mapper to emit every line of the input file, the reducer will evaluate this:
mean(100, 1000, 1300, 5100, 1200) -> 1740
which is 1740.
If we use the reducer as a combiner, the reducer will evaluate something like this:
mean( mean(100, 1000, 1300), mean(5100, 1200)) -> mean( 800, 3150) -> 1975
because each of the two mapper will evaluate locally the max function. In this case the result will be 1975, which is obviously wrong.
So, if we're computing a commutative and associative function and we want to improve the performance of our job, we can use our reducer as a combiner; if we want to improve performance but we're computing a function that is not commutative and associative, we have to rewrite the mapper or to write a new combiner from stratch.
from: http://andreaiacono.blogspot.com/2014/03/hadoop-combiners.html
Hadoop Combiners的更多相关文章
- 更为详细的介绍Hadoop combiners-More about Hadoop combiners
Hadoop combiners are a very powerful tool to speed up our computations. We already saw what a combin ...
- Hadoop学习笔记—8.Combiner与自定义Combiner
一.Combiner的出现背景 1.1 回顾Map阶段五大步骤 在第四篇博文<初识MapReduce>中,我们认识了MapReduce的八大步凑,其中在Map阶段总共五个步骤,如下图所示: ...
- Hadoop日记Day17---计数器、map规约、分区学习
一.Hadoop计数器 1.1 什么是Hadoop计数器 Haoop是处理大数据的,不适合处理小数据,有些大数据问题是小数据程序是处理不了的,他是一个高延迟的任务,有时处理一个大数据需要花费好几个小时 ...
- [BigData]关于Hadoop学习笔记第四天(PPT总结)(一)
课程安排 Partitioner编程** 自定义排序编程** Combiner编程** 常见的MapReduce算法** ---------------------------加深拓展-------- ...
- hadoop调优之一:概述
hadoop集群性能低下的常见原因 (一)硬件环境 1.CPU/内存不足,或未充分利用 2.网络原因 3.磁盘原因 (二)map任务原因 1.输入文件中小文件过多,导致多次启动和停止JVM进程.可以设 ...
- 一脸懵逼学习Hadoop中的MapReduce程序中自定义分组的实现
1:首先搞好实体类对象: write 是把每个对象序列化到输出流,readFields是把输入流字节反序列化,实现WritableComparable,Java值对象的比较:一般需要重写toStrin ...
- hadoop两大核心之一:MapReduce总结
MapReduce是一种分布式计算模型,由Google提出,主要用于搜索领域,MapReduce程序 本质上是并行运行的,因此可以解决海量数据的计算问题. MapReduce任务过程被分为两个处理阶段 ...
- hadoop调优之一:概述 分类: A1_HADOOP B3_LINUX 2015-03-13 20:51 395人阅读 评论(0) 收藏
hadoop集群性能低下的常见原因 (一)硬件环境 1.CPU/内存不足,或未充分利用 2.网络原因 3.磁盘原因 (二)map任务原因 1.输入文件中小文件过多,导致多次启动和停止JVM进程.可以设 ...
- Hadoop 三剑客之 —— 分布式计算框架 MapReduce
一.MapReduce概述 二.MapReduce编程模型简述 三.combiner & partitioner 四.MapReduce词频统计案例 4.1 项目简介 ...
随机推荐
- 实现celery中出现拥挤队列时,及时发邮件通知
里面有几个常用的功能,以后值得借鉴. 如获取脚本目录,IP,获取shell返回值,发送邮件等.. 上午写完,中午测试,下午上线~~ #!/usr/bin/env python # -*- coding ...
- SGU 209. Areas
209. Areas time limit per test: 0.25 sec.memory limit per test: 65536 KB input: standardoutput: stan ...
- karma+requirejs
下面的介绍以karma能正常运行为前提,看karma系列文章:http://www.cnblogs.com/laixiangran/tag/Karma/ 目录结构 步骤 安装 npm install ...
- ref:PHP反序列化漏洞成因及漏洞挖掘技巧与案例
ref:https://www.anquanke.com/post/id/84922 PHP反序列化漏洞成因及漏洞挖掘技巧与案例 一.序列化和反序列化 序列化和反序列化的目的是使得程序间传输对象会更加 ...
- JDOM读取xml
[摘 要]JDOM是一个开源项目,它基于树型结构,利用纯JAVA的技术对XML文档实现解析.生成.序列化以及多种操作. 一.JDOM 简介 JDOM是一个开源项目,它基于树型结构,利用纯JAVA的技术 ...
- EOJ 3247 铁路修复计划
二分,最小生成树. 二分一下$k$,然后每次算最小生成树验证即可,事实证明,$cmp$函数,参数用引用还是能提高效率的,不引用一直$TLE$,时限有点卡常. 然后错误的代码好像$AC$了啊,$L$和$ ...
- VIM块操作
一.可视模式 按v启用可视模式,之后移动光标可以选择. 如: 如果想整行操作,则用大写的V,再移动光标可以按行为单位进行选择. 二.列块操作 在 word中有一个功能,按alt加鼠标拖动,可以 ...
- 同等条件下,mongo为什么比mysql快?
写操作MongoDB比传统数据库快的根本原因是Mongo使用的内存映射技术 - 写入数据时候只要在内存里完成就可以返回给应用程序,这样并发量自然就很高.而保存到硬体的操作则在后台异步完成. 读操作Mo ...
- AppDomain.CurrentDomain.BaseDirectory是什么
AppDomain.CurrentDomain.BaseDirectory 是获取基目录,它由程序集冲突解决程序用来探测程序集.由显示的路径可以看出,它代表的是程序集所在的目录,它具有读取和写入的属性 ...
- 【BZOJ 2656】2656: [Zjoi2012]数列(sequence) (高精度)
2656: [Zjoi2012]数列(sequence) Time Limit: 2 Sec Memory Limit: 128 MBSubmit: 1499 Solved: 786 Descri ...