In the last post and in the preceding one we saw how to write a MapReduce program for finding the top-n items of a data set. The difference between the two was that the first program (which we call basic) emitted to the reducers every single item read from input, while the second (which we call enhanced) made a partial computation and emitted only a subset of the input. The enhanced top-n optimizes network transmissions (the less the key-value pairs emitted, the less network is used for transmitting them from mapper to reducer) and reduces the number of keys shuffled and sorted; but this is obtained at the cost of rewriting of the mapper.

If we look at the code of the mapper of the enhanced top-n , we can see that it implements the idea behind the reducer: it uses a Map for making a partial count of the words and emits every word only once; looking at the reducer's code, we see that it implements the same idea. If we could execute the code of the reducer of the basic top-n after the mapper has run on every machine (with its subset of data), we would obtain exactly the same result than rewriting the mapper as in the enhanced. This is exactly what Hadoop combiners do: they're executed just after the mapper on every machine for improving performance. For telling Hadoop which class to use as a combiner, we can use the Job.setCombinerClass() method.

Caution: using the reducer as a combiner works only if the function we're computing is both commutative (a + b = b + a) and associative (a + (b + c) = (a + b) + c). 
Let's make an example. Suppose we're analyzing the traffic of a website and we have an input file with the number of visits per day like this (YYYYMMDD value):

20140401 100
20140331 1000
20140330 1300
20140329 5100
20140328 1200

We want to find which is the day with the highest number of visits. 
Let's say that we have two mappers; the first one receives the first three lines and the second receives the last two. If we write the mapper to emit every line, the reducer will evaluate something like this:

max(100, 1000, 1300, 5100, 1200) -> 5100

and the max is 5100. 
If we use the reducer as a combiner, the reducer will evaluate something like this:

max( max(100, 1000, 1300), max(5100, 1200)) -> max( 1300, 5100) -> 5100

because each of the two mapper will evaluate locally the max function. In this case the result will be 5100 as well, since the function we're evaluating (the max function) is both commutative and associative.

Let's say that now we need to compute the average number of visits per day. If we write the mapper to emit every line of the input file, the reducer will evaluate this:

mean(100, 1000, 1300, 5100, 1200) -> 1740

which is 1740. 
If we use the reducer as a combiner, the reducer will evaluate something like this:

mean( mean(100, 1000, 1300), mean(5100, 1200)) -> mean( 800, 3150) -> 1975

because each of the two mapper will evaluate locally the max function. In this case the result will be 1975, which is obviously wrong.

So, if we're computing a commutative and associative function and we want to improve the performance of our job, we can use our reducer as a combiner; if we want to improve performance but we're computing a function that is not commutative and associative, we have to rewrite the mapper or to write a new combiner from stratch.

from: http://andreaiacono.blogspot.com/2014/03/hadoop-combiners.html

Hadoop Combiners的更多相关文章

  1. 更为详细的介绍Hadoop combiners-More about Hadoop combiners

    Hadoop combiners are a very powerful tool to speed up our computations. We already saw what a combin ...

  2. Hadoop学习笔记—8.Combiner与自定义Combiner

    一.Combiner的出现背景 1.1 回顾Map阶段五大步骤 在第四篇博文<初识MapReduce>中,我们认识了MapReduce的八大步凑,其中在Map阶段总共五个步骤,如下图所示: ...

  3. Hadoop日记Day17---计数器、map规约、分区学习

    一.Hadoop计数器 1.1 什么是Hadoop计数器 Haoop是处理大数据的,不适合处理小数据,有些大数据问题是小数据程序是处理不了的,他是一个高延迟的任务,有时处理一个大数据需要花费好几个小时 ...

  4. [BigData]关于Hadoop学习笔记第四天(PPT总结)(一)

    课程安排 Partitioner编程** 自定义排序编程** Combiner编程** 常见的MapReduce算法** ---------------------------加深拓展-------- ...

  5. hadoop调优之一:概述

    hadoop集群性能低下的常见原因 (一)硬件环境 1.CPU/内存不足,或未充分利用 2.网络原因 3.磁盘原因 (二)map任务原因 1.输入文件中小文件过多,导致多次启动和停止JVM进程.可以设 ...

  6. 一脸懵逼学习Hadoop中的MapReduce程序中自定义分组的实现

    1:首先搞好实体类对象: write 是把每个对象序列化到输出流,readFields是把输入流字节反序列化,实现WritableComparable,Java值对象的比较:一般需要重写toStrin ...

  7. hadoop两大核心之一:MapReduce总结

    MapReduce是一种分布式计算模型,由Google提出,主要用于搜索领域,MapReduce程序 本质上是并行运行的,因此可以解决海量数据的计算问题. MapReduce任务过程被分为两个处理阶段 ...

  8. hadoop调优之一:概述 分类: A1_HADOOP B3_LINUX 2015-03-13 20:51 395人阅读 评论(0) 收藏

    hadoop集群性能低下的常见原因 (一)硬件环境 1.CPU/内存不足,或未充分利用 2.网络原因 3.磁盘原因 (二)map任务原因 1.输入文件中小文件过多,导致多次启动和停止JVM进程.可以设 ...

  9. Hadoop 三剑客之 —— 分布式计算框架 MapReduce

    一.MapReduce概述 二.MapReduce编程模型简述 三.combiner & partitioner 四.MapReduce词频统计案例         4.1 项目简介      ...

随机推荐

  1. Graves of the Internet - 互联网坟墓

    Graves of the Internet - 互联网坟墓 互联网公司逝去产品列表 以此祭奠那些夕阳下的奔跑,祭奠那些逝去的青春 演示地址 点击 这里 https://myvin.github.io ...

  2. centos7 mongodb3.4 安装

    上传tgz 安装包 [root@localhost install_pack]# ll total 274840 -rw-r--r--. 1 root root 9393241 Jun 2 14:36 ...

  3. 微软企业库5.0 学习之路——扩展学习篇、库中的依赖关系注入(重构 Microsoft Enterprise Library)[转]

    这篇文章是我在patterns & practices看到的一篇有关EntLib5.0的文章,主要介绍了EntLib5.0的这次的架构变化由来,觉得很不错,大家可以看一下! 在过去几年中,依赖 ...

  4. 洛谷P2047 [NOI2007]社交网络 [图论,最短路计数]

    题目传送门 社交网络 题目描述 在社交网络(social network)的研究中,我们常常使用图论概念去解释一些社会现象.不妨看这样的一个问题.在一个社交圈子里有n个人,人与人之间有不同程度的关系. ...

  5. Jquery操作select标签的常用方法

    <select id="search"> <option value='1'>baidu</option> <option value=' ...

  6. PHP函数声明(二)

    PHP的变量的范围 1.局部变量:在函数中声明的变量就是局部变量,只能在自己的函数内部使用. 2.全局变量:函数外声明,在变量声明以后的,直到整个脚本结束前都可以使用,包括在函数中和{}中都可以使用 ...

  7. 【BZOJ 1449】 1449: [JSOI2009]球队收益 (最小费用流)

    1449: [JSOI2009]球队收益 Time Limit: 5 Sec  Memory Limit: 64 MBSubmit: 841  Solved: 483 Description Inpu ...

  8. PHP链接sqlserver出现中文乱码

    PHP通过dblib扩展链接sqlserver,使用的是freetds,出现中文乱码. 在freetds的配置文件中(/usr/local/freetds/etc/freetds.conf),[glo ...

  9. java-银行卡基本信息查询

    用于验证的请求接口: https://ccdcapi.alipay.com/validateAndCacheCardInfo.json?_input_charset=utf-8&cardNo= ...

  10. 设计模式 -- 桥接模式(Bridge)

    写在前面的话:读书破万卷,编码如有神--------------------------------------------------------------------主要内容包括: 初始桥接模式 ...