function map(String name, String document):
// name: document name
// document: document contents
for each word w in document:
emit (w, ) function reduce(String word, Iterator partialCounts):
// word: a word
// partialCounts: a list of aggregated partial counts
sum =
for each pc in partialCounts:
sum += pc
emit (word, sum)

The prototypical MapReduce example counts the appearance of each word in a set of documents:[14]

Here, each document is split into words, and each word is counted by the map function, using the word as the result key. The framework puts together all the pairs with the same key and feeds them to the same call to reduce. Thus, this function just needs to sum all of its input values to find the total appearances of that word.

 SELECT age, AVG(contacts)
FROM social.person
GROUP BY age
ORDER BY age
function Map is
input: integer K1 between and , representing a batch of million social.person records
for each social.person record in the K1 batch do
let Y be the person's age
let N be the number of contacts the person has
produce one output record (Y,(N,))
repeat
end function function Reduce is
input: age (in years) Y
for each input record (Y,(N,C)) do
Accumulate in S the sum of N*C
Accumulate in Cnew the sum of C
repeat
let A be S/Cnew
produce one output record (Y,(A,Cnew))
end function
-- map output #: age, quantity of contacts
,
,
,
-- map output #: age, quantity of contacts
,
,
-- map output #: age, quantity of contacts
,
-- reduce step #: age, average of contacts
,

(9*3+9*2+10*1)/(3+2+1)

(9*5+10*1)/(5+1)

imagine that for a database of 1.1 billion people, one would like to compute the average number of social contacts a person has according to age

Dataflow

The frozen part of the MapReduce framework is a large distributed sort. The hot spots, which the application defines, are:

  • an input reader
  • a Map function
  • a partition function
  • a compare function
  • a Reduce function
  • an output writer

w

https://zh.wikipedia.org/wiki/MapReduce

^ "我们的灵感来自lisp和其他函数式编程语言中的古老的映射和归纳操作." -"MapReduce:大规模集群上的简单数据处理方式"

MapReduceGoogle提出的一个软件架构,用于大规模数据集(大于1TB)的并行运算。概念“Map(映射)”和“Reduce(归纳)”,及他们的主要思想,都是从函数式编程语言借来的,还有从矢量编程语言借来的特性。[1]

当前的软件实现是指定一个Map(映射)函数,用来把一组键值对映射成一组新的键值对,指定并发的Reduce(归纳)函数,用来保证所有映射的键值对中的每一个共享相同的键组。

映射和归纳

简单来說,一个映射函数就是对一些独立元素组成的概念上的列表(例如,一个测试成绩的列表)的每一个元素进行指定的操作(比如,有人发现所有学生的成绩都被高估了一分,他可以定义一个“减一”的映射函数,用来修正这个错误。)。事实上,每个元素都是被独立操作的,而原始列表没有被更改,因为这里创建了一个新的列表来保存新的答案。这就是说,Map操作是可以高度并行的,这对高性能要求的应用以及并行计算领域的需求非常有用。

而归纳操作指的是对一个列表的元素进行适当的合并(继续看前面的例子,如果有人想知道班级的平均分该怎么做?他可以定义一个归纳函数,通过让列表中的奇數(odd)或偶數(even)元素跟自己的相邻的元素相加的方式把列表减半,如此递归运算直到列表只剩下一个元素,然后用这个元素除以人数,就得到了平均分)。虽然他不如映射函数那么并行,但是因为归纳总是有一个简单的答案,大规模的运算相对独立,所以归纳函数在高度并行环境下也很有用。

分布和可靠性

MapReduce通过把对数据集的大规模操作分发给网络上的每个节点实现可靠性;每个节点会周期性的把完成的工作和状态的更新报告回来。如果一个节点保持沉默超过一个预设的时间间隔,主节点(类同Google檔案系統中的主服务器)记录下这个节点状态为死亡,并把分配给这个节点的数据发到别的节点。每个操作使用命名文件的不可分割操作以确保不会发生并行线程间的冲突;当文件被改名的时候,系统可能会把他们复制到任务名以外的另一个名字上去。(避免副作用)。

归纳操作工作方式很类似,但是由于归纳操作在并行能力较差,主节点会尽量把归纳操作调度在一个节点上,或者离需要操作的数据尽可能近的节点上了;这个特性可以满足Google的需求,因为他们有足够的带宽,他们的内部网络没有那么多的机器。

MapReduce is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster.[1][2]

A MapReduce program is composed of a Map() procedure (method) that performs filtering and sorting (such as sorting students by first name into queues, one queue for each name) and a Reduce() method that performs a summary operation (such as counting the number of students in each queue, yielding name frequencies). The "MapReduce System" (also called "infrastructure" or "framework") orchestrates the processing by marshalling the distributed servers, running the various tasks in parallel, managing all communications and data transfers between the various parts of the system, and providing for redundancy and fault tolerance.

The model is a specialization of the split-apply-combine strategy for data analysis.[3] It is inspired by the map and reduce functions commonly used in functional programming,[4] although their purpose in the MapReduce framework is not the same as in their original forms.[5] The key contributions of the MapReduce framework are not the actual map and reduce functions (which, for example, resemble the 1995 Message Passing Interface standard's[6] reduce[7] and scatter[8] operations), but the scalability and fault-tolerance achieved for a variety of applications by optimizing the execution engine. As such, a single-threaded implementation of MapReduce will usually not be faster than a traditional (non-MapReduce) implementation; any gains are usually only seen with multi-threaded implementations.[9] The use of this model is beneficial only when the optimized distributed shuffle operation (which reduces network communication cost) and fault tolerance features of the MapReduce framework come into play. Optimizing the communication cost is essential to a good MapReduce algorithm.[10]

MapReduce libraries have been written in many programming languages, with different levels of optimization. A popular open-source implementation that has support for distributed shuffles is part of Apache Hadoop. The name MapReduce originally referred to the proprietary Google technology, but has since been genericized. By 2014, Google was no longer using MapReduce as their primary Big Data processing model,[11] and development on Apache Mahout had moved on to more capable and less disk-oriented mechanisms that incorporated full map and reduce capabilities.[12]

2014 MapReduce的更多相关文章

  1. Hadoop MapReduce执行过程详解(带hadoop例子)

    https://my.oschina.net/itblog/blog/275294 摘要: 本文通过一个例子,详细介绍Hadoop 的 MapReduce过程. 分析MapReduce执行过程 Map ...

  2. PageRank算法简介及Map-Reduce实现

    PageRank对网页排名的算法,曾是Google发家致富的法宝.以前虽然有实验过,但理解还是不透彻,这几天又看了一下,这里总结一下PageRank算法的基本原理. 一.什么是pagerank Pag ...

  3. Window7中Eclipse运行MapReduce程序报错的问题

    按照文档:http://www.micmiu.com/bigdata/hadoop/hadoop2x-eclipse-mapreduce-demo/安装配置好Eclipse后,运行WordCount程 ...

  4. Hadoop学习之Mapreduce执行过程详解

    一.MapReduce执行过程 MapReduce运行时,首先通过Map读取HDFS中的数据,然后经过拆分,将每个文件中的每行数据分拆成键值对,最后输出作为Reduce的输入,大体执行流程如下图所示: ...

  5. Strata 2014 上的 AzureCAT 粉笔会谈

     本周,AzureCAT 团队非常高兴在 Strata 会议上首次集体亮相.对于那些对 AzureCAT 团队不太熟悉的人来说,我们是 Microsoft 云与企业部门一个核心的国际性团队,由大约 ...

  6. HADOOP之MAPREDUCE程序应用二

    摘要:MapReduce程序进行单词计数. 关键词:MapReduce程序  单词计数 数据源:人工构造英文文档file1.txt,file2.txt. file1.txt 内容 Hello   Ha ...

  7. Hadoop之MapReduce程序应用三

    摘要:MapReduce程序进行数据去重. 关键词:MapReduce   数据去重 数据源:人工构造日志数据集log-file1.txt和log-file2.txt. log-file1.txt内容 ...

  8. Mapreduce参数调节

    http://blog.javachen.com/2014/06/24/tuning-in-mapreduce/ 本文主要记录Hadoop 2.x版本中MapReduce参数调优,不涉及Yarn的调优 ...

  9. Hadoop MapReduce开发最佳实践(上篇)

    body{ font-family: "Microsoft YaHei UI","Microsoft YaHei",SimSun,"Segoe UI& ...

随机推荐

  1. Grow heap (frag case) to 6.437MB for 1114126-byte allocation

    本篇文章是对Grow heap (frag case) 堆内存过大的问题进行了详细的分析介绍,需要的朋友参考下 对于Android开发者来说虽然使用了可以自动管理内存的Java语言,但是对于内存管理不 ...

  2. object hook实现禁止创建文件

    object hook实现禁止创建文件 原理不说了,大伙都懂得.. 要解决的问题: ,FILE_NON_DIRECTORY_FILE); if (!NT_SUCCESS(status)) { dpri ...

  3. WifiStateMachine学习笔记

    WifiStateMachine 1. 初始化 传入接口名称wlanInterface 新建一个WiFi类型的NetworkInfo 发一个ssid为null的广播 电池 NetworkManagem ...

  4. activiti designer下载地址

    http://www.activiti.org/designer/update/ http://www.activiti.org/designer/archived/       这个地址貌似不能用了 ...

  5. SVN学习(二)——SVN 提交、更新、解决冲突等操作步骤

    1. 纳入版本控制 ①新建文件abc.txt ②在文件上点右键 ③添加后文件图标发生变化 2. 提交 ①使用TortoiseSVN可以提交具体某一个文件,或某一个目录下的所有改变.方法就是在想要提交的 ...

  6. 监听器如何获取Spring配置文件

    我们在做项目的时候,会用到监听器去获取spring的配置文件,然后从中拿出我们需要的bean出来,比如做网站首页,假设商品的后台业务逻辑都做好了,我们需要创建一个监听器,在项目启动时将首页的数据查询出 ...

  7. 【MyBatis学习02】走进MyBatis的世界

    mybatis是个持久层的框架,用来执行数据库操作的,无外乎增删改查,上一节对mybatis有了宏观上的了解后,这一篇博客主要通过一个小示例来入门mybatis,先看一下要写的示例需求: 根据用户id ...

  8. JS防后退跳转

    location.replace("http://www.baidu.com");

  9. UVa 11997 K Smallest Sums 优先队列&&打有序表&&归并

    UVA - 11997 id=18702" target="_blank" style="color:blue; text-decoration:none&qu ...

  10. C# Activator.CreateInstance 动态创建类的实例(二)

    using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.T ...