hadoop2.2 API http://hadoop.apache.org/docs/r0.23.9/api/index.html junit API http://junit.org/javadoc/4.10/ mrunit mrunit API http://mrunit.apache.org/documentation/javadocs/1.0.0/index.html mrunit package-summary http://mrunit.apache.org/documentati…
1. Reading data from a hadoop URL 说明:想要让java从hadoop的dfs里读取数据,则java 必须能够识别hadoop hdfs URL schema, 因此我们应该将hdfs的FsUrlStreamHandlerFactory作为一个实例提供给java, java的setURLStreamHandlerFactory方法可以实现此功能: 注意:此方法有缺陷,由于在java里,setURLStreamHandlerFactorymethod在每一个JVM里…
mr自带的例子中的源码SecondarySort,我重新写了一下,基本没变. 这个例子中定义的map和reduce如下,关键是它对输入输出类型的定义:(java泛型编程) public static class Map extends Mapper<LongWritable, Text, IntPair, IntWritable> public static class Reduce extends Reducer<IntPair, NullWritable, IntWritable,…
引用地址:http://www.cnblogs.com/lucius/p/3442381.html examples: Overview This document explains how to write unit tests for your map reduce code, and testing your mapper and reducer logic on your desktop without having any Hadoop environment setup. Let's…
/* matrix-matrix multiplication on Hadoop A x B = C constraint: A, B, C must be of the same size I use this to evaluate the efficiency of Hadoop for matrix multiplication, so I really don't care to handle non-square matrices. ===Data preparation====…
examples: Overview This document explains how to write unit tests for your map reduce code, and testing your mapper and reducer logic on your desktop without having any Hadoop environment setup. Let's look at some code For testing your map and reduce…
继承关系: 1. java.util Interface Map.Entry<K,V> description: public static interface Map.Entry<K,V> methods: Modifier and Type Method and Description boolean equals(Object o) Compares the specified object with this entry for equality. K getKey()…
Becareful! All nodes include need to install ruby! #!/usr/bin/ruby # Ruby code for map.rb ARGF.each do |line| # remove any newline line = line.chomp # do nothing will lines shorter than 2 characters next if ! line || line.length < 2 # grab our key…
