实现Hadoop的Writable接口Implementing Writable interface of Hadoop
As we saw in the previous posts, Hadoop makes an heavy use of network transmissions for executing its jobs. As Doug Cutting (the creator of Hadoop) explaines in this post on the Lucene mailing list, java.io.Serializable is too heavy for Hadoop's needs and so a new interface has been developed: Writable. Every object you need to emit from mapper to reducers or as an output has to implement this interface in order to make Hadoop trasmit the data from/to the nodes in the cluster.
Hadoop comes with several wrappers around primitive types and widely used classes in Java:
Java primitive | Writable implementation |
---|---|
boolean | BooleanWritable |
byte | ByteWritable |
short | ShortWritable |
int | IntWritable VIntWritable |
float | FloatWritable |
long | LongWritable VLongWritable |
double | DoubleWritable |
Java class | Writable implementation |
---|---|
String | Text |
byte[] | BytesWritable |
Object | ObjectWritable |
null | NullWritable |
Java collection | Writable implementation |
---|---|
array | ArrayWritable ArrayPrimitiveWritable TwoDArrayWritable |
Map | MapWritable |
SortedMap | SortedMapWritable |
enum | EnumWritable |
For example, if we need a mapper to emit a String, we need to use a Text object wrapped around the string we want to emit.
The interface Writable defines two methods:
- public void write(DataOutput dataOutput) throws IOException
- public void readFields(DataInput dataInput) throws IOException
The first method, write() is used for writing the data onto the stream, while the second method, readFields(), is used for reading data from the stream. The wrappers we saw above just send and receive their binary representation over a stream.
Since Hadoop needs also to sort data while in the shuffle-and-sort phase, it needs also the Comparable interface to be implemented, so it defines the WritableComparable interface which is an interface that implements both Writable and Comparable.
If we need to emit a custom object which has no default wrapper, we need to create a class that implements the WritableComparable interface. In the mean example we saw on this post, we used the SumCount class, which is a class that implements WritableComparable (the source code is available on github):
public class SumCount implements WritableComparable<SumCount> { DoubleWritable sum;
IntWritable count; public SumCount() {
set(new DoubleWritable(0), new IntWritable(0));
} public SumCount(Double sum, Integer count) {
set(new DoubleWritable(sum), new IntWritable(count));
} public void set(DoubleWritable sum, IntWritable count) {
this.sum = sum;
this.count = count;
} public DoubleWritable getSum() {
return sum;
} public IntWritable getCount() {
return count;
} public void addSumCount(SumCount sumCount) {
set(new DoubleWritable(this.sum.get() + sumCount.getSum().get()), new IntWritable(this.count.get() + sumCount.getCount().get()));
} @Override
public void write(DataOutput dataOutput) throws IOException { sum.write(dataOutput);
count.write(dataOutput);
} @Override
public void readFields(DataInput dataInput) throws IOException { sum.readFields(dataInput);
count.readFields(dataInput);
} @Override
public int compareTo(SumCount sumCount) { // compares the first of the two values
int comparison = sum.compareTo(sumCount.sum); // if they're not equal, return the value of compareTo between the "sum" value
if (comparison != 0) {
return comparison;
} // else return the value of compareTo between the "count" value
return count.compareTo(sumCount.count);
} @Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false; SumCount sumCount = (SumCount) o; return count.equals(sumCount.count) && sum.equals(sumCount.sum);
} @Override
public int hashCode() {
int result = sum.hashCode();
result = 31 * result + count.hashCode();
return result;
}
}
As we can see, it's very easy to code the two methods defined by the Writable interface: they just call the write() and readFields() method of the primitive types of the properties of SumCount class; it's important setting the properties in the same order in both read() and writeFields(), otherwise it will not work. The other methods of this class are the getters, setters and the methods needed by the Comparable interface, which should be nothing new to a Java developer.
from: http://andreaiacono.blogspot.com/2014/05/implementing-writable-interface-of.html
实现Hadoop的Writable接口Implementing Writable interface of Hadoop的更多相关文章
- 如何实现多个接口Implementing Multiple Interface
4.实现多个接口Implementing Multiple Interface 接口的优势:马克-to-win:类可以实现多个接口.与之相反,类只能继承一个超类(抽象类或其他类). A class c ...
- Hadoop中序列化与Writable接口
学习笔记,整理自<Hadoop权威指南 第3版> 一.序列化 序列化:序列化是将 内存 中的结构化数据 转化为 能在网络上传输 或 磁盘中进行永久保存的二进制流的过程:反序列化:序列化的逆 ...
- hadoop中的序列化与Writable接口
本文地址:http://www.cnblogs.com/archimedes/p/hadoop-writable-interface.html,转载请注明源地址. 简介 序列化和反序列化就是结构化对象 ...
- Hadoop序列化与Writable接口(一)
Hadoop序列化与Writable接口(一) 序列化 序列化(serialization)是指将结构化的对象转化为字节流,以便在网络上传输或者写入到硬盘进行永久存储:相对的反序列化(deserial ...
- Hadoop Serialization hadoop序列化详解(最新版) (1)【java和hadoop序列化比较和writable接口】
初学java的人肯定对java序列化记忆犹新.最开始很多人并不会一下子理解序列化的意义所在.这样子是因为很多人还是对java最底层的特性不是特别理解,当你经验丰富,对java理解更加深刻之后,你就会发 ...
- eclipse 提交作业到JobTracker Hadoop的数据类型要求必须实现Writable接口
问:在eclipse中的写的代码如何提交作业到JobTracker中的哪?答:(1)在eclipse中调用的job.waitForCompletion(true)实际上执行如下方法 connect() ...
- Hadoop基础-序列化与反序列化(实现Writable接口)
Hadoop基础-序列化与反序列化(实现Writable接口) 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.序列化简介 1>.什么是序列化 序列化也称串行化,是将结构化 ...
- Hadoop序列化与Writable接口(二)
Hadoop序列化与Writable接口(二) 上一篇文章Hadoop序列化与Writable接口(一)介绍了Hadoop序列化,Hadoop Writable接口以及如何定制自己的Writable类 ...
- hadoop学习第四天-Writable和WritableComparable序列化接口的使用&&MapReduce中传递javaBean的简单例子
一. 为什么javaBean要继承Writable和WritableComparable接口? 1. 如果一个javaBean想要作为MapReduce的key或者value,就一定要实现序列化,因为 ...
随机推荐
- loadrunner日志信息
日志分两种1.在VUGEN中运行后的日志2.在controller中运行后的日志 日志设置分两步:1.首先,在VUGEN或controller中run-time setting, 选中always s ...
- Python并发编程系列之多进程(multiprocessing)
1 引言 本篇博文主要对Python中并发编程中的多进程相关内容展开详细介绍,Python进程主要在multiprocessing模块中,本博文以multiprocessing种Process类为中心 ...
- poj-2253-poj-1797_最短路练习
title: poj-2253-poj-1797_最短路练习 date: 2018-11-17 11:48:51 tags: acm 刷题 categories: ACM-最短路 概述 一道最短路的变 ...
- 【基础知识】Dom基础
[学习日记]Dom基础 1. 内容:使用JavaScript操作Dom进行DHTML开发 2. 目标:能共使用JavaScript操作Dom实现常见的DHTML效果 3. DHTML= C ...
- 《jQuery基础教程》读书笔记
最近在看<jQuery基础教程>这本书,做了点读书笔记以备回顾,不定期更新. 第一章第二章比较基础,就此略过了... 第三章 事件 jQuery中$(document).ready()与j ...
- wpf企业应用之主从结构列表
主从结构在企业级应用中相当常见,这里结合我的例子谈一下wpf中主从结构列表展示的常用做法,具体效果见 wpf企业级开发中的几种常见业务场景. 首先,Model有两种,主表对应model(假设为mode ...
- [HNOI2011]数学作业 --- 矩阵优化
[HNOI2011]数学作业 题目描述: 小 C 数学成绩优异,于是老师给小 C 留了一道非常难的数学作业题: 给定正整数 N 和 M ,要求计算\(Concatenate(1..N)\; Mod\; ...
- POJ 3678 Katu Puzzle 2-SAT 强连通分量 tarjan
http://poj.org/problem?id=3678 给m条连接两个点的边,每条边有一个权值0或1,有一个运算方式and.or或xor,要求和这条边相连的两个点经过边上的运算后的结果是边的权值 ...
- Codeforces Round #196 (Div. 2) A. Puzzles 水题
A. Puzzles Time Limit: 2 Sec Memory Limit: 60 MB 题目连接 http://acm.zju.edu.cn/onlinejudge/showProblem ...
- Linux High Availabi RHCS
-----本文大纲 简介 术语 环境配置 实现过程 命令行管理工具 ------------- 一.简介 RHCS 即 RedHat Cluster Suite ,中文意思即红帽集群套件.红帽集群套件 ...