实现Hadoop的Writable接口Implementing Writable interface of Hadoop
As we saw in the previous posts, Hadoop makes an heavy use of network transmissions for executing its jobs. As Doug Cutting (the creator of Hadoop) explaines in this post on the Lucene mailing list, java.io.Serializable is too heavy for Hadoop's needs and so a new interface has been developed: Writable. Every object you need to emit from mapper to reducers or as an output has to implement this interface in order to make Hadoop trasmit the data from/to the nodes in the cluster.
Hadoop comes with several wrappers around primitive types and widely used classes in Java:
Java primitive | Writable implementation |
---|---|
boolean | BooleanWritable |
byte | ByteWritable |
short | ShortWritable |
int | IntWritable VIntWritable |
float | FloatWritable |
long | LongWritable VLongWritable |
double | DoubleWritable |
Java class | Writable implementation |
---|---|
String | Text |
byte[] | BytesWritable |
Object | ObjectWritable |
null | NullWritable |
Java collection | Writable implementation |
---|---|
array | ArrayWritable ArrayPrimitiveWritable TwoDArrayWritable |
Map | MapWritable |
SortedMap | SortedMapWritable |
enum | EnumWritable |
For example, if we need a mapper to emit a String, we need to use a Text object wrapped around the string we want to emit.
The interface Writable defines two methods:
- public void write(DataOutput dataOutput) throws IOException
- public void readFields(DataInput dataInput) throws IOException
The first method, write() is used for writing the data onto the stream, while the second method, readFields(), is used for reading data from the stream. The wrappers we saw above just send and receive their binary representation over a stream.
Since Hadoop needs also to sort data while in the shuffle-and-sort phase, it needs also the Comparable interface to be implemented, so it defines the WritableComparable interface which is an interface that implements both Writable and Comparable.
If we need to emit a custom object which has no default wrapper, we need to create a class that implements the WritableComparable interface. In the mean example we saw on this post, we used the SumCount class, which is a class that implements WritableComparable (the source code is available on github):
public class SumCount implements WritableComparable<SumCount> { DoubleWritable sum;
IntWritable count; public SumCount() {
set(new DoubleWritable(0), new IntWritable(0));
} public SumCount(Double sum, Integer count) {
set(new DoubleWritable(sum), new IntWritable(count));
} public void set(DoubleWritable sum, IntWritable count) {
this.sum = sum;
this.count = count;
} public DoubleWritable getSum() {
return sum;
} public IntWritable getCount() {
return count;
} public void addSumCount(SumCount sumCount) {
set(new DoubleWritable(this.sum.get() + sumCount.getSum().get()), new IntWritable(this.count.get() + sumCount.getCount().get()));
} @Override
public void write(DataOutput dataOutput) throws IOException { sum.write(dataOutput);
count.write(dataOutput);
} @Override
public void readFields(DataInput dataInput) throws IOException { sum.readFields(dataInput);
count.readFields(dataInput);
} @Override
public int compareTo(SumCount sumCount) { // compares the first of the two values
int comparison = sum.compareTo(sumCount.sum); // if they're not equal, return the value of compareTo between the "sum" value
if (comparison != 0) {
return comparison;
} // else return the value of compareTo between the "count" value
return count.compareTo(sumCount.count);
} @Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false; SumCount sumCount = (SumCount) o; return count.equals(sumCount.count) && sum.equals(sumCount.sum);
} @Override
public int hashCode() {
int result = sum.hashCode();
result = 31 * result + count.hashCode();
return result;
}
}
As we can see, it's very easy to code the two methods defined by the Writable interface: they just call the write() and readFields() method of the primitive types of the properties of SumCount class; it's important setting the properties in the same order in both read() and writeFields(), otherwise it will not work. The other methods of this class are the getters, setters and the methods needed by the Comparable interface, which should be nothing new to a Java developer.
from: http://andreaiacono.blogspot.com/2014/05/implementing-writable-interface-of.html
实现Hadoop的Writable接口Implementing Writable interface of Hadoop的更多相关文章
- 如何实现多个接口Implementing Multiple Interface
4.实现多个接口Implementing Multiple Interface 接口的优势:马克-to-win:类可以实现多个接口.与之相反,类只能继承一个超类(抽象类或其他类). A class c ...
- Hadoop中序列化与Writable接口
学习笔记,整理自<Hadoop权威指南 第3版> 一.序列化 序列化:序列化是将 内存 中的结构化数据 转化为 能在网络上传输 或 磁盘中进行永久保存的二进制流的过程:反序列化:序列化的逆 ...
- hadoop中的序列化与Writable接口
本文地址:http://www.cnblogs.com/archimedes/p/hadoop-writable-interface.html,转载请注明源地址. 简介 序列化和反序列化就是结构化对象 ...
- Hadoop序列化与Writable接口(一)
Hadoop序列化与Writable接口(一) 序列化 序列化(serialization)是指将结构化的对象转化为字节流,以便在网络上传输或者写入到硬盘进行永久存储:相对的反序列化(deserial ...
- Hadoop Serialization hadoop序列化详解(最新版) (1)【java和hadoop序列化比较和writable接口】
初学java的人肯定对java序列化记忆犹新.最开始很多人并不会一下子理解序列化的意义所在.这样子是因为很多人还是对java最底层的特性不是特别理解,当你经验丰富,对java理解更加深刻之后,你就会发 ...
- eclipse 提交作业到JobTracker Hadoop的数据类型要求必须实现Writable接口
问:在eclipse中的写的代码如何提交作业到JobTracker中的哪?答:(1)在eclipse中调用的job.waitForCompletion(true)实际上执行如下方法 connect() ...
- Hadoop基础-序列化与反序列化(实现Writable接口)
Hadoop基础-序列化与反序列化(实现Writable接口) 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.序列化简介 1>.什么是序列化 序列化也称串行化,是将结构化 ...
- Hadoop序列化与Writable接口(二)
Hadoop序列化与Writable接口(二) 上一篇文章Hadoop序列化与Writable接口(一)介绍了Hadoop序列化,Hadoop Writable接口以及如何定制自己的Writable类 ...
- hadoop学习第四天-Writable和WritableComparable序列化接口的使用&&MapReduce中传递javaBean的简单例子
一. 为什么javaBean要继承Writable和WritableComparable接口? 1. 如果一个javaBean想要作为MapReduce的key或者value,就一定要实现序列化,因为 ...
随机推荐
- 二、 sql*plus常用命令
一.sys用户和system用户Oracle安装会自动的生成sys用户和system用户(1).sys用户是超级用户,具有最高权限,具有sysdba角色,有create database的权限,该用户 ...
- PHP程序员未来路在何方
PHP 从诞生到现在已经有20多年历史,从Web时代兴起到移动互联网退潮,互联网领域各种编程语言和技术层出不穷, Node.js . GO . Python 不断地在挑战 PHP 的地位.这些技术的推 ...
- Ubuntu18.04 之jdk安装与环境配置
1.oracle官网下载压缩包. 下载地址为: https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133 ...
- 如何保证Redis中的数据都是热点数据
redis 内存数据集大小上升到一定大小的时候,就会施行数据淘汰策略.redis 提供 6种数据淘汰策略:volatile-lru:从已设置过期时间的数据集(server.db[i].expires) ...
- vdp配置
转:http://jiangjianlong.blog.51cto.com/3735273/1902879 本文将介绍VDP 6.1.2的部署与配置,主要内容包括部署VDP的OVA模板.初始化配置VD ...
- keystone 认证深度研究分析
一.Keystone Token深度概述 Keystone作为OpenStack项目基础认证模块,目前支持的token类型分别是uuid.pkiz.pki.fernet. 首先,简要叙述一下这四种类型 ...
- 读书笔记_Effective_C++_条款三十二:确定你的public继承继承塑模出is-a关系
这一条款是说的是公有继承的逻辑,如果使用继承,而且继承是公有继承的话,一定要确保子类是一种父类(is-a关系).这种逻辑可能与生活中的常理不相符,比如企鹅是生蛋的,所有企鹅是鸟类的一种,直观来看,我们 ...
- Tasker to auto record incoming or outgoing call
Tasker to auto record incoming or outgoing call most of time, i was thinking of tasker can do the jo ...
- XDM、GDM和KDM
XDM.GDM.KDM是三种X Window的显示管理器 (1)XDM(默认的X Window System Display Manager)(2)GDM(gnome提供的Display Manage ...
- systemtap 2.8 news
* What's new in version 2.8, 2015-06-17 - SystemTap has improved support for probing golang programs ...