解读:CombineFileInputFormat类
MR-Job默认的输入格式FileInputFormat为每一个小文件生成一个切片。CombineFileInputFormat通过将多个“小文件”合并为一个"切片"(在形成切片的过程中也考虑同一节点、同一机架的数据本地性),让每一个Mapper任务可以处理更多的数据,从而提高MR任务的执行速度。详见 MR案例:CombineFileInputFormat
1).三个重要的属性:
- maxSplitSize:切片大小最大值。可通过属性 "mapreduce.input.fileinputformat.split.maxsize" 或 CombineFileInputFormat.setMaxInputSplitSize()方法进行设置【不设置,则所有输入只启动一个map任务】
- minSplitSizeNode:同一节点的数据块形成切片时,切片大小的最小值。可通过属性 "mapreduce.input.fileinputformat.split.minsize.per.node" 或 CombineFileInputFormat.setMinSplitSizeNode()方法进行设置
- minSplitSizeRack:同一机架的数据块形成切片时,切片大小的最小值。可通过属性 "mapreduce.input.fileinputformat.split.minsize.per.rack" 或 CombineFileInputFormat.setMinSplitSizeRack()方法进行设置
- 大小关系:maxSplitSize > minSplitSizeNode > minSplitSizeRack
2).切片的形成过程:
2.1. 不断迭代节点列表,逐个节点 (以数据块为单位) 形成切片(Local Split)
a. 如果maxSplitSize == 0,则整个节点上的Block数据形成一个切片
b. 如果maxSplitSize != 0,遍历并累加每个节点上的数据块,如果累加数据块大小 >= maxSplitSize,则将这些数据块形成一个切片。继续该过程,直到剩余数据块累加大小 < maxSplitSize 。则进行下一步
a. 遍历并累加这个机架上所有节点的数据块 (这些数据块即上一步遗留下来的数据块),如果累加数据块大小 >= maxSplitSize,则将这些数据块形成一个切片。继续该过程,直到剩余数据块累加大小<maxSplitSize。则进行下一步
b. 如果剩余数据块累加大小 >= minSplitSizeRack,则将这些剩余数据块形成一个切片。如果剩余数据块累加大小 < minSplitSizeRack,则这些数据块留待后续处理
3).源码:getSplit()
@Override
public List<InputSplit> getSplits(JobContext job)
throws IOException {
long minSizeNode = ;
long minSizeRack = ;
long maxSize = ;
Configuration conf = job.getConfiguration(); // 通过setxxxSplitSize()方法设置的参数值会覆盖掉从配置文件中读取的参数值
if (minSplitSizeNode != ) {
minSizeNode = minSplitSizeNode;
} else {
minSizeNode = conf.getLong(SPLIT_MINSIZE_PERNODE, );
}
if (minSplitSizeRack != ) {
minSizeRack = minSplitSizeRack;
} else {
minSizeRack = conf.getLong(SPLIT_MINSIZE_PERRACK, );
}
if (maxSplitSize != ) {
maxSize = maxSplitSize;
} else { //如果maxSize没有配置,整个Node生成一个Split
maxSize = conf.getLong("mapreduce.input.fileinputformat.split.maxsize", );
}
if (minSizeNode != && maxSize != && minSizeNode > maxSize) {
throw new IOException("Minimum split size pernode " + minSizeNode +
" cannot be larger than maximum split size " +
maxSize);
}
if (minSizeRack != && maxSize != && minSizeRack > maxSize) {
throw new IOException("Minimum split size per rack " + minSizeRack +
" cannot be larger than maximum split size " +
maxSize);
}
if (minSizeRack != && minSizeNode > minSizeRack) {
throw new IOException("Minimum split size per node " + minSizeNode +
" cannot be larger than minimum split " +
"size per rack " + minSizeRack);
} //获取输入路径中的所有文件
List<FileStatus> stats = listStatus(job);
List<InputSplit> splits = new ArrayList<InputSplit>();
if (stats.size() == ) {
return splits;
} // 迭代为每个过滤池中的文件生成切片
//一个切片中的数据块只可能来自于同一个过滤池,但可以来自同一个过滤池中的不同文件
for (MultiPathFilter onepool : pools) {
ArrayList<FileStatus> myPaths = new ArrayList<FileStatus>();
//获取满足当前过滤池实例onepool的所有文件myPaths
for (Iterator<FileStatus> iter = stats.iterator(); iter.hasNext();) {
FileStatus p = iter.next();
if (onepool.accept(p.getPath())) {
myPaths.add(p); // add it to my output set
iter.remove();
}
}
//为mypaths中的文件生成切片
getMoreSplits(job, myPaths, maxSize, minSizeNode, minSizeRack, splits);
} //为不属于任何过滤池的文件生成切片
getMoreSplits(job, stats, maxSize, minSizeNode, minSizeRack, splits); //free up rackToNodes map
rackToNodes.clear();
return splits;
}
4).源码:getMoreSplits()
无论是满足某过滤池实例 onePool 条件的文件,还是不属于任何过滤池的文件,可以笼统地理解为 "一批文件",getMoreSplits()就是为这一批文件生成切片的。
/**
* Return all the splits in the specified set of paths
*/
private void getMoreSplits(JobContext job, List<FileStatus> stats,
long maxSize, long minSizeNode, long minSizeRack,
List<InputSplit> splits)
throws IOException {
Configuration conf = job.getConfiguration(); //OneFileInfo类:代表一个文件
OneFileInfo[] files;
//rackToBlocks:机架和数据块的对应关系,即某一个机架上有哪些数据块;
HashMap<String, List<OneBlockInfo>> rackToBlocks =
new HashMap<String, List<OneBlockInfo>>(); //blockToNodes:数据块与节点的对应关系,即一块数据块的“拷贝”位于哪些节点
HashMap<OneBlockInfo, String[]> blockToNodes =
new HashMap<OneBlockInfo, String[]>(); //nodeToBlocks:节点和数据块的对应关系,即某一个节点上有哪些数据块;
HashMap<String, Set<OneBlockInfo>> nodeToBlocks =
new HashMap<String, Set<OneBlockInfo>>(); files = new OneFileInfo[stats.size()];
if (stats.size() == ) {
return;
} /**
* 迭代这"一批文件",为每一个文件构建OneFileInfo对象
* OneFileInfo对象在构建过程中维护了上述三个对应关系的信息。
* 迭代完成之后,即可以认为数据块、节点、机架相互之间的对应关系已经建立完毕
* 接下来可以根据这些信息生成切片
*/
long totLength = ;
int i = ;
for (FileStatus stat : stats) {
files[i] = new OneFileInfo(stat, conf, isSplitable(job, stat.getPath()),
rackToBlocks, blockToNodes, nodeToBlocks,
rackToNodes, maxSize);
totLength += files[i].getLength();
} //切片的形成过程
createSplits(nodeToBlocks, blockToNodes, rackToBlocks, totLength,
maxSize, minSizeNode, minSizeRack, splits);
}
5).源码:createSplits()
@VisibleForTesting
void createSplits(Map<String, Set<OneBlockInfo>> nodeToBlocks,
Map<OneBlockInfo, String[]> blockToNodes,
Map<String, List<OneBlockInfo>> rackToBlocks,
long totLength,
long maxSize,
long minSizeNode,
long minSizeRack,
List<InputSplit> splits
) { //保存当前切片所包含的数据块
ArrayList<OneBlockInfo> validBlocks = new ArrayList<OneBlockInfo>(); //保存当前切片的大小
long curSplitSize = ; int totalNodes = nodeToBlocks.size();
long totalLength = totLength; Multiset<String> splitsPerNode = HashMultiset.create();
Set<String> completedNodes = new HashSet<String>(); while(true) {
// it is allowed for maxSize to be 0. Disable smoothing load for such cases //逐个节点(数据块)形成切片
// process all nodes and create splits that are local to a node. Generate
// one split per node iteration, and walk over nodes multiple times to
// distribute the splits across nodes.
for (Iterator<Map.Entry<String, Set<OneBlockInfo>>> iter = nodeToBlocks
.entrySet().iterator(); iter.hasNext();) {
Map.Entry<String, Set<OneBlockInfo>> one = iter.next(); String node = one.getKey(); // Skip the node if it has previously been marked as completed.
if (completedNodes.contains(node)) {
continue;
} Set<OneBlockInfo> blocksInCurrentNode = one.getValue(); // for each block, copy it into validBlocks. Delete it from
// blockToNodes so that the same block does not appear in
// two different splits.
Iterator<OneBlockInfo> oneBlockIter = blocksInCurrentNode.iterator();
while (oneBlockIter.hasNext()) {
OneBlockInfo oneblock = oneBlockIter.next(); // Remove all blocks which may already have been assigned to other
// splits.
if(!blockToNodes.containsKey(oneblock)) {
oneBlockIter.remove();
continue;
} validBlocks.add(oneblock);
blockToNodes.remove(oneblock);
curSplitSize += oneblock.length; // if the accumulated split size exceeds the maximum, then
// create this split. //如果数据块累积大小大于或等于maxSize,则形成一个切片
if (maxSize != && curSplitSize >= maxSize) {
// create an input split and add it to the splits array
addCreatedSplit(splits, Collections.singleton(node), validBlocks);
totalLength -= curSplitSize;
curSplitSize = ; splitsPerNode.add(node); // Remove entries from blocksInNode so that we don't walk these
// again.
blocksInCurrentNode.removeAll(validBlocks);
validBlocks.clear(); // Done creating a single split for this node. Move on to the next
// node so that splits are distributed across nodes.
break;
} }
if (validBlocks.size() != ) {
// This implies that the last few blocks (or all in case maxSize=0)
// were not part of a split. The node is complete. // if there were any blocks left over and their combined size is
// larger than minSplitNode, then combine them into one split.
// Otherwise add them back to the unprocessed pool. It is likely
// that they will be combined with other blocks from the
// same rack later on.
// This condition also kicks in when max split size is not set. All
// blocks on a node will be grouped together into a single split. // 如果剩余数据块大小大于或等于minSizeNode,则将这些数据块构成一个切片;
// 如果剩余数据块大小小于minSizeNode,则将这些数据块归还给blockToNodes,交由后期“同一机架”过程处理 if (minSizeNode != && curSplitSize >= minSizeNode
&& splitsPerNode.count(node) == ) {
// haven't created any split on this machine. so its ok to add a
// smaller one for parallelism. Otherwise group it in the rack for
// balanced size create an input split and add it to the splits
// array
addCreatedSplit(splits, Collections.singleton(node), validBlocks);
totalLength -= curSplitSize;
splitsPerNode.add(node);
// Remove entries from blocksInNode so that we don't walk this again.
blocksInCurrentNode.removeAll(validBlocks);
// The node is done. This was the last set of blocks for this node.
} else {
// Put the unplaced blocks back into the pool for later rack-allocation.
for (OneBlockInfo oneblock : validBlocks) {
blockToNodes.put(oneblock, oneblock.hosts);
}
}
validBlocks.clear();
curSplitSize = ;
completedNodes.add(node);
} else { // No in-flight blocks.
if (blocksInCurrentNode.size() == ) {
// Node is done. All blocks were fit into node-local splits.
completedNodes.add(node);
} // else Run through the node again.
}
} // Check if node-local assignments are complete.
if (completedNodes.size() == totalNodes || totalLength == ) {
// All nodes have been walked over and marked as completed or all blocks
// have been assigned. The rest should be handled via rackLock assignment.
LOG.info("DEBUG: Terminated node allocation with : CompletedNodes: "
+ completedNodes.size() + ", size left: " + totalLength);
break;
}
}
//逐个机架(数据块)形成切片
// if blocks in a rack are below the specified minimum size, then keep them
// in 'overflow'. After the processing of all racks is complete, these
// overflow blocks will be combined into splits.
//overflowBlocks用于保存“同一机架”过程处理之后剩余的数据块
ArrayList<OneBlockInfo> overflowBlocks = new ArrayList<OneBlockInfo>();
Set<String> racks = new HashSet<String>(); // Process all racks over and over again until there is no more work to do.
while (blockToNodes.size() > ) { // Create one split for this rack before moving over to the next rack.
// Come back to this rack after creating a single split for each of the
// remaining racks.
// Process one rack location at a time, Combine all possible blocks that
// reside on this rack as one split. (constrained by minimum and maximum
// split size). //依次处理每个机架
for (Iterator<Map.Entry<String, List<OneBlockInfo>>> iter =
rackToBlocks.entrySet().iterator(); iter.hasNext();) { Map.Entry<String, List<OneBlockInfo>> one = iter.next();
racks.add(one.getKey());
List<OneBlockInfo> blocks = one.getValue(); // for each block, copy it into validBlocks. Delete it from
// blockToNodes so that the same block does not appear in
// two different splits.
boolean createdSplit = false; //依次处理该机架的每个数据块
for (OneBlockInfo oneblock : blocks) {
if (blockToNodes.containsKey(oneblock)) {
validBlocks.add(oneblock);
blockToNodes.remove(oneblock);
curSplitSize += oneblock.length; // if the accumulated split size exceeds the maximum, then
// create this split.如果数据块累积大小大于或等于maxSize,则形成一个切片
if (maxSize != && curSplitSize >= maxSize) {
// create an input split and add it to the splits array
addCreatedSplit(splits, getHosts(racks), validBlocks);
createdSplit = true;
break;
}
}
} // if we created a split, then just go to the next rack
if (createdSplit) {
curSplitSize = ;
validBlocks.clear();
racks.clear();
continue;
} if (!validBlocks.isEmpty()) { //如果剩余数据块大小大于或等于minSizeRack,则将这些数据块构成一个切片
if (minSizeRack != && curSplitSize >= minSizeRack) {
// if there is a minimum size specified, then create a single split
// otherwise, store these blocks into overflow data structure
addCreatedSplit(splits, getHosts(racks), validBlocks);
} else {
// There were a few blocks in this rack that
// remained to be processed. Keep them in 'overflow' block list.
// These will be combined later. //如果剩余数据块大小小于minSizeRack,则将这些数据块加入overflowBlocks
overflowBlocks.addAll(validBlocks);
}
}
curSplitSize = ;
validBlocks.clear();
racks.clear();
}
} assert blockToNodes.isEmpty();
assert curSplitSize == ;
assert validBlocks.isEmpty();
assert racks.isEmpty(); //遍历并累加剩余数据块
for (OneBlockInfo oneblock : overflowBlocks) {
validBlocks.add(oneblock);
curSplitSize += oneblock.length; // This might cause an exiting rack location to be re-added,
// but it should be ok.
for (int i = ; i < oneblock.racks.length; i++) {
racks.add(oneblock.racks[i]);
} // if the accumulated split size exceeds the maximum, then
// create this split.
// 如果剩余数据块大小大于或等于maxSize,则将这些数据块构成一个切片
if (maxSize != && curSplitSize >= maxSize) {
// create an input split and add it to the splits array
addCreatedSplit(splits, getHosts(racks), validBlocks);
curSplitSize = ;
validBlocks.clear();
racks.clear();
}
} //剩余数据块形成一个切片
if (!validBlocks.isEmpty()) {
addCreatedSplit(splits, getHosts(racks), validBlocks);
}
}
解读:CombineFileInputFormat类的更多相关文章
- 【java源码】解读HashTable类背后的实现细节
HashTable这个类实现了哈希表从key映射到value的数据结构形式.任何非null的对象都可以作为key或者value. 要在hashtable中存储和检索对象,作为key的对象必须实现has ...
- 详细解读LruCache类
LruCache是android提供的一个缓存工具类,其算法是最近最少使用算法.它把最近使用的对象用“强引用”存储在LinkedHashMap中,并且把最近最少使用的对象在缓存值达到预设定值之前就从内 ...
- 逐步解读String类(一)
一句题外话 面试刚入行的Java新手,侧重基础知识:面试有多年工作经验的老鸟,多侧重对具体问题的解决策略. 从一类面试题说起 考察刚入行菜鸟对基础知识的掌握程度,面试官提出关于String类的内容挺常 ...
- MR案例:CombineFileInputFormat
CombineFileInputFormat是一个抽象类.Hadoop提供了两个实现类CombineTextInputFormat和CombineSequenceFileInputFormat. 此案 ...
- 详细解读Volley(三)—— ImageLoader & NetworkImageView
ImageLoader是一个加载网络图片的封装类,其内部还是由ImageRequest来实现的.但因为源码中没有提供磁盘缓存的设置,所以咱们还需要去源码中进行修改,让我们可以更加自如的设定是否进行磁盘 ...
- Java类的继承与多态特性-入门笔记
相信对于继承和多态的概念性我就不在怎么解释啦!不管你是.Net还是Java面向对象编程都是比不缺少一堂课~~Net如此Java亦也有同样的思想成分包含其中. 继承,多态,封装是Java面向对象的3大特 ...
- MR案例:小文件处理方案
HDFS被设计来存储大文件,而有时候会有大量的小文件生成,造成NameNode资源的浪费,同时也影响MapReduce的处理效率.有哪些方案可以合并这些小文件,或者提高处理小文件的效率呢? 1). 所 ...
- Asp.net 面向接口可扩展框架之核心容器(含测试代码下载)
新框架的容器部分终于调通了!容器实在太重要了,所以有用了一个名词叫“核心容器”. 容器为什么那么重要呢?这个有必要好好说道说道. 1.首先我们从框架名称面向接口编程说起,什么是面向接口编程?(这个度娘 ...
- jdk研究——java.lang
jdk研究 volatile 是什么意思? 如何看jdk源码? 如何调试源码!---------仔细解读关键类,关键代码,常用的api的解释! 自己有疑问的不懂地方-------- 不懂的太多怎么办. ...
随机推荐
- 170404、java版ftp操作工具类
package com.rick.utils; import java.io.File; import java.io.FileInputStream; import java.io.FileNotF ...
- 160229-01、web页面常用功能js实现
web页面常用功能js实现 1.网页未加载时弹出新窗口 <body onunload="window.open('http://www.a68.cn');">< ...
- KVM虚拟机克隆及快照管理
一,克隆 查看虚拟机硬盘位置(其中centos1为虚拟机名称) virsh edit centos1 克隆(centos1为需要克隆的虚拟机名称centos2为克隆后的虚拟机名称CentOS2.qco ...
- FROM_UNIXTIME(unix_timestamp), FROM_UNIXTIME(unix_timestamp,format)
w SELECT ro.*, FROM_UNIXTIME(ro.wstart,'%Y%m%d') FROM room_order ro
- LayoutInflater的动态增加控件
在实际开发中LayoutInflater这个类是非常有用的,它的作用类似于 findViewById(),不同点是LayoutInflater是用来找layout下xml布局文件. 而findView ...
- sql server 测试delete后数据空间情况
总结结论: [1]如果是索引组织表,删除的数据空间是会被文件设置为可用状态,其他表都可以使用. [2]如果是堆表,删除数据空间也会设置为可用状态,但是只能给被删除数据的表使用. [3]truncate ...
- C#线程池ThreadPool
线程池可以看做容纳线程的容器: 一个应用程序最多只能有一个线程池: 设置线程数量ThreadPool.SetMaxThreads(initDownCardThreadPool, maxDownCard ...
- MyEclipse 10的使用技巧
默认快捷键 :Shift+Alt+s 然后选择generater getter and setter,这是快捷键.或者右键source里边有 generater getter and setter. ...
- sql之密码保存
HashBytes (Transact-SQL) 其他版本 返回其输入的 MD2.MD4.MD5.SHA 或 SHA1 哈希值. Transact-SQL 语法约定 语法 Has ...
- python的曲线平滑工具,及python画一条线中包含不同粗细不同颜色的画线方法
from scipy.signal import savgol_filter import matplotlib.pyplot as plt cc = savgol_filter(c, 99, 1) ...