【原创】大数据基础之HDFS(1)HDFS新创建文件如何分配Datanode
HDFS中的File由Block组成,一个File包含一个或多个Block,当创建File时会创建一个Block,然后根据配置的副本数量(默认是3)申请3个Datanode来存放这个Block;
通过hdfs fsck命令可以查看一个文件具体的Block、Datanode、Rack信息,例如:
hdfs fsck /tmp/test.sql -files -blocks -locations -racks
Connecting to namenode via http://name_node:50070
FSCK started by hadoop (auth:SIMPLE) from /client for path /tmp/test.sql at Thu Dec 13 15:44:12 CST 2018
/tmp/test.sql 16 bytes, 1 block(s): OK
0. BP-436366437-name_node-1493982655699:blk_1449692331_378721485 len=16 repl=3 [/DEFAULT/server111:50010, /DEFAULT/server121:50010, /DEFAULT/server43:50010]Status: HEALTHY
Total size: 16 B
Total dirs: 0
Total files: 1
Total symlinks: 0
Total blocks (validated): 1 (avg. block size 16 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 193
Number of racks: 1
FSCK ended at Thu Dec 13 15:44:12 CST 2018 in 1 millisecondsThe filesystem under path '/tmp/test.sql' is HEALTHY
那3个Datanode是如何选择出来的?有一个优先级:
1 当前机架(相对hdfs client而言)
2 远程机架(相对hdfs client而言)
3 另一机架
4 全部随机
然后每个机架能选择几个Datanode(即maxNodesPerRack)有一个计算公式,详见代码
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer
- private int findNewDatanode(final DatanodeInfo[] original
- ) throws IOException {
- if (nodes.length != original.length + 1) {
- throw new IOException(
- new StringBuilder()
- .append("Failed to replace a bad datanode on the existing pipeline ")
- .append("due to no more good datanodes being available to try. ")
- .append("(Nodes: current=").append(Arrays.asList(nodes))
- .append(", original=").append(Arrays.asList(original)).append("). ")
- .append("The current failed datanode replacement policy is ")
- .append(dfsClient.dtpReplaceDatanodeOnFailure).append(", and ")
- .append("a client may configure this via '")
- .append(DFSConfigKeys.DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_POLICY_KEY)
- .append("' in its configuration.")
- .toString());
- }
注释:当没有找到新的datanode时会报异常,报错如下:
Caused by: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[server82:50010], original=[server.82:50010]).
The current failed datanode replacement policy is ALWAYS, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
- private void addDatanode2ExistingPipeline() throws IOException {
- ...
- final DatanodeInfo[] original = nodes;
- final LocatedBlock lb = dfsClient.namenode.getAdditionalDatanode(
- src, fileId, block, nodes, storageIDs,
- failed.toArray(new DatanodeInfo[failed.size()]),
- 1, dfsClient.clientName);
- setPipeline(lb);
- //find the new datanode
- final int d = findNewDatanode(original);
注释:会调用getAdditionalDatanode方法来获取1个新的datanode,此处略去很多调用堆栈
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault
- private DatanodeStorageInfo[] chooseTarget(int numOfReplicas,
- Node writer,
- List<DatanodeStorageInfo> chosenStorage,
- boolean returnChosenNodes,
- Set<Node> excludedNodes,
- long blocksize,
- final BlockStoragePolicy storagePolicy) {
- ...
- int[] result = getMaxNodesPerRack(chosenStorage.size(), numOfReplicas);
- numOfReplicas = result[0];
- int maxNodesPerRack = result[1];
- ...
- final Node localNode = chooseTarget(numOfReplicas, writer, excludedNodes,
- blocksize, maxNodesPerRack, results, avoidStaleNodes, storagePolicy,
- EnumSet.noneOf(StorageType.class), results.isEmpty());
注释:此处maxNodesPerRack表示每个机架最多只能分配几个datanode
- private Node chooseTarget(int numOfReplicas,
- Node writer,
- final Set<Node> excludedNodes,
- final long blocksize,
- final int maxNodesPerRack,
- final List<DatanodeStorageInfo> results,
- final boolean avoidStaleNodes,
- final BlockStoragePolicy storagePolicy,
- final EnumSet<StorageType> unavailableStorages,
- final boolean newBlock) {
- ...
- if (numOfResults <= 1) {
- chooseRemoteRack(1, dn0, excludedNodes, blocksize, maxNodesPerRack,
- results, avoidStaleNodes, storageTypes);
- if (--numOfReplicas == 0) {
- return writer;
- }
- }
注释:此处会尝试在远程机架(即与已有的datanode不同的机架)获取一个新的datanode
- protected void chooseRemoteRack(int numOfReplicas,
- DatanodeDescriptor localMachine,
- Set<Node> excludedNodes,
- long blocksize,
- int maxReplicasPerRack,
- List<DatanodeStorageInfo> results,
- boolean avoidStaleNodes,
- EnumMap<StorageType, Integer> storageTypes)
- throws NotEnoughReplicasException {
- ...
- chooseRandom(numOfReplicas, "~" + localMachine.getNetworkLocation(),
- excludedNodes, blocksize, maxReplicasPerRack, results,
- avoidStaleNodes, storageTypes);
注释:此处会在所有可选的datanode中随机选择一个
- protected DatanodeStorageInfo chooseRandom(int numOfReplicas,
- String scope,
- Set<Node> excludedNodes,
- long blocksize,
- int maxNodesPerRack,
- List<DatanodeStorageInfo> results,
- boolean avoidStaleNodes,
- EnumMap<StorageType, Integer> storageTypes)
- throws NotEnoughReplicasException {
- ...
- int numOfAvailableNodes = clusterMap.countNumOfAvailableNodes(
- scope, excludedNodes);
- ...
- if (numOfReplicas>0) {
- String detail = enableDebugLogging;
- if (LOG.isDebugEnabled()) {
- if (badTarget && builder != null) {
- detail = builder.toString();
- builder.setLength(0);
- } else {
- detail = "";
- }
- }
- throw new NotEnoughReplicasException(detail);
- }
注释:如果由于一些原因(比如节点磁盘满或者下线),导致numOfAvailableNodes计算结果为0,会抛出NotEnoughReplicasException
其中maxNodesPerRack计算逻辑如下:
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault
- /**
- * Calculate the maximum number of replicas to allocate per rack. It also
- * limits the total number of replicas to the total number of nodes in the
- * cluster. Caller should adjust the replica count to the return value.
- *
- * @param numOfChosen The number of already chosen nodes.
- * @param numOfReplicas The number of additional nodes to allocate.
- * @return integer array. Index 0: The number of nodes allowed to allocate
- * in addition to already chosen nodes.
- * Index 1: The maximum allowed number of nodes per rack. This
- * is independent of the number of chosen nodes, as it is calculated
- * using the target number of replicas.
- */
- private int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
- int clusterSize = clusterMap.getNumOfLeaves();
- int totalNumOfReplicas = numOfChosen + numOfReplicas;
- if (totalNumOfReplicas > clusterSize) {
- numOfReplicas -= (totalNumOfReplicas-clusterSize);
- totalNumOfReplicas = clusterSize;
- }
- // No calculation needed when there is only one rack or picking one node.
- int numOfRacks = clusterMap.getNumOfRacks();
- if (numOfRacks == 1 || totalNumOfReplicas <= 1) {
- return new int[] {numOfReplicas, totalNumOfReplicas};
- }
- int maxNodesPerRack = (totalNumOfReplicas-1)/numOfRacks + 2;
- // At this point, there are more than one racks and more than one replicas
- // to store. Avoid all replicas being in the same rack.
- //
- // maxNodesPerRack has the following properties at this stage.
- // 1) maxNodesPerRack >= 2
- // 2) (maxNodesPerRack-1) * numOfRacks > totalNumOfReplicas
- // when numOfRacks > 1
- //
- // Thus, the following adjustment will still result in a value that forces
- // multi-rack allocation and gives enough number of total nodes.
- if (maxNodesPerRack == totalNumOfReplicas) {
- maxNodesPerRack--;
- }
- return new int[] {numOfReplicas, maxNodesPerRack};
- }
注释:
int maxNodesPerRack = (totalNumOfReplicas-1)/numOfRacks + 2;
if (maxNodesPerRack == totalNumOfReplicas) {
maxNodesPerRack--;
}
【原创】大数据基础之HDFS(1)HDFS新创建文件如何分配Datanode的更多相关文章
- 大数据学习(一)-------- HDFS
需要精通java开发,有一定linux基础. 1.简介 大数据就是对海量数据进行数据挖掘. 已经有了很多框架方便使用,常用的有hadoop,storm,spark,flink等,辅助框架hive,ka ...
- 【原创】大数据基础之Zookeeper(2)源代码解析
核心枚举 public enum ServerState { LOOKING, FOLLOWING, LEADING, OBSERVING; } zookeeper服务器状态:刚启动LOOKING,f ...
- 【原创】大数据基础之Kerberos(2)hive impala hdfs访问
1 hive # kadmin.local -q 'ktadd -k /tmp/hive3.keytab -norandkey hive/server03@TEST.COM'# kinit -kt / ...
- 大数据基础总结---HDFS分布式文件系统
HDFS分布式文件系统 文件系统的基本概述 文件系统定义:文件系统是一种存储和组织计算机数据的方法,它使得对其访问和查找变得容易. 文件名:在文件系统中,文件名是用于定位存储位置. 元数据(Metad ...
- 大数据技术之Hadoop(HDFS)
第1章 HDFS概述 1.1 HDFS产出背景及定义 1.2 HDFS优缺点 1.3 HDFS组成架构 1.4 HDFS文件块大小(面试重点) 第2章 HDFS的Shell操作(开发重点) 1.基本语 ...
- 大数据学习(02)——HDFS入门
Hadoop模块 提到大数据,Hadoop是一个绕不开的话题,我们来看看Hadoop本身包含哪些模块. Common是基础模块,这个是必须用的.剩下常用的就是HDFS和YARN. MapReduce现 ...
- 【原创】大数据基础之Impala(1)简介、安装、使用
impala2.12 官方:http://impala.apache.org/ 一 简介 Apache Impala is the open source, native analytic datab ...
- 大数据学习之旅1——HDFS版本演化
最近开始学习大数据,发现大数据有很多很多组件,我现在负责的是HDFS(Hadoop分布式储存系统)的学习,整理了一下HDFS的版本情况.因为HDFS是Hadoop的重要组成部分,所以有关HDFS的版本 ...
- 大数据之路week07--day01(HDFS学习,Java代码操作HDFS,将HDFS文件内容存入到Mysql)
一.HDFS概述 数据量越来越多,在一个操作系统管辖的范围存不下了,那么就分配到更多的操作系统管理的磁盘中,但是不方便管理和维护,因此迫切需要一种系统来管理多台机器上的文件,这就是分布式文件管理系统 ...
随机推荐
- 在Web界面中实现Excel数据大量导入的处理方式
在早期Bootstrap框架介绍中,我的随笔<结合bootstrap fileinput插件和Bootstrap-table表格插件,实现文件上传.预览.提交的导入Excel数据操作流程> ...
- TensorFlow基础
TensorFlow基础 SkySeraph 2017 Email:skyseraph00#163.com 更多精彩请直接访问SkySeraph个人站点:www.skyseraph.com Over ...
- (转)JMeter学习逻辑控制器
JMeter中的Logic Controller用于为Test Plan中的节点添加逻辑控制器. JMeter中的Logic Controller分为两类:一类用来控制Test Plan执行过程中节点 ...
- Oracle物化视图的创建及使用
oracle物化视图 一.oracle物化视图基本概念 物化视图首先需要创建物化视图日志, oracle依据用户创建的物化视图日志来创建物化视图日志表, 物化视图日志表的名称为mlog$_后面跟 ...
- Listen 指令
L:44
- 基于scrapy-redis的分布式爬虫
一.介绍 1.原生的scrapy框架 原生的scrapy框架是实现不了分布式的,其原因有: 1. 因为多台机器上部署的scrapy会各自拥有各自的调度器,这样就使得多台机器无法分配start_urls ...
- Git submodule - 子模块【转】
子模块 有种情况我们经常会遇到:某个工作中的项目需要包含并使用另一个项目. 也许是第三方库,或者你独立开发的,用于多个父项目的库. 现在问题来了:你想要把它们当做两个独立的项目,同时又想在一个项目中使 ...
- [JSOI2008]Blue Mary开公司[李超线段树]
题面 bzoj luogu 好久以前听lxl讲过 咕掉了.. 竟然又遇到了 安利blog #include <cmath> #include <cstring> #includ ...
- Linux saltstack常用模块
所有模块 salt '172.30.100.126' sys.list_modules #列出当前版本支持的模块 salt '*' sys.doc cp #显示指定模块的文档 archive模块 实现 ...
- 联想的笔记本有隐藏分区 导致无法安装win10 eufi启动 报错:windows无法更新计算机的启动配置。无法安装
联想的笔记本都带着类似一键还原等的系统恢复软件,这些软件往往是将出厂设置备份在单 独的一个分区,此分区默认为隐藏,在 Windows 的磁盘管理中可以看到.打开磁盘管理器 的方法是右击计算机——管理, ...