【原创】大数据基础之HDFS（1）HDFS新创建文件如何分配Datanode

HDFS中的File由Block组成，一个File包含一个或多个Block，当创建File时会创建一个Block，然后根据配置的副本数量（默认是3）申请3个Datanode来存放这个Block；

通过hdfs fsck命令可以查看一个文件具体的Block、Datanode、Rack信息，例如：

hdfs fsck /tmp/test.sql -files -blocks -locations -racks
Connecting to namenode via http://name_node:50070
FSCK started by hadoop (auth:SIMPLE) from /client for path /tmp/test.sql at Thu Dec 13 15:44:12 CST 2018
/tmp/test.sql 16 bytes, 1 block(s): OK
0. BP-436366437-name_node-1493982655699:blk_1449692331_378721485 len=16 repl=3 [/DEFAULT/server111:50010, /DEFAULT/server121:50010, /DEFAULT/server43:50010]

Status: HEALTHY
Total size: 16 B
Total dirs: 0
Total files: 1
Total symlinks: 0
Total blocks (validated): 1 (avg. block size 16 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 193
Number of racks: 1
FSCK ended at Thu Dec 13 15:44:12 CST 2018 in 1 milliseconds

The filesystem under path '/tmp/test.sql' is HEALTHY

那3个Datanode是如何选择出来的？有一个优先级：

1 当前机架（相对hdfs client而言）

2 远程机架（相对hdfs client而言）

3 另一机架

4 全部随机

然后每个机架能选择几个Datanode（即maxNodesPerRack）有一个计算公式，详见代码

org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer

    private int findNewDatanode(final DatanodeInfo[] original
 
        ) throws IOException {
 
      if (nodes.length != original.length + 1) {
 
        throw new IOException(
 
            new StringBuilder()
 
            .append("Failed to replace a bad datanode on the existing pipeline ")
 
            .append("due to no more good datanodes being available to try. ")
 
            .append("(Nodes: current=").append(Arrays.asList(nodes))
 
            .append(", original=").append(Arrays.asList(original)).append("). ")
 
            .append("The current failed datanode replacement policy is ")
 
            .append(dfsClient.dtpReplaceDatanodeOnFailure).append(", and ")
 
            .append("a client may configure this via '")
 
            .append(DFSConfigKeys.DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_POLICY_KEY)
 
            .append("' in its configuration.")
 
            .toString());
 
      }

注释：当没有找到新的datanode时会报异常，报错如下：

Caused by: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[server82:50010], original=[server.82:50010]).
The current failed datanode replacement policy is ALWAYS, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.

    private void addDatanode2ExistingPipeline() throws IOException {
 
...
 
      final DatanodeInfo[] original = nodes;
 
      final LocatedBlock lb = dfsClient.namenode.getAdditionalDatanode(
 
          src, fileId, block, nodes, storageIDs,
 
          failed.toArray(new DatanodeInfo[failed.size()]),
 
          1, dfsClient.clientName);
 
      setPipeline(lb);
 
      //find the new datanode
 
      final int d = findNewDatanode(original);

注释：会调用getAdditionalDatanode方法来获取1个新的datanode，此处略去很多调用堆栈

org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault

  private DatanodeStorageInfo[] chooseTarget(int numOfReplicas,
 
                                    Node writer,
 
                                    List<DatanodeStorageInfo> chosenStorage,
 
                                    boolean returnChosenNodes,
 
                                    Set<Node> excludedNodes,
 
                                    long blocksize,
 
                                    final BlockStoragePolicy storagePolicy) {
 
...
 
    int[] result = getMaxNodesPerRack(chosenStorage.size(), numOfReplicas);
 
    numOfReplicas = result[0];
 
    int maxNodesPerRack = result[1];
 
...
 
    final Node localNode = chooseTarget(numOfReplicas, writer, excludedNodes,
 
        blocksize, maxNodesPerRack, results, avoidStaleNodes, storagePolicy,
 
        EnumSet.noneOf(StorageType.class), results.isEmpty());

注释：此处maxNodesPerRack表示每个机架最多只能分配几个datanode

  private Node chooseTarget(int numOfReplicas,
 
                            Node writer,
 
                            final Set<Node> excludedNodes,
 
                            final long blocksize,
 
                            final int maxNodesPerRack,
 
                            final List<DatanodeStorageInfo> results,
 
                            final boolean avoidStaleNodes,
 
                            final BlockStoragePolicy storagePolicy,
 
                            final EnumSet<StorageType> unavailableStorages,
 
                            final boolean newBlock) {
 
...
 
      if (numOfResults <= 1) {
 
        chooseRemoteRack(1, dn0, excludedNodes, blocksize, maxNodesPerRack,
 
            results, avoidStaleNodes, storageTypes);
 
        if (--numOfReplicas == 0) {
 
          return writer;
 
        }
 
      }

注释：此处会尝试在远程机架（即与已有的datanode不同的机架）获取一个新的datanode

  protected void chooseRemoteRack(int numOfReplicas,
 
                                DatanodeDescriptor localMachine,
 
                                Set<Node> excludedNodes,
 
                                long blocksize,
 
                                int maxReplicasPerRack,
 
                                List<DatanodeStorageInfo> results,
 
                                boolean avoidStaleNodes,
 
                                EnumMap<StorageType, Integer> storageTypes)
 
                                    throws NotEnoughReplicasException {
 
...
 
      chooseRandom(numOfReplicas, "~" + localMachine.getNetworkLocation(),
 
          excludedNodes, blocksize, maxReplicasPerRack, results,
 
          avoidStaleNodes, storageTypes);

注释：此处会在所有可选的datanode中随机选择一个

  protected DatanodeStorageInfo chooseRandom(int numOfReplicas,
 
                            String scope,
 
                            Set<Node> excludedNodes,
 
                            long blocksize,
 
                            int maxNodesPerRack,
 
                            List<DatanodeStorageInfo> results,
 
                            boolean avoidStaleNodes,
 
                            EnumMap<StorageType, Integer> storageTypes)
 
                            throws NotEnoughReplicasException {
 
...
 
    int numOfAvailableNodes = clusterMap.countNumOfAvailableNodes(
 
        scope, excludedNodes);
 
...
 
    if (numOfReplicas>0) {
 
      String detail = enableDebugLogging;
 
      if (LOG.isDebugEnabled()) {
 
        if (badTarget && builder != null) {
 
          detail = builder.toString();
 
          builder.setLength(0);
 
        } else {
 
          detail = "";
 
        }
 
      }
 
      throw new NotEnoughReplicasException(detail);
 
    }

注释：如果由于一些原因（比如节点磁盘满或者下线），导致numOfAvailableNodes计算结果为0，会抛出NotEnoughReplicasException

其中maxNodesPerRack计算逻辑如下：

org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault

  /**
 
   * Calculate the maximum number of replicas to allocate per rack. It also
 
   * limits the total number of replicas to the total number of nodes in the
 
   * cluster. Caller should adjust the replica count to the return value.
 
   *
 
   * @param numOfChosen The number of already chosen nodes.
 
   * @param numOfReplicas The number of additional nodes to allocate.
 
   * @return integer array. Index 0: The number of nodes allowed to allocate
 
   *         in addition to already chosen nodes.
 
   *         Index 1: The maximum allowed number of nodes per rack. This
 
   *         is independent of the number of chosen nodes, as it is calculated
 
   *         using the target number of replicas.
 
   */
 
  private int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
 
    int clusterSize = clusterMap.getNumOfLeaves();
 
    int totalNumOfReplicas = numOfChosen + numOfReplicas;
 
    if (totalNumOfReplicas > clusterSize) {
 
      numOfReplicas -= (totalNumOfReplicas-clusterSize);
 
      totalNumOfReplicas = clusterSize;
 
    }
 
    // No calculation needed when there is only one rack or picking one node.
 
    int numOfRacks = clusterMap.getNumOfRacks();
 
    if (numOfRacks == 1 || totalNumOfReplicas <= 1) {
 
      return new int[] {numOfReplicas, totalNumOfReplicas};
 
    }
 
    int maxNodesPerRack = (totalNumOfReplicas-1)/numOfRacks + 2;
 
    // At this point, there are more than one racks and more than one replicas
 
    // to store. Avoid all replicas being in the same rack.
 
    //
 
    // maxNodesPerRack has the following properties at this stage.
 
    //   1) maxNodesPerRack >= 2
 
    //   2) (maxNodesPerRack-1) * numOfRacks > totalNumOfReplicas
 
    //          when numOfRacks > 1
 
    //
 
    // Thus, the following adjustment will still result in a value that forces
 
    // multi-rack allocation and gives enough number of total nodes.
 
    if (maxNodesPerRack == totalNumOfReplicas) {
 
      maxNodesPerRack--;
 
    }
 
    return new int[] {numOfReplicas, maxNodesPerRack};
 
  }

注释：

int maxNodesPerRack = (totalNumOfReplicas-1)/numOfRacks + 2;

if (maxNodesPerRack == totalNumOfReplicas) {

maxNodesPerRack--;

}