当前集群

主机名称 IP地址 角色 统一安装目录 统一安装用户
sht-sgmhadoopnn-01 172.16.101.55 namenode,resourcemanager

/usr/local/hadoop(软连接)

/usr/local/hadoop-2.7.4

/usr/local/zookeeper（软连接）

/usr/local/zookeeper-3.4.9

root
sht-sgmhadoopnn-02 172.16.101.56 namenode,resourcemanager
sht-sgmhadoopdn-01 172.16.101.58 datanode,nodemanager,journalnode,zookeeper
sht-sgmhadoopdn-02 172.16.101.59 datanode,nodemanager,journalnode,zookeeper
sht-sgmhadoopdn-03 172.16.101.60 datanode,nodemanager,journalnode,zookeeper

集群部署完成后增加datanode sht-sgmhadoopdn-04

部署参考 https://www.cnblogs.com/ilifeilong/p/10610993.html

1. 新datanode节点按照全新安装方式配置ssh无密码登录、系统变量、主机名解析、等

2.在namenode active节点sht-sgmhadoopnn-01修改配置文件

1）slaves

添加主机名sht-sgmhadoopdn-04至slaves文件

2）hdfs-site.xml

将dfs.replication参数值修改为4

3. 在namenode active节点sht-sgmhadoopnn-01将以上两个新修改的文件rsync到集群其他节点

# rsync -az --progress hdfs-site.xml root@172.16.101.56:/usr/local/hadoop/etc/hadoop/

# rsync -az --progress hdfs-site.xml root@172.16.101.58:/usr/local/hadoop/etc/hadoop/

# rsync -az --progress hdfs-site.xml root@172.16.101.59:/usr/local/hadoop/etc/hadoop/

# rsync -az --progress hdfs-site.xml root@172.16.101.60:/usr/local/hadoop/etc/hadoop/

# rsync -az --progress hdfs-site.xml root@172.16.101.66:/usr/local/hadoop/etc/hadoop/

# rsync -az --progress slaves root@172.16.101.56:/usr/local/hadoop/etc/hadoop/

# rsync -az --progress slaves root@172.16.101.58:/usr/local/hadoop/etc/hadoop/

# rsync -az --progress slaves root@172.16.101.59:/usr/local/hadoop/etc/hadoop/

# rsync -az --progress slaves root@172.16.101.60:/usr/local/hadoop/etc/hadoop/

# rsync -az --progress slaves root@172.16.101.66:/usr/local/hadoop/etc/hadoop/

4.  在namenode active节点sht-sgmhadoopnn-01将hadoop目录同步到新节点

# rsync -az --progress --exclude=data --exclude=logs  /usr/local/hadoop-2.7. root@sht-sgmhadoopdn-:/usr/local/

5. 在新节点上启动datanode和nodemanager角色

# hadoop-daemon.sh start datanode

# yarn-daemon.sh start nodemanager

6. 在namenode和resourcemanager 的active节点或standby节点的WEB界面验证

http://172.16.101.55:50070/dfshealth.html#tab-datanode

http://172.16.101.55:8088/cluster/nodes

7.重新均衡集群datanode数据（建议在standby namenode节点操作）

# hdfs balancer -threshold

输出log

# hdfs balancer -threshold 1

19/03/29 23:59:21 INFO balancer.Balancer: Using a threshold of 1.0

19/03/29 23:59:21 INFO balancer.Balancer: namenodes  = [hdfs://mycluster]

19/03/29 23:59:21 INFO balancer.Balancer: parameters = Balancer.Parameters [BalancingPolicy.Node, threshold = 1.0, max idle iteration = 5, number of nodes to be excluded = 0, number of nodes to be included = 0, run during upgrade = false]

Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved

19/03/29 23:59:24 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)

19/03/29 23:59:24 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)

19/03/29 23:59:24 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)

19/03/29 23:59:24 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 5 (default=5)

19/03/29 23:59:24 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)

19/03/29 23:59:24 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)

19/03/29 23:59:24 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)

19/03/29 23:59:24 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.66:50010

19/03/29 23:59:24 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.58:50010

19/03/29 23:59:24 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.60:50010

19/03/29 23:59:24 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.59:50010

19/03/29 23:59:24 INFO balancer.Balancer: 0 over-utilized: []

19/03/29 23:59:24 INFO balancer.Balancer: 1 underutilized: [172.16.101.66:50010:DISK]

19/03/29 23:59:24 INFO balancer.Balancer: Need to move 1.10 GB to make the cluster balanced.

19/03/29 23:59:24 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => underUtilized

19/03/29 23:59:24 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => belowAvgUtilized

19/03/29 23:59:24 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: underUtilized => aboveAvgUtilized

19/03/29 23:59:24 INFO balancer.Balancer: Decided to move 635.63 MB bytes from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK

19/03/29 23:59:24 INFO balancer.Balancer: Decided to move 147.43 MB bytes from 172.16.101.60:50010:DISK to 172.16.101.66:50010:DISK

19/03/29 23:59:24 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => underUtilized

19/03/29 23:59:24 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => belowAvgUtilized

19/03/29 23:59:24 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: underUtilized => aboveAvgUtilized

19/03/29 23:59:24 INFO balancer.Balancer: Will move 783.06 MB in this iteration

19/03/29 23:59:24 INFO balancer.Dispatcher: Limiting threads per target to the specified max.

19/03/29 23:59:24 INFO balancer.Dispatcher: Allocating 5 threads per target.

19/03/29 23:59:24 INFO balancer.Dispatcher: Start moving blk_1073741839_1015 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/29 23:59:24 INFO balancer.Dispatcher: Start moving blk_1073741846_1022 with size=134217728 from 172.16.101.60:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/29 23:59:24 INFO balancer.Dispatcher: Start moving blk_1073741838_1014 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/29 23:59:24 INFO balancer.Dispatcher: Start moving blk_1073741845_1021 with size=134217728 from 172.16.101.60:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/29 23:59:24 INFO balancer.Dispatcher: Start moving blk_1073741837_1013 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/29 23:59:52 INFO balancer.Dispatcher: Successfully moved blk_1073741838_1014 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/29 23:59:52 INFO balancer.Dispatcher: Start moving blk_1073741836_1012 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:00:14 INFO balancer.Dispatcher: Successfully moved blk_1073741836_1012 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:00:14 INFO balancer.Dispatcher: Start moving blk_1073741835_1011 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:00:38 INFO balancer.Dispatcher: Successfully moved blk_1073741835_1011 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:01:44 WARN balancer.Dispatcher: Failed to move blk_1073741837_1013 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741837_1013 received exception java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.101.66:22240 remote=/172.16.101.58:50010], block move is failed

19/03/30 00:01:44 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:01:44 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

19/03/30 00:02:07 WARN balancer.Dispatcher: Failed to move blk_1073741845_1021 with size=134217728 from 172.16.101.60:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741845_1021 received exception java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.101.66:22238 remote=/172.16.101.58:50010], block move is failed

19/03/30 00:02:07 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:02:07 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

19/03/30 00:02:11 WARN balancer.Dispatcher: Failed to move blk_1073741839_1015 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741839_1015 received exception java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.101.66:22232 remote=/172.16.101.58:50010], block move is failed

19/03/30 00:02:11 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:02:11 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

19/03/30 00:02:35 WARN balancer.Dispatcher: Failed to move blk_1073741846_1022 with size=134217728 from 172.16.101.60:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741846_1022 received exception java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.101.66:22234 remote=/172.16.101.58:50010], block move is failed

19/03/30 00:02:35 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:02:35 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

Mar 30, 2019 12:02:36 AM          0               384 MB             1.10 GB          783.06 MB

19/03/30 00:02:41 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)

19/03/30 00:02:41 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)

19/03/30 00:02:41 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)

19/03/30 00:02:41 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 5 (default=5)

19/03/30 00:02:41 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)

19/03/30 00:02:41 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)

19/03/30 00:02:41 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)

19/03/30 00:02:41 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.66:50010

19/03/30 00:02:41 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.59:50010

19/03/30 00:02:41 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.58:50010

19/03/30 00:02:41 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.60:50010

19/03/30 00:02:41 INFO balancer.Balancer: 0 over-utilized: []

19/03/30 00:02:41 INFO balancer.Balancer: 1 underutilized: [172.16.101.66:50010:DISK]

19/03/30 00:02:41 INFO balancer.Balancer: Need to move 833.58 MB to make the cluster balanced.

19/03/30 00:02:41 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => underUtilized

19/03/30 00:02:41 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => belowAvgUtilized

19/03/30 00:02:41 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: underUtilized => aboveAvgUtilized

19/03/30 00:02:41 INFO balancer.Balancer: Decided to move 538.88 MB bytes from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK

19/03/30 00:02:41 INFO balancer.Balancer: Decided to move 244.18 MB bytes from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK

19/03/30 00:02:41 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => underUtilized

19/03/30 00:02:41 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => belowAvgUtilized

19/03/30 00:02:41 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: underUtilized => aboveAvgUtilized

19/03/30 00:02:41 INFO balancer.Balancer: Will move 783.06 MB in this iteration

19/03/30 00:02:41 INFO balancer.Dispatcher: Limiting threads per target to the specified max.

19/03/30 00:02:41 INFO balancer.Dispatcher: Allocating 5 threads per target.

19/03/30 00:02:41 INFO balancer.Dispatcher: Start moving blk_1073741837_1013 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:02:41 INFO balancer.Dispatcher: Start moving blk_1073741834_1010 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:02:41 INFO balancer.Dispatcher: Start moving blk_1073741842_1018 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:02:41 INFO balancer.Dispatcher: Start moving blk_1073741841_1017 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:02:41 INFO balancer.Dispatcher: Start moving blk_1073741840_1016 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:02:41 WARN balancer.Dispatcher: Failed to move blk_1073741834_1010 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741834_1010 received exception java.io.IOException: Got error, status message Not able to copy block 1073741834 to /172.16.101.66:22256 because threads quota is exceeded., copy block BP-698223843-172.16.101.55-1553701973789:blk_1073741834_1010 from /172.16.101.58:50010, block move is failed

19/03/30 00:02:41 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:02:41 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

19/03/30 00:02:41 INFO balancer.Dispatcher: Start moving blk_1073741839_1015 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:02:41 WARN balancer.Dispatcher: Failed to move blk_1073741841_1017 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741841_1017 received exception java.io.IOException: Got error, status message Not able to copy block 1073741841 to /172.16.101.66:22258 because threads quota is exceeded., copy block BP-698223843-172.16.101.55-1553701973789:blk_1073741841_1017 from /172.16.101.58:50010, block move is failed

19/03/30 00:02:41 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:02:41 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

19/03/30 00:02:41 WARN balancer.Dispatcher: Failed to move blk_1073741840_1016 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741840_1016 received exception java.io.IOException: Got error, status message Not able to copy block 1073741840 to /172.16.101.66:22260 because threads quota is exceeded., copy block BP-698223843-172.16.101.55-1553701973789:blk_1073741840_1016 from /172.16.101.58:50010, block move is failed

19/03/30 00:02:41 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:02:41 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

19/03/30 00:02:41 WARN balancer.Dispatcher: Failed to move blk_1073741839_1015 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741839_1015 received exception java.io.IOException: Got error, status message Not able to copy block 1073741839 to /172.16.101.66:22262 because threads quota is exceeded., copy block BP-698223843-172.16.101.55-1553701973789:blk_1073741839_1015 from /172.16.101.58:50010, block move is failed

19/03/30 00:02:41 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:02:41 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

19/03/30 00:02:58 INFO balancer.Dispatcher: Successfully moved blk_1073741842_1018 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:02:58 INFO balancer.Dispatcher: Successfully moved blk_1073741837_1013 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

Mar 30, 2019 12:02:58 AM          1               640 MB           833.58 MB          783.06 MB

19/03/30 00:03:03 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)

19/03/30 00:03:03 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)

19/03/30 00:03:03 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)

19/03/30 00:03:03 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 5 (default=5)

19/03/30 00:03:03 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)

19/03/30 00:03:03 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)

19/03/30 00:03:03 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)

19/03/30 00:03:03 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.58:50010

19/03/30 00:03:03 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.66:50010

19/03/30 00:03:03 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.59:50010

19/03/30 00:03:03 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.60:50010

19/03/30 00:03:03 INFO balancer.Balancer: 0 over-utilized: []

19/03/30 00:03:03 INFO balancer.Balancer: 1 underutilized: [172.16.101.66:50010:DISK]

19/03/30 00:03:03 INFO balancer.Balancer: Need to move 640.08 MB to make the cluster balanced.

19/03/30 00:03:03 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => underUtilized

19/03/30 00:03:03 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => belowAvgUtilized

19/03/30 00:03:03 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: underUtilized => aboveAvgUtilized

19/03/30 00:03:03 INFO balancer.Balancer: Decided to move 474.38 MB bytes from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK

19/03/30 00:03:03 INFO balancer.Balancer: Decided to move 308.67 MB bytes from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK

19/03/30 00:03:03 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => underUtilized

19/03/30 00:03:03 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => belowAvgUtilized

19/03/30 00:03:03 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: underUtilized => aboveAvgUtilized

19/03/30 00:03:03 INFO balancer.Balancer: Will move 783.06 MB in this iteration

19/03/30 00:03:03 INFO balancer.Dispatcher: Limiting threads per target to the specified max.

19/03/30 00:03:03 INFO balancer.Dispatcher: Allocating 5 threads per target.

19/03/30 00:03:03 INFO balancer.Dispatcher: Start moving blk_1073741834_1010 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:03:03 INFO balancer.Dispatcher: Start moving blk_1073741833_1009 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:03:03 INFO balancer.Dispatcher: Start moving blk_1073741832_1008 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:03:03 INFO balancer.Dispatcher: Start moving blk_1073741828_1004 with size=21901927 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:03:03 INFO balancer.Dispatcher: Start moving blk_1073741827_1003 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:03:03 WARN balancer.Dispatcher: Failed to move blk_1073741828_1004 with size=21901927 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741828_1004 received exception java.io.IOException: Got error, status message Not able to copy block 1073741828 to /172.16.101.66:22272 because threads quota is exceeded., copy block BP-698223843-172.16.101.55-1553701973789:blk_1073741828_1004 from /172.16.101.58:50010, block move is failed

19/03/30 00:03:03 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:03:03 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

19/03/30 00:03:03 INFO balancer.Dispatcher: Start moving blk_1073741826_1002 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:03:03 WARN balancer.Dispatcher: Failed to move blk_1073741826_1002 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741826_1002 received exception java.io.IOException: Got error, status message Not able to copy block 1073741826 to /172.16.101.66:22274 because threads quota is exceeded., copy block BP-698223843-172.16.101.55-1553701973789:blk_1073741826_1002 from /172.16.101.58:50010, block move is failed

19/03/30 00:03:03 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:03:03 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

19/03/30 00:03:47 INFO balancer.Dispatcher: Successfully moved blk_1073741833_1009 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:05:12 WARN balancer.Dispatcher: Failed to move blk_1073741834_1010 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741834_1010 received exception java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.101.66:22266 remote=/172.16.101.58:50010], block move is failed

19/03/30 00:05:12 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:05:12 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

19/03/30 00:05:36 WARN balancer.Dispatcher: Failed to move blk_1073741827_1003 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741827_1003 received exception java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.101.66:22270 remote=/172.16.101.58:50010], block move is failed

19/03/30 00:05:36 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:05:36 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

19/03/30 00:06:11 WARN balancer.Dispatcher: Failed to move blk_1073741832_1008 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741832_1008 received exception java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.101.66:22268 remote=/172.16.101.58:50010], block move is failed

19/03/30 00:06:11 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:06:11 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

Mar 30, 2019 12:06:11 AM          2               768 MB           640.08 MB          783.06 MB

19/03/30 00:06:16 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)

19/03/30 00:06:16 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)

19/03/30 00:06:16 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)

19/03/30 00:06:16 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 5 (default=5)

19/03/30 00:06:16 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)

19/03/30 00:06:16 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)

19/03/30 00:06:16 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)

19/03/30 00:06:16 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.59:50010

19/03/30 00:06:16 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.58:50010

19/03/30 00:06:16 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.66:50010

19/03/30 00:06:16 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.60:50010

19/03/30 00:06:16 INFO balancer.Balancer: 0 over-utilized: []

19/03/30 00:06:16 INFO balancer.Balancer: 1 underutilized: [172.16.101.66:50010:DISK]

19/03/30 00:06:16 INFO balancer.Balancer: Need to move 458.28 MB to make the cluster balanced.

19/03/30 00:06:16 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => underUtilized

19/03/30 00:06:16 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => belowAvgUtilized

19/03/30 00:06:16 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: underUtilized => aboveAvgUtilized

19/03/30 00:06:16 INFO balancer.Balancer: Decided to move 413.78 MB bytes from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK

19/03/30 00:06:16 INFO balancer.Balancer: Decided to move 369.28 MB bytes from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK

19/03/30 00:06:16 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => underUtilized

19/03/30 00:06:16 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => belowAvgUtilized

19/03/30 00:06:16 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: underUtilized => aboveAvgUtilized

19/03/30 00:06:16 INFO balancer.Balancer: Will move 783.06 MB in this iteration

19/03/30 00:06:16 INFO balancer.Dispatcher: Limiting threads per target to the specified max.

19/03/30 00:06:16 INFO balancer.Dispatcher: Allocating 5 threads per target.

19/03/30 00:06:16 INFO balancer.Dispatcher: Start moving blk_1073741832_1008 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:06:16 INFO balancer.Dispatcher: Start moving blk_1073741828_1004 with size=21901927 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:06:16 INFO balancer.Dispatcher: Start moving blk_1073741826_1002 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:06:16 INFO balancer.Dispatcher: Start moving blk_1073741827_1003 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:06:16 INFO balancer.Dispatcher: Start moving blk_1073741834_1010 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:06:16 WARN balancer.Dispatcher: Failed to move blk_1073741834_1010 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741834_1010 received exception java.io.IOException: Got error, status message Not able to copy block 1073741834 to /172.16.101.66:22284 because threads quota is exceeded., copy block BP-698223843-172.16.101.55-1553701973789:blk_1073741834_1010 from /172.16.101.58:50010, block move is failed

19/03/30 00:06:16 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:06:16 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

19/03/30 00:06:16 INFO balancer.Dispatcher: Start moving blk_1073741825_1001 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:06:16 WARN balancer.Dispatcher: Failed to move blk_1073741825_1001 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741825_1001 received exception java.io.IOException: Got error, status message Not able to copy block 1073741825 to /172.16.101.66:22286 because threads quota is exceeded., copy block BP-698223843-172.16.101.55-1553701973789:blk_1073741825_1001 from /172.16.101.58:50010, block move is failed

19/03/30 00:06:16 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:06:16 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

19/03/30 00:06:19 INFO balancer.Dispatcher: Successfully moved blk_1073741828_1004 with size=21901927 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:06:49 INFO balancer.Dispatcher: Successfully moved blk_1073741832_1008 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:06:53 INFO balancer.Dispatcher: Successfully moved blk_1073741827_1003 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:08:36 WARN balancer.Dispatcher: Failed to move blk_1073741826_1002 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741826_1002 received exception java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.101.66:22280 remote=/172.16.101.58:50010], block move is failed

19/03/30 00:08:36 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:08:36 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

Mar 30, 2019 12:08:36 AM          3              1.02 GB           458.28 MB          783.06 MB

19/03/30 00:08:41 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)

19/03/30 00:08:41 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)

19/03/30 00:08:41 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)

19/03/30 00:08:41 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 5 (default=5)

19/03/30 00:08:41 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)

19/03/30 00:08:41 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)

19/03/30 00:08:41 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)

19/03/30 00:08:41 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.59:50010

19/03/30 00:08:41 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.58:50010

19/03/30 00:08:41 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.66:50010

19/03/30 00:08:41 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.60:50010

19/03/30 00:08:41 INFO balancer.Balancer: 0 over-utilized: []

19/03/30 00:08:41 INFO balancer.Balancer: 1 underutilized: [172.16.101.66:50010:DISK]

19/03/30 00:08:41 INFO balancer.Balancer: Need to move 248.99 MB to make the cluster balanced.

19/03/30 00:08:41 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => underUtilized

19/03/30 00:08:41 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => belowAvgUtilized

19/03/30 00:08:41 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: underUtilized => aboveAvgUtilized

19/03/30 00:08:41 INFO balancer.Balancer: Decided to move 344.02 MB bytes from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK

19/03/30 00:08:41 INFO balancer.Balancer: Decided to move 344.02 MB bytes from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK

19/03/30 00:08:41 INFO balancer.Balancer: Decided to move 95.03 MB bytes from 172.16.101.60:50010:DISK to 172.16.101.66:50010:DISK

19/03/30 00:08:41 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => underUtilized

19/03/30 00:08:41 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => belowAvgUtilized

19/03/30 00:08:41 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: underUtilized => aboveAvgUtilized

19/03/30 00:08:41 INFO balancer.Balancer: Will move 783.06 MB in this iteration

19/03/30 00:08:41 INFO balancer.Dispatcher: Limiting threads per target to the specified max.

19/03/30 00:08:41 INFO balancer.Dispatcher: Allocating 5 threads per target.

19/03/30 00:08:41 INFO balancer.Dispatcher: Start moving blk_1073741839_1015 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:08:41 INFO balancer.Dispatcher: Start moving blk_1073741834_1010 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:08:41 INFO balancer.Dispatcher: Start moving blk_1073741825_1001 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:08:41 INFO balancer.Dispatcher: Start moving blk_1073741826_1002 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:08:41 INFO balancer.Dispatcher: Start moving blk_1073741848_1024 with size=73209856 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:09:35 INFO balancer.Dispatcher: Successfully moved blk_1073741848_1024 with size=73209856 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:09:35 INFO balancer.Dispatcher: Start moving blk_1073741847_1023 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:09:40 INFO balancer.Dispatcher: Successfully moved blk_1073741839_1015 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:09:41 INFO balancer.Dispatcher: Successfully moved blk_1073741826_1002 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:09:57 INFO balancer.Dispatcher: Successfully moved blk_1073741825_1001 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:09:57 INFO balancer.Dispatcher: Successfully moved blk_1073741834_1010 with size=134217728 from 172.16.101.59:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010

19/03/30 00:12:28 WARN balancer.Dispatcher: Failed to move blk_1073741847_1023 with size=134217728 from 172.16.101.58:50010:DISK to 172.16.101.66:50010:DISK through 172.16.101.58:50010: Got error, status message opReplaceBlock BP-698223843-172.16.101.55-1553701973789:blk_1073741847_1023 received exception java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.101.66:22298 remote=/172.16.101.58:50010], block move is failed

19/03/30 00:12:28 INFO balancer.Dispatcher: DDatanode:172.16.101.58:50010 activateDelay 10.0 seconds

19/03/30 00:12:28 INFO balancer.Dispatcher: DDatanode:172.16.101.66:50010 activateDelay 10.0 seconds

Mar 30, 2019 12:12:28 AM          4              1.59 GB           248.99 MB          783.06 MB

19/03/30 00:12:33 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)

19/03/30 00:12:33 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)

19/03/30 00:12:33 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)

19/03/30 00:12:33 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 5 (default=5)

19/03/30 00:12:33 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)

19/03/30 00:12:33 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)

19/03/30 00:12:33 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)

19/03/30 00:12:33 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.59:50010

19/03/30 00:12:33 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.60:50010

19/03/30 00:12:33 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.58:50010

19/03/30 00:12:33 INFO net.NetworkTopology: Adding a new node: /default-rack/172.16.101.66:50010

19/03/30 00:12:33 INFO balancer.Balancer: 0 over-utilized: []

19/03/30 00:12:33 INFO balancer.Balancer: 0 underutilized: []

The cluster is balanced. Exiting...

Mar 30, 2019 12:12:33 AM          5              1.59 GB                 0 B               -1 B

Mar 30, 2019 12:12:34 AM Balancing took 13.216533333333333 minutes

再次查看hdfs集群负载

8. 修改hdfs集群中现有文件/目录的副本因子

现有的文件的备份系数仍是原来的值，hadoop并不会自动的按照新的备份系数调整，我们需要手动完成。

hdfs dfs -setrep -R -w  /

输出log

Replication 4 set: /CentOS-6.8-x86_64-bin-DVD2.iso

Replication 4 set: /hadoop-2.8.1.tar.gz

Replication 4 set: /slaves

Waiting for /CentOS-6.8-x86_64-bin-DVD2.iso ..................... done

Waiting for /hadoop-2.8.1.tar.gz ... done

Waiting for /slaves ... done

通过命令查看

# hdfs fsck /

Connecting to namenode via http://sht-sgmhadoopnn-01:50070/fsck?ugi=root&path=%2F

FSCK started by root (auth:SIMPLE) from /172.16.101.55 for path / at Sat Mar  :: CST

...Status: HEALTHY

 Total size:     B

 Total dirs:

 Total files:

 Total symlinks:

 Total blocks (validated):     (avg. block size  B)

 Minimally replicated blocks:     (100.0 %)

 Over-replicated blocks:     (0.0 %)

 Under-replicated blocks:     (0.0 %)

 Mis-replicated blocks:         (0.0 %)

 Default replication factor:

 Average block replication:    4.0

 Corrupt blocks:

 Missing replicas:         (0.0 %)

 Number of data-nodes:

 Number of racks:

FSCK ended at Sat Mar  :: CST  in  milliseconds

The filesystem under path '/' is HEALTHY

以上步骤在不重启hdfs集群下动态添加datanode节点 ，仍然建议在适当时重启hdfs集群。

Hadoop 2.7.4 HDFS+YRAN HA增加datanode和nodemanager的更多相关文章

  1. Hadoop 2.7.4 HDFS+YRAN HA删除datanode和nodemanager

    当前集群 主机名称 IP地址 角色 统一安装目录 统一安装用户 sht-sgmhadoopnn-01 172.16.101.55 namenode,resourcemanager /usr/local ...

  2. Hadoop 2.7.4 HDFS+YRAN HA部署

    实验环境 主机名称 IP地址 角色 统一安装目录 统一安装用户 sht-sgmhadoopnn-01 172.16.101.55 namenode,resourcemanager /usr/local ...

  3. 【Hadoop学习之四】HDFS HA搭建（QJM）

    环境 虚拟机:VMware 10 Linux版本:CentOS-6.5-x86_64 客户端:Xshell4 FTP:Xftp4 jdk8 hadoop-3.1.1 由于NameNode对于整个HDF ...

  4. 第九章 搭建Hadoop 2.2.0版本HDFS的HA配置

    Hadoop中的NameNode好比是人的心脏,非常重要,绝对不可以停止工作.在hadoop1时代,只有一个NameNode.如果该NameNode数据丢失或者不能工作,那么整个集群就不能恢复了.这是 ...

  5. hibench 对CDH5.13.1进行基准测试（测试项目hadoop\spark\)HDFS作HA高可靠性

    使用CDH 5.13.1部署了HADOOP集群之后,需要进行基准性能测试. 一.hibench 安装 1.安装位置要求. 因为是全量安装,其中有SPARK的测试(SPARK2.0). 安装位置在SPA ...

  6. 使用QJM实现HDFS的HA配置

    使用QJM实现HDFS的HA配置 1.背景 hadoop 2.0.0之前,namenode存在单点故障问题(SPOF,single point of failure),如果主机或进程不可用时,整个集群 ...

  7. HDFS的HA（高可用）

    HDFS的HA(高可用) 概述 (1)实现高可用最关键的策略是[消除单点故障].HA 严格来说应该分成各个组件的 HA 机制:HDFS 的 HA 和 YARN 的 HA. (2)Hadoop2.0 之 ...

  8. Hadoop【Hadoop-HA搭建(HDFS、YARN)】

    目录 0.HDFS-HA的工作机制 1. HDFS-HA集群配置 1.1 环境准备 1.2 规划集群 1.3 配置Zookeeper集群 2. 配置HDFS-HA集群 3. 启动HDFS-HA集群 4 ...

  9. Hadoop集群(二) HDFS搭建

    HDFS只是Hadoop最基本的一个服务,很多其他服务,都是基于HDFS展开的.所以部署一个HDFS集群,是很核心的一个动作,也是大数据平台的开始. 安装Hadoop集群,首先需要有Zookeeper ...

随机推荐

  1. Dataguard配置总结

    Dataguard配置总结 本例情形 在主库存在运行的情况下,增加配置dataguard备库,实现双机热备,高可用性. 主库要求,归档模式,强制归档. 主库idty 备库idty_st 1.密码文件 ...

  2. nmon安装与使用

    一.检查安装环境 1,# uname –a (查看操作系统信息,所检查服务器为64位操作系统) Linux jmeter 2.6.32-504.el6.x86_64 #1 SMP Wed Oct 15 ...

  3. laravel composer

    composer config -g repo.packagist composer https://packagist.phpcomposer.com 改安装包的全局镜像网址

  4. JavaScript自定义事件和触发(createEvent, dispatchEvent)

    $(dom).addEvent("ft", function() { alert("走起"); }); // 创建 var evt = document.cre ...

  5. sehll 小脚本的编写{基础}

    1.模拟linnux登录shell #/bin/bash echo -n "login:" read name echo -n "password:" read ...

  6. MacOS修改用户名后变为普通用户，无法创建管理员账号

    摘要:有的时候用户修改原电脑用户名,会把该用户降为普通用户,点击下方的锁会弹出下方图示,导致无法修改任何设置 解决方案: 重启电脑Restart按Command+S进入终端terminal输入以下命令 ...

  7. 【题解】Luogu P3674 小清新人渣的本愿

    原题传送门 这题还算简单(我记得我刚学oi时就来写这题,然后暴力都爆零了) 看见无修改,那么这题应该是莫队 维护两个bitset,第二个是第一个的反串,bitset内维护每个数字是否出现过 第一种操作 ...

  8. 自定义控件之SegmentControlView

    SegmentControlView配合PageView使用 效果图 核心代码 package com.example.segmentcontrolview; import android.conte ...

  9. c++的虚继承

    今天去面试了一家公司,真是套路深啊,套路深,原谅我是后知后觉,所以人吧总的长大,出差正常情况下都是有补贴的,为啥这部分也要算我工资一部分,名其名曰工资高,哈哈哈,自古套路方得人心 今天学习了一下c++ ...

  10. Python day 03

    dya 03 今日内容 整形 布尔类型 字符串 补充 运算符补充 in value = '我是中国人' # 判断'中国'是否是value所代指的字符串的子序列. v1 = '中国' in value ...