当前集群

主机名称	IP地址	角色	统一安装目录	统一安装用户
sht-sgmhadoopnn-01	172.16.101.55	namenode,resourcemanager	/usr/local/hadoop(软连接) /usr/local/hadoop-2.7.4 /usr/local/zookeeper（软连接） /usr/local/zookeeper-3.4.9	root
sht-sgmhadoopnn-02	172.16.101.56	namenode,resourcemanager
sht-sgmhadoopdn-01	172.16.101.58	datanode,nodemanager,journalnode,zookeeper
sht-sgmhadoopdn-02	172.16.101.59	datanode,nodemanager,journalnode,zookeeper
sht-sgmhadoopdn-03	172.16.101.60	datanode,nodemanager,journalnode,zookeeper
sht-sgmhadoopdn-04	172.16.101.66	datanode,nodemanager

因sht-sgmhadoopdn-04节点异常，需要从当前集群剔除

1. 在active节点sht-sgmhadoopnn-01修改配置文件

slaves

sht-sgmhadoopdn-

sht-sgmhadoopdn-

sht-sgmhadoopdn-

hdfs-site.xml

  <property>

        <name>dfs.replication</name>

        <value>3</value>

  </property>

hdfs-site.xml

  <property>

        <name>dfs.hosts</name>

        <value>/usr/local/hadoop/etc/hadoop/hdfs_includes</value>

  </property>

  <property>

        <name>dfs.hosts.exclude</name>

        <value>/usr/local/hadoop/etc/hadoop/hdfs_excludes</value>

  </property>

yarn-site.xml

  <property>

        <name>dfs.hosts</name>

        <value>/usr/local/hadoop/etc/hadoop/yarn_includes</value>

  </property>

  <property>

        <name>dfs.hosts.exclude</name>

        <value>/usr/local/hadoop/etc/hadoop/yarn_excludes</value>

  </property>

如果存在上述hosts配置，则可以直接进入第二步，否则，需要将上述参数添加至对应文件里，并重启hdfs和yarn集群

2. 添加白名单和黑名单

白名单

# echo sht-sgmhadoopdn- > /usr/local/hadoop/etc/hadoop/hdfs_includes

# echo sht-sgmhadoopdn- > /usr/local/hadoop/etc/hadoop/hdfs_includes

# echo sht-sgmhadoopdn- > /usr/local/hadoop/etc/hadoop/hdfs_includes

黑名单

# echo sht-sgmhadoopdn- > /usr/local/hadoop/etc/hadoop/hdfs_excludes

# echo sht-sgmhadoopdn- > /usr/local/hadoop/etc/hadoop/yarn_excludes

在所有节点更新上述参数

3. 在active节点sht-sgmhadoopnn-01刷新节点信息

此命令告诉resourcemanager重新读取配置文件yarn-site.xml，将存在yarn_excludes文件中的节点退出当前yarn集群，并不在分配job到该节点

# yarn rmadmin -refreshNodes

输出log

2019-03-30 00:40:48,751 INFO org.apache.hadoop.conf.Configuration: found resource yarn-site.xml at file:/usr/local/hadoop-2.7.4/etc/hadoop/yarn-site.xml

2019-03-30 00:40:48,755 INFO org.apache.hadoop.util.HostsFileReader: Setting the includes file to /usr/local/hadoop/etc/hadoop/yarn_includes

2019-03-30 00:40:48,755 INFO org.apache.hadoop.util.HostsFileReader: Setting the excludes file to /usr/local/hadoop/etc/hadoop/yarn_excludes

2019-03-30 00:40:48,755 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list

2019-03-30 00:40:48,755 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-01 to the list of included hosts from /usr/local/hadoop/etc/hadoop/yarn_includes

2019-03-30 00:40:48,755 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-02 to the list of included hosts from /usr/local/hadoop/etc/hadoop/yarn_includes

2019-03-30 00:40:48,755 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-03 to the list of included hosts from /usr/local/hadoop/etc/hadoop/yarn_includes

2019-03-30 00:40:48,756 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-04 to the list of excluded hosts from /usr/local/hadoop/etc/hadoop/yarn_excludes

2019-03-30 00:40:48,757 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Deactivating Node sht-sgmhadoopdn-04:21570 as it is now DECOMMISSIONED

2019-03-30 00:40:48,758 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: sht-sgmhadoopdn-04:21570 Node Transitioned from RUNNING to DECOMMISSIONED

2019-03-30 00:40:48,756 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root	IP=172.16.101.55	OPERATION=refreshNodes	TARGET=AdminService	RESULT=SUCCESS

2019-03-30 00:40:48,772 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Removed node sht-sgmhadoopdn-04:21570 clusterResource: <memory:12288, vCores:24>

2019-03-30 00:40:48,909 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Disallowed NodeManager nodeId: sht-sgmhadoopdn-04:21570 hostname: sht-sgmhadoopdn-04

nodemanager输出log

2019-03-30 00:40:48,910 WARN org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Recieved SHUTDOWN signal from Resourcemanager as part of heartbeat, hence shutting down.

2019-03-30 00:40:48,910 WARN org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Message from ResourceManager: Disallowed NodeManager nodeId: sht-sgmhadoopdn-04:21570 hostname: sht-sgmhadoopdn-04

2019-03-30 00:40:49,023 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8042

2019-03-30 00:40:49,125 INFO org.apache.hadoop.ipc.Server: Stopping server on 21570

2019-03-30 00:40:49,133 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder

2019-03-30 00:40:49,133 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 21570

2019-03-30 00:40:49,135 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is interrupted. Exiting.

2019-03-30 00:40:49,154 INFO org.apache.hadoop.ipc.Server: Stopping server on 8040

2019-03-30 00:40:49,159 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8040

2019-03-30 00:40:49,159 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder

2019-03-30 00:40:49,160 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Public cache exiting

2019-03-30 00:40:49,162 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager metrics system...

2019-03-30 00:40:49,163 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system stopped.

2019-03-30 00:40:49,163 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system shutdown complete.

2019-03-30 00:40:49,178 INFO org.apache.hadoop.util.ExitUtil: Exiting with status -1

2019-03-30 00:40:49,182 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NodeManager at sht-sgmhadoopdn-04/172.16.101.66

************************************************************/

此命令告诉hdfs重新读取配置文件hdfs-site.xml，将存在hdfs_excludes文件中的节点退出当前hdfs集群。

# hdfs dfsadmin -refreshNodes

输出log

2019-03-30 00:43:00,576 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-01 to the list of included hosts from /usr/local/hadoop/etc/hadoop/hdfs_includes

2019-03-30 00:43:00,576 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-02 to the list of included hosts from /usr/local/hadoop/etc/hadoop/hdfs_includes

2019-03-30 00:43:00,576 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-03 to the list of included hosts from /usr/local/hadoop/etc/hadoop/hdfs_includes

2019-03-30 00:43:00,577 INFO org.apache.hadoop.util.HostsFileReader: Adding sht-sgmhadoopdn-04 to the list of excluded hosts from /usr/local/hadoop/etc/hadoop/hdfs_excludes

2019-03-30 00:43:01,266 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020, call org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.sendHeartbeat from 172.16.101.66:16197 Call#1641 Retry#0

org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode because the host is not in the include-list: 172.16.101.66:50010

	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.handleHeartbeat(DatanodeManager.java:1340)

	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:4512)

	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.sendHeartbeat(NameNodeRpcServer.java:1298)

	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.sendHeartbeat(DatanodeProtocolServerSideTranslatorPB.java:114)

	at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:28754)

	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)

	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:422)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)

	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)

2019-03-30 00:43:28,046 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* removeDeadDatanode: lost heartbeat from 172.16.101.66:50010

2019-03-30 00:43:28,051 INFO org.apache.hadoop.net.NetworkTopology: Removing a node: /default-rack/172.16.101.66:50010

datanode输出log

2019-03-30 00:43:01,270 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-698223843-172.16.101.55-1553701973789 (Datanode Uuid 30bc4855-5dca-412a-bf9a-9529bcfe3ea0) service to sht-sgmhadoopnn-01/172.16.101.55:8020 is shutting down

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied communication with namenode because the host is not in the include-list: 172.16.101.66:50010

	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.handleHeartbeat(DatanodeManager.java:1340)

	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:4512)

	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.sendHeartbeat(NameNodeRpcServer.java:1298)

	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.sendHeartbeat(DatanodeProtocolServerSideTranslatorPB.java:114)

	at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:28754)

	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)

	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:422)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)

	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)

	at org.apache.hadoop.ipc.Client.call(Client.java:1476)

	at org.apache.hadoop.ipc.Client.call(Client.java:1413)

	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)

	at com.sun.proxy.$Proxy15.sendHeartbeat(Unknown Source)

	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:152)

	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:402)

	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:500)

	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659)

	at java.lang.Thread.run(Thread.java:745)

2019-03-30 00:43:01,272 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-698223843-172.16.101.55-1553701973789 (Datanode Uuid 30bc4855-5dca-412a-bf9a-9529bcfe3ea0) service to sht-sgmhadoopnn-01/172.16.101.55:8020

4. 执行hdfs集群负载均衡

# hdfs dfs -setrep -R -w   /

输出log

# hdfs dfs -setrep -R -w 3  /

Replication 3 set: /CentOS-6.8-x86_64-bin-DVD2.iso

Replication 3 set: /hadoop-2.8.1.tar.gz

Replication 3 set: /slaves

Replication 3 set: /yarn-env.sh

Waiting for /CentOS-6.8-x86_64-bin-DVD2.iso ...

WARNING: the waiting time may be long for DECREASING the number of replications.

............................................................................................... done

Waiting for /hadoop-2.8.1.tar.gz ... done

Waiting for /slaves ... done

Waiting for /yarn-env.sh ... done

5.停止sht-sgmhadoopdn-04节点的datanode和nodemanager角色

6. 清空各节点黑白名单

# echo > /usr/local/hadoop/etc/hadoop/hdfs_excludes

# echo > /usr/local/hadoop/etc/hadoop/yarn_excludes

# echo > /usr/local/hadoop/etc/hadoop/hdfs_includes

# echo > /usr/local/hadoop/etc/hadoop/yarn_includes

7. 在适当时候建议重启hdfs和yarn集群

Hadoop 2.7.4 HDFS+YRAN HA删除datanode和nodemanager的更多相关文章

Hadoop 2.7.4 HDFS+YRAN HA增加datanode和nodemanager
当前集群主机名称 IP地址角色统一安装目录统一安装用户 sht-sgmhadoopnn-01 172.16.101.55 namenode,resourcemanager /usr/local ...
Hadoop 2.7.4 HDFS+YRAN HA部署
实验环境主机名称 IP地址角色统一安装目录统一安装用户 sht-sgmhadoopnn-01 172.16.101.55 namenode,resourcemanager /usr/local ...
【Hadoop学习之四】HDFS HA搭建（QJM）
环境虚拟机:VMware 10 Linux版本:CentOS-6.5-x86_64 客户端:Xshell4 FTP:Xftp4 jdk8 hadoop-3.1.1 由于NameNode对于整个HDF ...
第九章搭建Hadoop 2.2.0版本HDFS的HA配置
Hadoop中的NameNode好比是人的心脏,非常重要,绝对不可以停止工作.在hadoop1时代,只有一个NameNode.如果该NameNode数据丢失或者不能工作,那么整个集群就不能恢复了.这是 ...
hadoop(十)hdfs上传删除文件（完全分布式七)|12
集群测试上传小文件到集群,随便选择一个小文件上传到hdfs的根目录 [shaozhiqi@hadoop102 hadoop-3.1.2]$ bin/hdfs dfs -put wcinput/wc. ...
hibench 对CDH5.13.1进行基准测试（测试项目hadoop\spark\)HDFS作HA高可靠性
使用CDH 5.13.1部署了HADOOP集群之后,需要进行基准性能测试. 一.hibench 安装 1.安装位置要求. 因为是全量安装,其中有SPARK的测试(SPARK2.0). 安装位置在SPA ...
Hadoop（四）HDFS集群详解
前言前面几篇简单介绍了什么是大数据和Hadoop,也说了怎么搭建最简单的伪分布式和全分布式的hadoop集群.接下来这篇我详细的分享一下HDFS. HDFS前言: 设计思想:(分而治之)将大文件.大 ...
使用QJM实现HDFS的HA配置
使用QJM实现HDFS的HA配置 1.背景 hadoop 2.0.0之前,namenode存在单点故障问题(SPOF,single point of failure),如果主机或进程不可用时,整个集群 ...
HDFS的HA（高可用）
HDFS的HA(高可用) 概述 (1)实现高可用最关键的策略是[消除单点故障].HA 严格来说应该分成各个组件的 HA 机制:HDFS 的 HA 和 YARN 的 HA. (2)Hadoop2.0 之 ...

随机推荐

zw版足彩大数据&报价
zw版足彩大数据&报价 ::zw增强版足彩大数据,文件名后缀是'.dat' ::文件格式是标准文本格式,逗号分隔 ::zw增强版,在标准版赔率基础上,增加了倒数.比率两组归一化数据 ::zw版 ...
Spring Boot默认的JSON解析框架设置
方案一:启动类继承WebMvcConfigurerAdapter,覆盖方法configureMessageConverters ... @SpringBootApplication public cl ...
GridView用法
首先,gridview是封装好的,直接在设计界面使用,基本不需要写代码: 1.绑定数据源 GridView最好与LinQDatasourse配合使用,相匹配绑定数据: 2.外观控制整体控制自动选择 ...
php 版本升高后会出现之Deprecated: Function ereg_replace() is deprecated的解决方法
这个问题是因为php版本过高. 在php5.3中,正则函数ereg_replace已经废弃,而dedecms还继续用.有两个方案可以解决以上问题: 1.把php版本换到v5.3下. 2.继续使用v5. ...
操作mongodb
MongoDB数据库是以k-v形式存储在磁盘上的. import pymongoclient = pymongo.MongoClient(host='10.29.3.40',port=27017)db ...
python list用法
a = ['张三','李四','赵六','王五'] #打印列表 print(a) #增 a.apppend('徐七') print(a) a.insert(1,'张六') print(a) #删 a. ...
pandas替换一列中的汉字为数字
表格的一列“总金额”应该全部为数字,但其中少数项出现汉字,应该将汉字替换为数字,才能进行后面的计算. 先定义一个函数: def is_number(s): try: float(s) return T ...
FL Studio中的音频设置
在FL Studio中,有一步很关键的设置需要我们详细熟悉了解,它就是音频设置,什么是音频设置呢?它就是需要我们选择音频设备驱动程序并优化设置.在了解音频设备之前,我们先来看看什么是音频设备. 我们的 ...
Magento 2.2.5和2.2.6的bug 产品设置special price又删除后价格排序有误
Magento 2.2.5和2.2.6的bug 产品设置special price又删除后价格排序有误一.问题描述:版本2.2.5和2.2.6均有此问题,为Magento2的系统bug.为产品设置s ...
shell编程(五)之函数
function:函数函数只有被调用才会执行如何调用:给定函数名函数名出现的地方,会被自动替换为函数代码函数的生命周期:被调用时创建,返回时终止return命令返回自定义状态结果 0:成功 1- ...

Hadoop 2.7.4 HDFS+YRAN HA删除datanode和nodemanager

当前集群

因sht-sgmhadoopdn-04节点异常，需要从当前集群剔除

1. 在active节点sht-sgmhadoopnn-01修改配置文件

2. 添加白名单和黑名单

3. 在active节点sht-sgmhadoopnn-01刷新节点信息

4. 执行hdfs集群负载均衡

5.停止sht-sgmhadoopdn-04节点的datanode和nodemanager角色

6. 清空各节点黑白名单

7. 在适当时候建议重启hdfs和yarn集群

Hadoop 2.7.4 HDFS+YRAN HA删除datanode和nodemanager的更多相关文章

随机推荐

热门专题