Hadoop2.7.6_08_Federation联邦机制
前言:
本文章是在 Hadoop2.7.6_07_HA高可用 的基础上完成的,所以不清楚的可参见这篇文章。
1. Hadoop的federation机制
文件的元数据是放在namenode上的,只有一个Namespace(命名空间)。随着HDFS的数据越来越多,单个namenode的资源使用必然会达到上限,而且namenode的负载能力也会越来越高,限制HDFS的性能。
Federation即为“联邦”,该特性允许一个HDFS集群中存在多个NameNode同时对外提供服务,这些NameNode分管一部分目录(水平切分),彼此之间相互隔离,但共享底层的DataNode存储资源。
1.1. HDFS-federation图解
2. 主机规划
主机名称 |
外网IP |
内网IP |
操作系统 |
备注 |
安装软件 |
运行进程 |
mini01 |
10.0.0.111 |
172.16.1.111 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop |
NameNode、DFSZKFailoverController(zkfc) |
mini02 |
10.0.0.112 |
172.16.1.112 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop |
NameNode、DFSZKFailoverController(zkfc) |
mini03 |
10.0.0.113 |
172.16.1.113 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop、zookeeper |
NameNode、ResourceManager |
mini04 |
10.0.0.114 |
172.16.1.114 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop、zookeeper |
NameNode、ResourceManager |
mini05 |
10.0.0.115 |
172.16.1.115 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop、zookeeper |
DataNode、NodeManager、JournalNode、QuorumPeerMain |
mini06 |
10.0.0.116 |
172.16.1.116 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop、zookeeper |
DataNode、NodeManager、JournalNode、QuorumPeerMain |
mini07 |
10.0.0.117 |
172.16.1.117 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop、zookeeper |
DataNode、NodeManager、JournalNode、QuorumPeerMain |
注意:此规划是在HA的基础上进行的federation,即 mini01和mini02做HA,mini03和mini04做HA;之后再做federation。
HA的配置请参见:Hadoop2.7.6_07_HA高可用
Linux添加hosts信息,保证每台都可以相互ping通
[root@mini01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
:: localhost localhost.localdomain localhost6 localhost6.localdomain6 10.0.0.111 mini01
10.0.0.112 mini02
10.0.0.113 mini03
10.0.0.114 mini04
10.0.0.115 mini05
10.0.0.116 mini06
10.0.0.117 mini07
Windows的hosts文件修改
# 文件位置C:\Windows\System32\drivers\etc 在hosts中追加如下内容
…………………………………………
10.0.0.111 mini01
10.0.0.112 mini02
10.0.0.113 mini03
10.0.0.114 mini04
10.0.0.115 mini05
10.0.0.116 mini06
10.0.0.117 mini07
2.1. 注意事项
其中添加yun用户、实现yun用户免秘钥登录、Jdk【java8】、Zookeeper部署;请参见Hadoop2.7.6_07_HA高可用
3. Hadoop部署与配置修改
注意:每台机器的Hadoop以及配置相同
3.1. 部署
[yun@mini01 software]$ pwd
/app/software
[yun@mini01 software]$ ll
total
-rw-r--r-- yun yun Jun : CentOS-.4_hadoop-2.7..tar.gz
[yun@mini01 software]$ tar xf CentOS-.4_hadoop-2.7..tar.gz
[yun@mini01 software]$ mv hadoop-2.7./ /app/
[yun@mini01 software]$ cd
[yun@mini01 ~]$ ln -s hadoop-2.7./ hadoop
[yun@mini01 ~]$ ll
total
lrwxrwxrwx yun yun Jun : hadoop -> hadoop-2.7./
drwxr-xr-x yun yun Jun : hadoop-2.7.
lrwxrwxrwx yun yun May : jdk -> jdk1..0_112
drwxr-xr-x yun yun Sep jdk1..0_112
3.2. 环境变量
[root@mini01 profile.d]# pwd
/etc/profile.d
[root@mini01 profile.d]# vim hadoop.sh
export HADOOP_HOME="/app/hadoop"
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH [root@mini01 profile.d]# source /etc/profile # 生效
3.3. core-site.xml
[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
………………
--> <!-- Put site-specific property overrides in this file. --> <configuration>
<property>
<name>fs.defaultFS</name>
<value>viewfs:///</value>
</property> <property>
<name>fs.viewfs.mounttable.default.link./bi</name>
<value>hdfs://bi/</value>
</property> <property>
<name>fs.viewfs.mounttable.default.link./dt</name>
<value>hdfs://dt/</value>
</property> <!-- 指定hadoop临时目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
</property> <!-- 指定zookeeper地址 -->
<property>
<name>ha.zookeeper.quorum</name>
<value>mini03:,mini04:,mini05:,mini06:,mini07:</value>
</property> </configuration>
3.4. hdfs-site.xml
3.4.1. 在mini01和mini02中
# 在mini01、mini02、mini05、mini06、mini07 可使用如下配置
[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
……………………
--> <!-- Put site-specific property overrides in this file. --> <configuration>
<!--指定hdfs的nameservice为bi,dt,需要和core-site.xml中的保持一致 -->
<property>
<name>dfs.nameservices</name>
<value>bi,dt</value>
</property> <!-- bi下面有两个NameNode,分别是nn1,nn2 -->
<property>
<name>dfs.ha.namenodes.bi</name>
<value>nn1,nn2</value>
</property> <!-- dt下面有两个NameNode,分别是nn3,nn4 -->
<property>
<name>dfs.ha.namenodes.dt</name>
<value>nn3,nn4</value>
</property> <!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.bi.nn1</name>
<value>mini01:</value>
</property>
<!-- nn1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.bi.nn1</name>
<value>mini01:</value>
</property> <!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.bi.nn2</name>
<value>mini02:</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.bi.nn2</name>
<value>mini02:</value>
</property> <!-- nn3的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.dt.nn3</name>
<value>mini03:</value>
</property>
<!-- nn3的http通信地址 -->
<property>
<name>dfs.namenode.http-address.dt.nn3</name>
<value>mini03:</value>
</property> <!-- nn4的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.dt.nn4</name>
<value>mini04:</value>
</property>
<!-- nn4的http通信地址 -->
<property>
<name>dfs.namenode.http-address.dt.nn4</name>
<value>mini04:</value>
</property> <!-- 指定NameNode的edits元数据在JournalNode上的存放位置 -->
<!-- 在bi名称空间的两个namenode中用如下配置 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://mini05:8485;mini06:8485;mini07:8485/bi</value>
</property>
<!-- 在dt名称空间的两个namenode中用如下配置 # 在mini01和mini02中注释掉 -->
<!--
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://mini05:8485;mini06:8485;mini07:8485/dt</value>
</property>
--> <!-- 指定JournalNode在本地磁盘存放数据的位置 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/app/hadoop/journaldata</value>
</property> <!-- 开启NameNode失败自动切换 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property> <!-- 配置失败自动切换实现方式 -->
<property>
<name>dfs.client.failover.proxy.provider.bi</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.dt</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property> <!-- 配置隔离机制方法,多个机制用换行分割,即每个机制暂用一行-->
<!-- 其中shell(/bin/true) 表示可执行一个脚本 比如 shell(/app/yunwei/hadoop_fence.sh) -->
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property> <!-- 使用sshfence隔离机制时需要ssh免登陆 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/app/.ssh/id_rsa</value>
</property> <!-- 配置sshfence隔离机制超时时间 单位:毫秒 -->
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value></value>
</property> </configuration>
3.4.2. 在mini03和mini04中
# 在mini03、mini04 可使用如下配置
[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
……………………
--> <!-- Put site-specific property overrides in this file. --> <configuration>
<!--指定hdfs的nameservice为bi,dt,需要和core-site.xml中的保持一致 -->
<property>
<name>dfs.nameservices</name>
<value>bi,dt</value>
</property> <!-- bi下面有两个NameNode,分别是nn1,nn2 -->
<property>
<name>dfs.ha.namenodes.bi</name>
<value>nn1,nn2</value>
</property> <!-- dt下面有两个NameNode,分别是nn3,nn4 -->
<property>
<name>dfs.ha.namenodes.dt</name>
<value>nn3,nn4</value>
</property> <!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.bi.nn1</name>
<value>mini01:</value>
</property>
<!-- nn1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.bi.nn1</name>
<value>mini01:</value>
</property> <!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.bi.nn2</name>
<value>mini02:</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.bi.nn2</name>
<value>mini02:</value>
</property> <!-- nn3的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.dt.nn3</name>
<value>mini03:</value>
</property>
<!-- nn3的http通信地址 -->
<property>
<name>dfs.namenode.http-address.dt.nn3</name>
<value>mini03:</value>
</property> <!-- nn4的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.dt.nn4</name>
<value>mini04:</value>
</property>
<!-- nn4的http通信地址 -->
<property>
<name>dfs.namenode.http-address.dt.nn4</name>
<value>mini04:</value>
</property> <!-- 指定NameNode的edits元数据在JournalNode上的存放位置 -->
<!-- 在bi名称空间的两个namenode中用如下配置 # 在mini03和mini04中注释掉 -->
<!--
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://mini05:8485;mini06:8485;mini07:8485/bi</value>
</property>
-->
<!-- 在dt名称空间的两个namenode中用如下配置 # 在mini01和mini02中注释掉 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://mini05:8485;mini06:8485;mini07:8485/dt</value>
</property> <!-- 指定JournalNode在本地磁盘存放数据的位置 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/app/hadoop/journaldata</value>
</property> <!-- 开启NameNode失败自动切换 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property> <!-- 配置失败自动切换实现方式 -->
<property>
<name>dfs.client.failover.proxy.provider.bi</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.dt</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property> <!-- 配置隔离机制方法,多个机制用换行分割,即每个机制暂用一行-->
<!-- 其中shell(/bin/true) 表示可执行一个脚本 比如 shell(/app/yunwei/hadoop_fence.sh) -->
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property> <!-- 使用sshfence隔离机制时需要ssh免登陆 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/app/.ssh/id_rsa</value>
</property> <!-- 配置sshfence隔离机制超时时间 单位:毫秒 -->
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value></value>
</property> </configuration>
3.5. mapred-site.xml
# 较上篇文章无改变
[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ cp -a mapred-site.xml.template mapred-site.xml
[yun@mini01 hadoop]$ vim mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
……………………
--> <!-- Put site-specific property overrides in this file. --> <configuration>
<!-- 指定mr框架为yarn方式 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property> </configuration>
3.6. yarn-site.xml
# 较上篇文章无改变
[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim yarn-site.xml
<?xml version="1.0"?>
<!--
……………………
-->
<configuration> <!-- Site specific YARN configuration properties -->
<!-- 开启RM高可用 -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property> <!-- 指定RM的cluster id -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yrc</value>
</property> <!-- 指定RM的名字 -->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property> <!-- 分别指定RM的地址 -->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>mini03</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>mini04</value>
</property> <!-- 指定zk集群地址 -->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>mini03:,mini04:,mini05:,mini06:,mini07:</value>
</property> <!-- reduce 获取数据的方式 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property> </configuration>
3.7. 修改slaves
slaves是指定子节点的位置,因为要在mini01上启动HDFS、在mini03启动yarn,所以mini01上的slaves文件指定的是datanode的位置,mini03上的slaves文件指定的是nodemanager的位置
# 较上篇文章无改变
[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim slaves
mini05
mini06
mini07
PS:改后配置后,将这些配置拷到其他Hadoop机器
4. 启动相关服务
注意:第一次启动时严格按照下面的步骤!!!!!!!
4.1. 启动zookeeper集群
前面已经启动了,这里就不说了
4.2. 启动journalnode
# 根据规划在mini05、mini06、mini07 启动
# 在第一次格式化的时候需要先启动journalnode 之后就不必了
[yun@mini05 ~]$ hadoop-daemon.sh start journalnode # 已经配置环境变量,所以不用进入到响应的目录
starting journalnode, logging to /app/hadoop-2.7./logs/hadoop-yun-journalnode-mini05.out
[yun@mini05 ~]$ jps
QuorumPeerMain
Jps
JournalNode
4.3. 在bi的格式化
4.3.1. 格式化HDFS
# 在mini01上执行命令
[yun@mini01 ~]$ hdfs namenode -format -clusterID CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f # 其中UUID可以在网上生成一个
// :: INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = mini01/10.0.0.111
STARTUP_MSG: args = [-format, -clusterID, CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f]
STARTUP_MSG: version = 2.7.6
STARTUP_MSG: classpath = ………………
STARTUP_MSG: build = Unknown -r Unknown; compiled by 'root' on 2018-06-08T08:30Z
STARTUP_MSG: java = 1.8.0_112
************************************************************/
// :: INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
// :: INFO namenode.NameNode: createNameNode [-format, -clusterID, CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f]
Formatting using clusterid: CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f
// :: INFO namenode.FSNamesystem: No KeyProvider found.
// :: INFO namenode.FSNamesystem: fsLock is fair: true
// :: INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
// :: INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=
// :: INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
// :: INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to :::00.000
// :: INFO blockmanagement.BlockManager: The block deletion will start around Jul ::
// :: INFO util.GSet: Computing capacity for map BlocksMap
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
// :: INFO blockmanagement.BlockManager: defaultReplication =
// :: INFO blockmanagement.BlockManager: maxReplication =
// :: INFO blockmanagement.BlockManager: minReplication =
// :: INFO blockmanagement.BlockManager: maxReplicationStreams =
// :: INFO blockmanagement.BlockManager: replicationRecheckInterval =
// :: INFO blockmanagement.BlockManager: encryptDataTransfer = false
// :: INFO blockmanagement.BlockManager: maxNumBlocksToLog =
// :: INFO namenode.FSNamesystem: fsOwner = yun (auth:SIMPLE)
// :: INFO namenode.FSNamesystem: supergroup = supergroup
// :: INFO namenode.FSNamesystem: isPermissionEnabled = true
// :: INFO namenode.FSNamesystem: Determined nameservice ID: bi
// :: INFO namenode.FSNamesystem: HA Enabled: true
// :: INFO namenode.FSNamesystem: Append Enabled: true
// :: INFO util.GSet: Computing capacity for map INodeMap
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO namenode.FSDirectory: ACLs enabled? false
// :: INFO namenode.FSDirectory: XAttrs enabled? true
// :: INFO namenode.FSDirectory: Maximum size of an xattr:
// :: INFO namenode.NameNode: Caching file names occuring more than times
// :: INFO util.GSet: Computing capacity for map cachedBlocks
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes =
// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.extension =
// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets =
// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users =
// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = ,,
// :: INFO namenode.FSNamesystem: Retry cache on namenode is enabled
// :: INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is millis
// :: INFO util.GSet: Computing capacity for map NameNodeRetryCache
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO namenode.FSImage: Allocated new BlockPoolId: BP--10.0.0.111-
// :: INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted.
// :: INFO namenode.FSImageFormatProtobuf: Saving image file /app/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
// :: INFO namenode.FSImageFormatProtobuf: Image file /app/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size bytes saved in seconds.
// :: INFO namenode.NNStorageRetentionManager: Going to retain images with txid >=
// :: INFO util.ExitUtil: Exiting with status
// :: INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at mini01/10.0.0.111
************************************************************/
拷贝到mini02
#格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件,这里我配置的是/app/hadoop/tmp,然后将/app/hadoop/tmp拷贝到mini02的/app/hadoop/下。
# 方法1:
[yun@mini01 hadoop]$ pwd
/app/hadoop
[yun@mini01 hadoop]$ scp -r tmp/ yun@mini02:/app/hadoop
VERSION % .4KB/s :
seen_txid % .0KB/s :
fsimage_0000000000000000000.md5 % .7KB/s :
fsimage_0000000000000000000 % .1KB/s : ##########################
# 方法2:##也可以这样,建议hdfs namenode -bootstrapStandby # 不过需要mini02的Hadoop起来才行
4.3.2. 格式化ZKFC
# 在mini01上执行一次即可
[yun@mini01 current]$ hdfs zkfc -formatZK
// :: INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at mini01/10.0.0.111:
// :: INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.-, built on // : GMT
// :: INFO zookeeper.ZooKeeper: Client environment:host.name=mini01
// :: INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_112
// :: INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
// :: INFO zookeeper.ZooKeeper: Client environment:java.home=/app/jdk1..0_112/jre
// :: INFO zookeeper.ZooKeeper: Client environment:java.class.path=……………………
// :: INFO zookeeper.ZooKeeper: Client environment:java.library.path=/app/hadoop-2.7./lib/native
// :: INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
// :: INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
// :: INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
// :: INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
// :: INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.-.el7.x86_64
// :: INFO zookeeper.ZooKeeper: Client environment:user.name=yun
// :: INFO zookeeper.ZooKeeper: Client environment:user.home=/app
// :: INFO zookeeper.ZooKeeper: Client environment:user.dir=/app/hadoop-2.7.
// :: INFO zookeeper.ZooKeeper: Initiating client connection, connectString=mini03:,mini04:,mini05:,mini06:,mini07: sessionTimeout= watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@7f3b84b8
// :: INFO zookeeper.ClientCnxn: Opening socket connection to server mini04/10.0.0.114:. Will not attempt to authenticate using SASL (unknown error)
// :: INFO zookeeper.ClientCnxn: Socket connection established to mini04/10.0.0.114:, initiating session
// :: INFO zookeeper.ClientCnxn: Session establishment complete on server mini04/10.0.0.114:, sessionid = 0x464538f087b0003, negotiated timeout =
// :: INFO ha.ActiveStandbyElector: Session connected.
// :: INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/bi in ZK.
// :: INFO zookeeper.ZooKeeper: Session: 0x464538f087b0003 closed
// :: INFO zookeeper.ClientCnxn: EventThread shut down
4.4. 在dt的格式化
4.4.1. 格式化HDFS
# 在mini03上执行命令
[yun@mini03 hadoop]$ hdfs namenode -format -clusterID CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f # clusterID必须与bi的相同
// :: INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = mini03/10.0.0.113
STARTUP_MSG: args = [-format, -clusterID, CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f]
STARTUP_MSG: version = 2.7.6
STARTUP_MSG: classpath = ……………………
STARTUP_MSG: build = Unknown -r Unknown; compiled by 'root' on 2018-06-08T08:30Z
STARTUP_MSG: java = 1.8.0_112
************************************************************/
// :: INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
// :: INFO namenode.NameNode: createNameNode [-format, -clusterID, CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f]
Formatting using clusterid: CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f
// :: INFO namenode.FSNamesystem: No KeyProvider found.
// :: INFO namenode.FSNamesystem: fsLock is fair: true
// :: INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
// :: INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=
// :: INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
// :: INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to :::00.000
// :: INFO blockmanagement.BlockManager: The block deletion will start around Jul ::
// :: INFO util.GSet: Computing capacity for map BlocksMap
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
// :: INFO blockmanagement.BlockManager: defaultReplication =
// :: INFO blockmanagement.BlockManager: maxReplication =
// :: INFO blockmanagement.BlockManager: minReplication =
// :: INFO blockmanagement.BlockManager: maxReplicationStreams =
// :: INFO blockmanagement.BlockManager: replicationRecheckInterval =
// :: INFO blockmanagement.BlockManager: encryptDataTransfer = false
// :: INFO blockmanagement.BlockManager: maxNumBlocksToLog =
// :: INFO namenode.FSNamesystem: fsOwner = yun (auth:SIMPLE)
// :: INFO namenode.FSNamesystem: supergroup = supergroup
// :: INFO namenode.FSNamesystem: isPermissionEnabled = true
// :: INFO namenode.FSNamesystem: Determined nameservice ID: dt
// :: INFO namenode.FSNamesystem: HA Enabled: true
// :: INFO namenode.FSNamesystem: Append Enabled: true
// :: INFO util.GSet: Computing capacity for map INodeMap
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO namenode.FSDirectory: ACLs enabled? false
// :: INFO namenode.FSDirectory: XAttrs enabled? true
// :: INFO namenode.FSDirectory: Maximum size of an xattr:
// :: INFO namenode.NameNode: Caching file names occuring more than times
// :: INFO util.GSet: Computing capacity for map cachedBlocks
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes =
// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.extension =
// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets =
// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users =
// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = ,,
// :: INFO namenode.FSNamesystem: Retry cache on namenode is enabled
// :: INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is millis
// :: INFO util.GSet: Computing capacity for map NameNodeRetryCache
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO namenode.FSImage: Allocated new BlockPoolId: BP--10.0.0.113-
// :: INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted.
// :: INFO namenode.FSImageFormatProtobuf: Saving image file /app/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
// :: INFO namenode.FSImageFormatProtobuf: Image file /app/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size bytes saved in seconds.
// :: INFO namenode.NNStorageRetentionManager: Going to retain images with txid >=
// :: INFO util.ExitUtil: Exiting with status
// :: INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at mini03/10.0.0.113
************************************************************/
拷贝到mini04
#格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件,这里我配置的是/app/hadoop/tmp,然后将/app/hadoop/tmp拷贝到mini04的/app/hadoop/下。
# 方法1:
[yun@mini03 hadoop]$ pwd
/app/hadoop
[yun@mini03 hadoop]$ scp -r tmp/ yun@mini04:/app/hadoop
VERSION % .4KB/s :
seen_txid % .0KB/s :
fsimage_0000000000000000000.md5 % .7KB/s :
fsimage_0000000000000000000 % .1KB/s : ##########################
# 方法2:##也可以这样,建议hdfs namenode -bootstrapStandby # 不过需要mini04的Hadoop起来才行
4.4.2. 格式化ZKFC
# 在mini03上执行一次即可
[yun@mini03 hadoop]$ hdfs zkfc -formatZK
// :: INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at mini03/10.0.0.113:
// :: INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.-, built on // : GMT
// :: INFO zookeeper.ZooKeeper: Client environment:host.name=mini03
// :: INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_112
// :: INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
// :: INFO zookeeper.ZooKeeper: Client environment:java.home=/app/jdk1..0_112/jre
// :: INFO zookeeper.ZooKeeper: Client environment:java.class.path=……………………
// :: INFO zookeeper.ZooKeeper: Client environment:java.library.path=/app/hadoop-2.7./lib/native
// :: INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
// :: INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
// :: INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
// :: INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
// :: INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.-.el7.x86_64
// :: INFO zookeeper.ZooKeeper: Client environment:user.name=yun
// :: INFO zookeeper.ZooKeeper: Client environment:user.home=/app
// :: INFO zookeeper.ZooKeeper: Client environment:user.dir=/app/hadoop-2.7.
// :: INFO zookeeper.ZooKeeper: Initiating client connection, connectString=mini03:,mini04:,mini05:,mini06:,mini07: sessionTimeout= watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@7f3b84b8
// :: INFO zookeeper.ClientCnxn: Opening socket connection to server mini05/10.0.0.115:. Will not attempt to authenticate using SASL (unknown error)
// :: INFO zookeeper.ClientCnxn: Socket connection established to mini05/10.0.0.115:, initiating session
// :: INFO zookeeper.ClientCnxn: Session establishment complete on server mini05/10.0.0.115:, sessionid = 0x56455467ae50003, negotiated timeout =
// :: INFO ha.ActiveStandbyElector: Session connected.
// :: INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/dt in ZK.
// :: INFO zookeeper.ZooKeeper: Session: 0x56455467ae50003 closed
// :: INFO zookeeper.ClientCnxn: EventThread shut down
4.5. 启动HDFS
# 在mini01操作一次即可
[yun@mini01 hadoop]$ start-dfs.sh
Starting namenodes on [mini01 mini02 mini03 mini04]
mini01: starting namenode, logging to /app/hadoop-2.7./logs/hadoop-yun-namenode-mini01.out
mini02: starting namenode, logging to /app/hadoop-2.7./logs/hadoop-yun-namenode-mini02.out
mini04: starting namenode, logging to /app/hadoop-2.7./logs/hadoop-yun-namenode-mini04.out
mini03: starting namenode, logging to /app/hadoop-2.7./logs/hadoop-yun-namenode-mini03.out
mini06: starting datanode, logging to /app/hadoop-2.7./logs/hadoop-yun-datanode-mini06.out
mini07: starting datanode, logging to /app/hadoop-2.7./logs/hadoop-yun-datanode-mini07.out
mini05: starting datanode, logging to /app/hadoop-2.7./logs/hadoop-yun-datanode-mini05.out
Starting journal nodes [mini05 mini06 mini07]
mini05: journalnode running as process . Stop it first.
mini07: journalnode running as process . Stop it first.
mini06: journalnode running as process . Stop it first.
Starting ZK Failover Controllers on NN hosts [mini01 mini02 mini03 mini04]
mini04: starting zkfc, logging to /app/hadoop-2.7./logs/hadoop-yun-zkfc-mini04.out
mini01: starting zkfc, logging to /app/hadoop-2.7./logs/hadoop-yun-zkfc-mini01.out
mini02: starting zkfc, logging to /app/hadoop-2.7./logs/hadoop-yun-zkfc-mini02.out
mini03: starting zkfc, logging to /app/hadoop-2.7./logs/hadoop-yun-zkfc-mini03.out
4.6. 启动YARN
#####注意#####:是在mini03上执行start-yarn.sh,把namenode和resourcemanager分开是因为性能问题
# 因为他们都要占用大量资源,所以把他们分开了,他们分开了就要分别在不同的机器上启动
[yun@mini03 ~]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /app/hadoop-2.7./logs/yarn-yun-resourcemanager-mini03.out
mini07: starting nodemanager, logging to /app/hadoop-2.7./logs/yarn-yun-nodemanager-mini07.out
mini06: starting nodemanager, logging to /app/hadoop-2.7./logs/yarn-yun-nodemanager-mini06.out
mini05: starting nodemanager, logging to /app/hadoop-2.7./logs/yarn-yun-nodemanager-mini05.out ################################
# 在mini04启动 resourcemanager
[yun@mini04 ~]$ yarn-daemon.sh start resourcemanager # 也可用start-yarn.sh
starting resourcemanager, logging to /app/hadoop-2.7./logs/yarn-yun-resourcemanager-mini04.out
4.7. 启动说明
# 第一次启动的时候请严格按照上面的步骤【第一次涉及格式化问题】
# 第二次以及之后,步骤为: 启动zookeeper、HDFS、YARN
5. 浏览器访问
5.1. bi访问
http://mini01:50070
http://mini02:50070
5.2. dt访问
http://mini03:50070
http://mini04:50070
6. 简单操作
6.1. HDFS创建目录与上传
[yun@mini02 software]$ pwd
/app/software
[yun@mini02 software]$ ll
total
-rw-r--r-- yun yun Jun : CentOS-.4_hadoop-2.7..tar.gz
-rw-r--r-- yun yun May : eclipse-jee-oxygen-3a-win32-x86_64.zip
-rw-r--r-- yun yun Oct jdk1..0_112.tar.gz
-rw-rw-r-- yun yun Jul : test
[yun@mini02 software]$ hadoop fs -ls / # 通过命令行可见,是进行了逻辑分区的
Found items
-r-xr-xr-x - yun yun -- : /bi
-r-xr-xr-x - yun yun -- : /dt
[yun@mini02 software]$ hadoop fs -mkdir /bi/software # 在bi创建目录
[yun@mini02 software]$
[yun@mini02 software]$ hadoop fs -mkdir /dt/software # 在dt创建目录
[yun@mini02 software]$
[yun@mini02 software]$ hadoop fs -put jdk1..0_112.tar.gz /bi/software # 上传文件到bi
[yun@mini02 software]$ hadoop fs -put eclipse-jee-oxygen-3a-win32-x86_64.zip /dt/software # 上传文件到dt
[yun@mini02 software]$
[yun@mini02 software]$
[yun@mini02 software]$ hadoop fs -ls /bi/software
Found items
-rw-r--r-- yun supergroup -- : /bi/software/jdk1..0_112.tar.gz
[yun@mini02 software]$
[yun@mini02 software]$
[yun@mini02 software]$ hadoop fs -ls /dt/software
Found items
-rw-r--r-- yun supergroup -- : /dt/software/eclipse-jee-oxygen-3a-win32-x86_64.zip
6.2. Datanode目录结构
[yun@mini05 current]$ pwd
/app/hadoop/tmp/dfs/data/current
[yun@mini05 current]$ ll
total
drwx------ yun yun Jul : BP--10.0.0.111-
drwx------ yun yun Jul : BP--10.0.0.113-
-rw-rw-r-- yun yun Jul : VERSION
[yun@mini05 current]$ tree
.
├── BP--10.0.0.111-
│ ├── current
│ │ ├── finalized
│ │ │ └── subdir0
│ │ │ └── subdir0
│ │ │ ├── blk_1073741825
│ │ │ ├── blk_1073741825_1001.meta
│ │ │ ├── blk_1073741826
│ │ │ └── blk_1073741826_1002.meta
│ │ ├── rbw
│ │ └── VERSION
│ ├── scanner.cursor
│ └── tmp
├── BP--10.0.0.113-
│ ├── current
│ │ ├── finalized
│ │ │ └── subdir0
│ │ │ └── subdir0
│ │ │ ├── blk_1073741825
│ │ │ ├── blk_1073741825_1001.meta
│ │ │ ├── blk_1073741826
│ │ │ ├── blk_1073741826_1002.meta
│ │ │ ├── blk_1073741827
│ │ │ └── blk_1073741827_1003.meta
│ │ ├── rbw
│ │ └── VERSION
│ ├── scanner.cursor
│ └── tmp
└── VERSION directories, files
Hadoop2.7.6_08_Federation联邦机制的更多相关文章
- Hadoop记录-Federation联邦机制
在Hadoop2.0之前,HDFS的单NameNode设计带来诸多问题: 单点故障.内存受限,制约集群扩展性和缺乏隔离机制(不同业务使用同一个NameNode导致业务相互影响)等 为了解决这些问题, ...
- 2-10 就业课(2.0)-oozie:11、hadoop的federation(联邦机制,了解一下)
==================================================== Hadoop Federation 背景概述 单NameNode的架构使得HDFS在集群扩展性 ...
- HDFS 10 - HDFS 的联邦机制(Federation 机制)
目录 1 - 为什么需要联邦 2 - Federation 架构设计 3 HDFS Federation 的不足 版权声明 1 - 为什么需要联邦 单 NameNode 的架构存在的问题:当集群中数据 ...
- Hadoop_HDFS架构和HA机制
Hadoop学习笔记总结 01.HDFS架构 1. NameNode和ResourceManager NameNode负责HDFS,从节点是DataNode:ResourceManager负责MapR ...
- Cloudera Hadoop 5& Hadoop高阶管理及调优课程(CDH5,Hadoop2.0,HA,安全,管理,调优)
1.课程环境 本课程涉及的技术产品及相关版本: 技术 版本 Linux CentOS 6.5 Java 1.7 Hadoop2.0 2.6.0 Hadoop1.0 1.2.1 Zookeeper 3. ...
- Prometheus监控学习笔记之prometheus的federation机制
0x00 概述 有时候对于一个公司,k8s集群或是所谓的caas只是整个技术体系的一部分,往往这个时候监控系统不仅仅要k8s集群以及k8s中部署的应用,而且要监控传统部署的项目.也就是说整个监控系统不 ...
- 联邦学习(Federated Learning)
联邦学习简介 联邦学习(Federated Learning)是一种新兴的人工智能基础技术,在 2016 年由谷歌最先提出,原本用于解决安卓手机终端用户在本地更新模型的问题,其设计目标是 ...
- Federal Learning(联邦学习)认知
本人是学生党,同时也是小菜鸡一枚,撞运气有机会能够给老师当项目助理,在这个过程中肯定会学到一些有趣的知识,就在此平台上记录一下,在知识点方面有不对的还请各位指正. What(什么是联邦学习?) 联邦学 ...
- 大数据学习笔记4 - Hadoop的优化与发展(Hadoop 2.0)
前面介绍了Hadoop核心组件HDFS和MapReduce,Hadoop发展之初在架构设计和应用性能方面仍然存在不足,Hadoop的优化与发展一方面体现在两个核心组件的架构设计改进,一方面体现在Had ...
随机推荐
- 用RIPv2实现网络区域的互通
1.动态路由的分类: DV协议:距离矢量协议 距离矢量:路由器只能够知道邻居路由的信息 LS协议:链路状态协议 链路状态:路由器能够知道所在协议内的所有信息 RIP协议的全程是:路由信息协议(DV协议 ...
- zookeeper集群操作【这里只说明简单的操作步骤,zk的相关参数、说明请参考官方文档】
本文版权归 远方的风lyh和博客园共有,欢迎转载,但须保留此段声明,并给出原文链接,谢谢合作. [这里是在一台机器上搭建的 zk伪集群] 1.从官网下载下载zk http://apa ...
- js常用面试题整理
1.array操作关键字: pop() 删除最后一个:push最后添加一个或者多个:reverse颠倒数组:shift删除第一个元素:unshift首部添加元素:concat衔接两个数组:join把数 ...
- 负载均衡+session共享(memcached-session-manager实现)
前言 先给大家伙拜个年,祝大家:新的一年健健康康,平平安安! 本文的形成参考了很多人的博客,最多的应该是青葱岁月兄的这篇博客,大家可以先去看下,熟悉一些内容,因为本文是直接实践,一些理论性的知识就需要 ...
- 深入React技术栈之初入React世界
1.1 react简介 react并不是完整的MVM/MVVM框架,专注于提供清晰.简洁的View层解决方案. 传统开发模式,要更新页面需要手动操作DOM元素.如图1.1所示,React在DOM上封装 ...
- rootkit(kbeast-v1)
Rootkit有应用级.内核级和硬件级 用的比较多的是内核级别,比如基于linux LKM编写的rootkit rootkit可以理解为一个超级管理员的工具箱,这个工具箱通过调用系统LKM接口可以动态 ...
- Python装饰器举例分析
概述 装饰器本质上是一个Python函数,它可以让其他函数在不需要做任何代码变动的前提下增加额外功能,装饰器的返回值也是一个函数对象. 我们要需要一个能测试函数运行时间的decorator,可以定义如 ...
- Sharepoint 2010 工作流状态值
在Sharepoint2010中,如果要使用工作流状态值进行筛选,必须使用内部值,不能使用文字,要不然是筛选不出来的. 进行中:2 已取消:4 已批准:16 拒绝:17 下边是已取消的工作流状态:
- c# 获取本机IP
/// <summary> /// 获取本机IP /// </summary> /// <returns></returns> public stati ...
- IdentityServer4-客户端定义-翻译
客户端定义(Defining Client) 客户端可以从你的IDS服务器请求tokens. 通常,客户端需要遵循下面的通用设置: 一个唯一的Client ID 如果需要还可以提供密码 允许与toke ...