Hadoop部署方式-完全分布式(Fully-Distributed Mode)

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

　　

　　本博客搭建的虚拟机是伪分布式环境（https://www.cnblogs.com/yinzhengjie/p/9058415.html）链接克隆出来的，我们只需要修改一下配置文件就可以轻松实现完全分布式部署了，部署架构是一个NameNode和三个DataNode，如果身为一个专业的运维人员你可能会一眼看出来这个集群存在单点故障，别着急，关于高可用集群部署请参考：https://www.cnblogs.com/yinzhengjie/p/9070017.html。

如果你是mac用户推荐使用"Parallets "，如果你是windows系统推荐使用“VMware Workstation”，如果是Linux用户的小伙伴推荐使用“VirtualBox”。我的实验环境在windows上操作的，安装的是VMware Workstation。

一.实验环境准备
　　需要准备四台Linux操作系统的服务器，配置参数最好一样，由于我的虚拟机是之前伪分布式部署而来的，因此我的环境都一致，并且每天虚拟机默认都是Hadoop伪分布式哟！
1>.NameNode服务器（172.16.30.101）

2>.DataNode服务器（172.16.30.102）

3>.DataNode服务器（172.16.30.103）

4>.DataNode服务器（172.16.30.104）

二.修改Hadoop的配置文件

　　修改的配置文件路径是我之前拷贝的full目录，绝对路径是：“/soft/hadoop/etc/full”,修改这个目录下的文件之后，我们将hadoop目录连接过来即可，当你需要伪分布式或者本地模式的时候只需要改变软连接指向的目录即可，这样就轻松实现了三种模式配置文件和平相处的局面。

1>.core-site.xml 配置文件

[yinzhengjie@s101 ~]$ more /soft/hadoop/etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<configuration>

                <property>

                        <name>fs.defaultFS</name>

                        <value>hdfs://s101:8020</value>

                </property>

                <property>

                        <name>hadoop.tmp.dir</name>

                        <value>/home/yinzhengjie/hadoop</value>

                </property>

</configuration>

<!--

core-site.xml配置文件的作用：

    用于定义系统级别的参数，如HDFS URL、Hadoop的临时

目录以及用于rack-aware集群中的配置文件的配置等，此中的参

数定义会覆盖core-default.xml文件中的默认配置。

fs.defaultFS 参数的作用：

        #声明namenode的地址，相当于声明hdfs文件系统。

hadoop.tmp.dir 参数的作用：

        #声明hadoop工作目录的地址。

-->

[yinzhengjie@s101 ~]$

2>.hdfs-site.xml 配置文件

[yinzhengjie@s101 ~]$ more /soft/hadoop/etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

        <property>

                <name>dfs.replication</name>

                <value></value>

        </property>

</configuration>

<!--

hdfs-site.xml 配置文件的作用：

        #HDFS的相关设定，如文件副本的个数、块大小及是否使用强制权限

等，此中的参数定义会覆盖hdfs-default.xml文件中的默认配置.

dfs.replication 参数的作用：

        #为了数据可用性及冗余的目的，HDFS会在多个节点上保存同一个数据

块的多个副本，其默认为3个。而只有一个节点的伪分布式环境中其仅用

保存一个副本即可，这可以通过dfs.replication属性进行定义。它是一个

软件级备份。

-->

[yinzhengjie@s101 ~]$

3>.mapred-site.xml 配置文件

[yinzhengjie@s101 ~]$ more /soft/hadoop/etc/hadoop/mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

        <property>

                <name>mapreduce.framework.name</name>

                <value>yarn</value>

        </property>

</configuration>

<!--

mapred-site.xml 配置文件的作用：

        #HDFS的相关设定，如reduce任务的默认个数、任务所能够使用内存

的默认上下限等，此中的参数定义会覆盖mapred-default.xml文件中的

默认配置.

mapreduce.framework.name 参数的作用：

        #指定MapReduce的计算框架，有三种可选，第一种：local(本地),第

二种是classic(hadoop一代执行框架),第三种是yarn(二代执行框架)，我

们这里配置用目前版本最新的计算框架yarn即可。

-->

[yinzhengjie@s101 ~]$

4>.yarn-site.xml配置文件

[yinzhengjie@s101 ~]$ more /soft/hadoop/etc/hadoop/yarn-site.xml

<?xml version="1.0"?>

<configuration>

                <property>

                        <name>yarn.resourcemanager.hostname</name>

                        <value>s101</value>

                </property>

        <property>

                <name>yarn.nodemanager.aux-services</name>

                <value>mapreduce_shuffle</value>

        </property>

</configuration>

<!--

yarn-site.xml配置文件的作用：

        #主要用于配置调度器级别的参数.

yarn.resourcemanager.hostname 参数的作用：

        #指定资源管理器(resourcemanager)的主机名

yarn.nodemanager.aux-services 参数的作用：

        #指定nodemanager使用shuffle

-->

[yinzhengjie@s101 ~]$

 5>.slaves配置文件

[yinzhengjie@s101 ~]$ more /soft/hadoop/etc/hadoop/slaves

#该配置文件的作用：是NameNode用与记录需要连接哪些DataNode服务器节点，用与启动或停止服务时发送远程命令指令的目标主机。

s102

s103

s104

[yinzhengjie@s101 ~]$

三.在NameNode节点上配置免密码登录各DataNode节点

1>.在本地上生成公私秘钥对(生成之前，把上次部署伪分布式的秘钥删除掉)

[yinzhengjie@s101 ~]$ rm -rf ~/.ssh/*

[yinzhengjie@s101 ~]$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

Generating public/private rsa key pair.

Your identification has been saved in /home/yinzhengjie/.ssh/id_rsa.

Your public key has been saved in /home/yinzhengjie/.ssh/id_rsa.pub.

The key fingerprint is:

a3:a4:ae:d8:f7:7f:a2:b6:d6:15:74:29:de:fb:14:08 yinzhengjie@s101

The key's randomart image is:

+--[ RSA 2048]----+

|             .   |

|          E o    |

|         o = .   |

|          o o .  |

|      . S  . . . |

|     o . .. . .  |

|    . .. .   o   |

| o .. o o .   .  |

|. oo.+++.o       |

+-----------------+

[yinzhengjie@s101 ~]$

2>.使用ssh-copy-id命令分配公钥到DataNode服务器（172.16.30.101）

[yinzhengjie@s101 ~]$ ssh-copy-id yinzhengjie@s101

The authenticity of host 's101 (172.16.30.101)' can't be established.

ECDSA key fingerprint is fa::bc::7e::eb::1e:bc:a8:c9:ce::ba:7b.

Are you sure you want to continue connecting (yes/no)? yes

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO:  key(s) remain to be installed -- if you are prompted now it is to install the new keys

yinzhengjie@s101's password: 

Number of key(s) added: 

Now try logging into the machine, with:   "ssh 'yinzhengjie@s101'"

and check to make sure that only the key(s) you wanted were added.

[yinzhengjie@s101 ~]$ ssh s101

Last login: Fri May  ::  from 172.16.30.1

[yinzhengjie@s101 ~]$ who

yinzhengjie pts/        -- : (172.16.30.1)

yinzhengjie pts/        -- : (s101)

[yinzhengjie@s101 ~]$ exit

logout

Connection to s101 closed.

[yinzhengjie@s101 ~]$ who

yinzhengjie pts/        -- : (172.16.30.1)

[yinzhengjie@s101 ~]$

3>.使用ssh-copy-id命令分配公钥到DataNode服务器（172.16.30.102）

[yinzhengjie@s101 ~]$ ssh-copy-id yinzhengjie@s102

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO:  key(s) remain to be installed -- if you are prompted now it is to install the new keys

yinzhengjie@s102's password: 

Number of key(s) added: 

Now try logging into the machine, with:   "ssh 'yinzhengjie@s102'"

and check to make sure that only the key(s) you wanted were added.

[yinzhengjie@s101 ~]$ ssh s102

Last login: Fri May  ::  from 172.16.30.1

[yinzhengjie@s102 ~]$ who

yinzhengjie pts/        -- : (172.16.30.1)

yinzhengjie pts/        -- : (s101)

[yinzhengjie@s102 ~]$ exit

logout

Connection to s102 closed.

[yinzhengjie@s101 ~]$ who

yinzhengjie pts/        -- : (172.16.30.1)

[yinzhengjie@s101 ~]$

4>.使用ssh-copy-id命令分配公钥到DataNode服务器（172.16.30.103）

[yinzhengjie@s101 ~]$ ssh-copy-id yinzhengjie@s103

The authenticity of host 's103 (172.16.30.103)' can't be established.

ECDSA key fingerprint is fa::bc::7e::eb::1e:bc:a8:c9:ce::ba:7b.

Are you sure you want to continue connecting (yes/no)? yes

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO:  key(s) remain to be installed -- if you are prompted now it is to install the new keys

yinzhengjie@s103's password: 

Number of key(s) added: 

Now try logging into the machine, with:   "ssh 'yinzhengjie@s103'"

and check to make sure that only the key(s) you wanted were added.

[yinzhengjie@s101 ~]$ ssh s103

Last login: Fri May  ::  from 172.16.30.1

[yinzhengjie@s103 ~]$ who

yinzhengjie pts/        -- : (172.16.30.1)

yinzhengjie pts/        -- : (s101)

[yinzhengjie@s103 ~]$ exit

logout

Connection to s103 closed.

[yinzhengjie@s101 ~]$ who

yinzhengjie pts/        -- : (172.16.30.1)

[yinzhengjie@s101 ~]$

5>.使用ssh-copy-id命令分配公钥到DataNode服务器（172.16.30.104）

[yinzhengjie@s101 ~]$ ssh-copy-id yinzhengjie@s104

The authenticity of host 's104 (172.16.30.104)' can't be established.

ECDSA key fingerprint is fa::bc::7e::eb::1e:bc:a8:c9:ce::ba:7b.

Are you sure you want to continue connecting (yes/no)? yes

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO:  key(s) remain to be installed -- if you are prompted now it is to install the new keys

yinzhengjie@s104's password: 

Number of key(s) added: 

Now try logging into the machine, with:   "ssh 'yinzhengjie@s104'"

and check to make sure that only the key(s) you wanted were added.

[yinzhengjie@s101 ~]$ ssh s104

Last login: Fri May  ::  from 172.16.30.1

[yinzhengjie@s104 ~]$ who

yinzhengjie pts/        -- : (172.16.30.1)

yinzhengjie pts/        -- : (s101)

[yinzhengjie@s104 ~]$ exit

logout

Connection to s104 closed.

[yinzhengjie@s101 ~]$ who

yinzhengjie pts/        -- : (172.16.30.1)

[yinzhengjie@s101 ~]$

　　注意：以上是普通使配置免密登录，root用户配置方法一致，最好也配置上root用户的免密登录，因为下文我会执行相应的shell脚本。

[yinzhengjie@s101 ~]$ su

Password:

[root@s101 yinzhengjie]# cd

[root@s101 ~]#

[root@s101 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

Generating public/private rsa key pair.

Created directory '/root/.ssh'.

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

9b::9a:ca:d2:f9:a5:::::be::3a:ed: root@s101

The key's randomart image is:

+--[ RSA ]----+

|            ..   |

|            ..   |

|           E o.  |

|          . o o..|

|        S .+ + o.|

|         *  * o  |

|     . .= o. o   |

|    ..o. +.      |

|     .o.o.       |

+-----------------+

[root@s101 ~]#

[root@s101 ~]# ssh-copy-id root@s101

The authenticity of host 's101 (172.16.30.101)' can't be established.

ECDSA key fingerprint is fa::bc::7e::eb::1e:bc:a8:c9:ce::ba:7b.

Are you sure you want to continue connecting (yes/no)? yes

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO:  key(s) remain to be installed -- if you are prompted now it is to install the new keys

root@s101's password: 

Number of key(s) added: 

Now try logging into the machine, with:   "ssh 'root@s101'"

and check to make sure that only the key(s) you wanted were added.

[root@s101 ~]# ssh s101

Last login: Fri May  ::

[root@s101 ~]# who

yinzhengjie pts/        -- : (172.16.30.1)

root     pts/        -- : (s101)

[root@s101 ~]# exit

logout

Connection to s101 closed.

[root@s101 ~]# who

yinzhengjie pts/        -- : (172.16.30.1)

[root@s101 ~]#

生成秘钥对并使用ssh-copy-id命令分配公钥到DataNode服务器（172.16.30.101）

[root@s101 ~]# ssh-copy-id root@s102

The authenticity of host 's102 (172.16.30.102)' can't be established.

ECDSA key fingerprint is fa::bc::7e::eb::1e:bc:a8:c9:ce::ba:7b.

Are you sure you want to continue connecting (yes/no)? yes

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO:  key(s) remain to be installed -- if you are prompted now it is to install the new keys

root@s102's password: 

Number of key(s) added: 

Now try logging into the machine, with:   "ssh 'root@s102'"

and check to make sure that only the key(s) you wanted were added.

[root@s101 ~]# ssh s102

Last login: Fri May  ::

[root@s102 ~]# who

yinzhengjie pts/        -- : (172.16.30.1)

root     pts/        -- : (s101)

[root@s102 ~]# exit

logout

Connection to s102 closed.

[root@s101 ~]# who

yinzhengjie pts/        -- : (172.16.30.1)

[root@s101 ~]#

使用ssh-copy-id命令分配公钥到DataNode服务器（172.16.30.102）

[root@s101 ~]# ssh-copy-id root@s103

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO:  key(s) remain to be installed -- if you are prompted now it is to install the new keys

root@s103's password: 

Number of key(s) added: 

Now try logging into the machine, with:   "ssh 'root@s103'"

and check to make sure that only the key(s) you wanted were added.

[root@s101 ~]# ssh s103

Last login: Fri May  ::  from s101

[root@s103 ~]# who

yinzhengjie pts/        -- : (172.16.30.1)

root     pts/        -- : (s101)

[root@s103 ~]# exit

logout

Connection to s103 closed.

[root@s101 ~]# who

yinzhengjie pts/        -- : (172.16.30.1)

[root@s101 ~]#

使用ssh-copy-id命令分配公钥到DataNode服务器（172.16.30.103）

[root@s101 ~]# ssh-copy-id root@s104

The authenticity of host 's104 (172.16.30.104)' can't be established.

ECDSA key fingerprint is fa::bc::7e::eb::1e:bc:a8:c9:ce::ba:7b.

Are you sure you want to continue connecting (yes/no)? yes

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO:  key(s) remain to be installed -- if you are prompted now it is to install the new keys

root@s104's password: 

Number of key(s) added: 

Now try logging into the machine, with:   "ssh 'root@s104'"

and check to make sure that only the key(s) you wanted were added.

[root@s101 ~]# ssh s104

Last login: Fri May  ::

[root@s104 ~]# who

yinzhengjie pts/        -- : (172.16.30.1)

root     pts/        -- : (s101)

[root@s104 ~]# exit

logout

Connection to s104 closed.

[root@s101 ~]# who

yinzhengjie pts/        -- : (172.16.30.1)

[root@s101 ~]#

使用ssh-copy-id命令分配公钥到DataNode服务器（172.16.30.104）

四.自定义shell脚本

1>.编写在批量执行Linux命令的脚本（起名:xcall.sh）

[yinzhengjie@s101 ~]$ su

Password:

[root@s101 yinzhengjie]#

[root@s101 yinzhengjie]# ll

total

drwxrwxr-x.  yinzhengjie yinzhengjie         May  : hadoop

-rw-r--r--.  root        root         Aug    hadoop-2.7..tar.gz

-rw-r--r--.  root        root         May    jdk-8u131-linux-x64.tar.gz

-rw-rw-r--.  yinzhengjie yinzhengjie        May  : xcall.sh

[root@s101 yinzhengjie]# more xcall.sh

#!/bin/bash

#@author :yinzhengjie

#blog:http://www.cnblogs.com/yinzhengjie

#EMAIL:y1053419035@qq.com

#判断用户是否传参

if [ $# -lt  ];then

        echo "请输入参数"

        exit

fi

#获取用户输入的命令

cmd=$@

for (( i=;i<=;i++ ))

do

        #使终端变绿色

        tput setaf

        echo ============= s$i $cmd ============

        #使终端变回原来的颜色，即白灰色

        tput setaf

        #远程执行命令

        ssh s$i $cmd

        #判断命令是否执行成功

        if [ $? ==  ];then

                echo "命令执行成功"

        fi

done

[root@s101 yinzhengjie]#

[root@s101 yinzhengjie]# sh xcall.sh "ln -s /soft/jdk/bin/jps /usr/local/bin/jps"

============= s101 ln -s /soft/jdk/bin/jps /usr/local/bin/jps ============

命令执行成功

============= s102 ln -s /soft/jdk/bin/jps /usr/local/bin/jps ============

命令执行成功

============= s103 ln -s /soft/jdk/bin/jps /usr/local/bin/jps ============

命令执行成功

============= s104 ln -s /soft/jdk/bin/jps /usr/local/bin/jps ============

命令执行成功

[root@s101 yinzhengjie]# sh xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[root@s101 yinzhengjie]#

2>.编写远程同步脚本(起名：xrsync.sh)

　　在写同步脚本之前，每个客户端都应该安装的有rsync这个软件，检查各个虚拟机是否可以正常链接互联网，如果可以访问互联网直接yum安装一下，在各个虚拟机可以访问网络且yum源配置正确的情况下也可以用咱们上面的脚本（xcall.sh）执行批量安装哟！

[root@s101 yinzhengjie]# sh xcall.sh "yum -y install rsync"

============= s101 yum -y install rsync ============

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirrors.aliyun.com

 * extras: mirrors.huaweicloud.com

 * updates: mirrors.huaweicloud.com

Resolving Dependencies

--> Running transaction check

---> Package rsync.x86_64 :3.1.-.el7 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

================================================================================

 Package         Arch             Version                  Repository      Size

================================================================================

Installing:

 rsync           x86_64           3.1.-.el7              base            k

Transaction Summary

================================================================================

Install   Package

Total download size:  k

Installed size:  k

Downloading packages:

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

  Installing : rsync-3.1.-.el7.x86_64                                     /

  Verifying  : rsync-3.1.-.el7.x86_64                                     / 

Installed:

  rsync.x86_64 :3.1.-.el7                                                    

Complete!

命令执行成功

============= s102 yum -y install rsync ============

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirrors.aliyun.com

 * extras: mirrors.huaweicloud.com

 * updates: mirrors.huaweicloud.com

Resolving Dependencies

--> Running transaction check

---> Package rsync.x86_64 :3.1.-.el7 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

================================================================================

 Package         Arch             Version                  Repository      Size

================================================================================

Installing:

 rsync           x86_64           3.1.-.el7              base            k

Transaction Summary

================================================================================

Install   Package

Total download size:  k

Installed size:  k

Downloading packages:

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

  Installing : rsync-3.1.-.el7.x86_64                                     /

  Verifying  : rsync-3.1.-.el7.x86_64                                     / 

Installed:

  rsync.x86_64 :3.1.-.el7                                                    

Complete!

命令执行成功

============= s103 yum -y install rsync ============

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirrors.aliyun.com

 * extras: mirrors.huaweicloud.com

 * updates: mirrors.huaweicloud.com

Resolving Dependencies

--> Running transaction check

---> Package rsync.x86_64 :3.1.-.el7 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

================================================================================

 Package         Arch             Version                  Repository      Size

================================================================================

Installing:

 rsync           x86_64           3.1.-.el7              base            k

Transaction Summary

================================================================================

Install   Package

Total download size:  k

Installed size:  k

Downloading packages:

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

  Installing : rsync-3.1.-.el7.x86_64                                     /

  Verifying  : rsync-3.1.-.el7.x86_64                                     / 

Installed:

  rsync.x86_64 :3.1.-.el7                                                    

Complete!

命令执行成功

============= s104 yum -y install rsync ============

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirrors.aliyun.com

 * extras: mirrors.huaweicloud.com

 * updates: mirrors.huaweicloud.com

Resolving Dependencies

--> Running transaction check

---> Package rsync.x86_64 :3.1.-.el7 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

================================================================================

 Package         Arch             Version                  Repository      Size

================================================================================

Installing:

 rsync           x86_64           3.1.-.el7              base            k

Transaction Summary

================================================================================

Install   Package

Total download size:  k

Installed size:  k

Downloading packages:

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

  Installing : rsync-3.1.-.el7.x86_64                                     /

  Verifying  : rsync-3.1.-.el7.x86_64                                     / 

Installed:

  rsync.x86_64 :3.1.-.el7                                                    

Complete!

命令执行成功

[root@s101 yinzhengjie]#

批量安装rsync服务：“[root@s101 yinzhengjie]# sh xcall.sh "yum -y install rsync"”

#!/bin/bash

#@author :yinzhengjie

#blog:http://www.cnblogs.com/yinzhengjie

#EMAIL:y1053419035@qq.com

#判断用户是否传参

if [ $# -lt  ];then

    echo "请输入参数";

    exit

fi

#获取文件路径

file=$@

#获取子路径

filename=`basename $file`

#获取父路径

dirpath=`dirname $file`

#获取完整路径

cd $dirpath

fullpath=`pwd -P`

#同步文件到DataNode

for (( i=;i<=;i++ ))

do

    #使终端变绿色

    tput setaf

    echo =========== s$i %file ===========

    #使终端变回原来的颜色，即白灰色

    tput setaf

    #远程执行命令

    rsync -lr $filename `whoami`@s$i:$fullpath

    #判断命令是否执行成功

    if [ $? ==  ];then

        echo "命令执行成功"

    fi

done

3>.将自定义脚本添加执行权限并链接到PATH环境变量中

[yinzhengjie@s101 ~]$ su

Password:

[root@s101 yinzhengjie]# ll

total

drwxrwxr-x.  yinzhengjie yinzhengjie         May  : hadoop

-rw-r--r--.  root        root         Aug    hadoop-2.7..tar.gz

-rw-r--r--.  root        root         May    jdk-8u131-linux-x64.tar.gz

-rw-rw-r--.  yinzhengjie yinzhengjie        May  : xcall.sh

-rw-rw-r--.  yinzhengjie yinzhengjie        May  : xrsync.sh

[root@s101 yinzhengjie]# chmod  xcall.sh xrsync.sh

[root@s101 yinzhengjie]# ll

total

drwxrwxr-x.  yinzhengjie yinzhengjie         May  : hadoop

-rw-r--r--.  root        root         Aug    hadoop-2.7..tar.gz

-rw-r--r--.  root        root         May    jdk-8u131-linux-x64.tar.gz

-rwxr-xr-x.  yinzhengjie yinzhengjie        May  : xcall.sh

-rwxr-xr-x.  yinzhengjie yinzhengjie        May  : xrsync.sh

[root@s101 yinzhengjie]#

[root@s101 yinzhengjie]# cp xcall.sh  xrsync.sh /usr/local/bin/

[root@s101 yinzhengjie]# xcall.sh "ip addr | grep global "

============= s101 ip addr | grep global ============

    inet 172.16.30.101/ brd 172.16.30.255 scope global eno16777736

命令执行成功

============= s102 ip addr | grep global ============

    inet 172.16.30.102/ brd 172.16.30.255 scope global eno16777736

命令执行成功

============= s103 ip addr | grep global ============

    inet 172.16.30.103/ brd 172.16.30.255 scope global eno16777736

命令执行成功

============= s104 ip addr | grep global ============

    inet 172.16.30.104/ brd 172.16.30.255 scope global eno16777736

命令执行成功

[root@s101 yinzhengjie]#

五.启动服务并验证是否成功

1>.使用自定义脚本同步配置文件

[yinzhengjie@s101 ~]$ xrsync.sh /soft/hadoop/etc/full/

=========== s102 %file ===========

命令执行成功

=========== s103 %file ===========

命令执行成功

=========== s104 %file ===========

命令执行成功

[yinzhengjie@s101 ~]$

2>.修改s102-s104的符号链接

[yinzhengjie@s101 ~]$ xcall.sh "ln -sfT /soft/hadoop/etc/full/ /soft/hadoop/etc/hadoop"

============= s101 ln -sfT /soft/hadoop/etc/full/ /soft/hadoop/etc/hadoop ============

命令执行成功

============= s102 ln -sfT /soft/hadoop/etc/full/ /soft/hadoop/etc/hadoop ============

命令执行成功

============= s103 ln -sfT /soft/hadoop/etc/full/ /soft/hadoop/etc/hadoop ============

命令执行成功

============= s104 ln -sfT /soft/hadoop/etc/full/ /soft/hadoop/etc/hadoop ============

命令执行成功

[yinzhengjie@s101 ~]$

3>.格式化文件系统

[yinzhengjie@s101 ~]$ hdfs namenode -format

// :: INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = s101/172.16.30.101

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 2.7.3

STARTUP_MSG:   classpath = /soft/hadoop-2.7.3/etc/hadoop:/soft/hadoop-2.7.3/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/stax-api-1.0-2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/activation-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jets3t-0.9.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/httpclient-4.2.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/httpcore-4.2.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-lang-2.6.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-configuration-1.6.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-digester-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/avro-1.7.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/paranamer-2.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-compress-1.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/xz-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/gson-2.2.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/hadoop-auth-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/zookeeper-3.4.6.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/curator-framework-2.7.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/curator-client-2.7.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jsch-0.1.42.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/junit-4.11.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/hamcrest-core-1.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/mockito-all-1.8.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/hadoop-annotations-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/guava-11.0.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jsr305-3.0.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-cli-1.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-math3-3.1.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/xmlenc-0.52.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-httpclient-3.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-logging-1.1.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-codec-1.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-net-3.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-collections-3.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/servlet-api-2.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jetty-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jetty-util-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jsp-api-2.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jersey-json-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jettison-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3-tests.jar:/soft/hadoop-2.7.3/share/hadoop/common/hadoop-nfs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/guava-11.0.2.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-2.7.3-tests.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-lang-2.6.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/guava-11.0.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-cli-1.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/activation-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/xz-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/servlet-api-2.5.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-codec-1.4.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-client-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/guice-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/javax.inject-1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/aopalliance-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-json-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jettison-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jetty-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-api-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-client-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-registry-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/xz-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/guice-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/javax.inject-1.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/junit-4.11.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/contrib/capacity-scheduler/*.jar

STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff; compiled by 'root' on 2016-08-18T01:41Z

STARTUP_MSG:   java = 1.8.0_131

************************************************************/

// :: INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]

// :: INFO namenode.NameNode: createNameNode [-format]

Formatting using clusterid: CID-4be4a490-3e1b--b613-d37bb9f15790

// :: INFO namenode.FSNamesystem: No KeyProvider found.

// :: INFO namenode.FSNamesystem: fsLock is fair:true

// :: INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=

// :: INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true

// :: INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to :::00.000

// :: INFO blockmanagement.BlockManager: The block deletion will start around  May  ::

// :: INFO util.GSet: Computing capacity for map BlocksMap

// :: INFO util.GSet: VM type       = -bit

// :: INFO util.GSet: 2.0% max memory  MB = 17.8 MB

// :: INFO util.GSet: capacity      = ^ =  entries

// :: INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false

// :: INFO blockmanagement.BlockManager: defaultReplication         =

// :: INFO blockmanagement.BlockManager: maxReplication             =

// :: INFO blockmanagement.BlockManager: minReplication             =

// :: INFO blockmanagement.BlockManager: maxReplicationStreams      =

// :: INFO blockmanagement.BlockManager: replicationRecheckInterval =

// :: INFO blockmanagement.BlockManager: encryptDataTransfer        = false

// :: INFO blockmanagement.BlockManager: maxNumBlocksToLog          =

// :: INFO namenode.FSNamesystem: fsOwner             = yinzhengjie (auth:SIMPLE)

// :: INFO namenode.FSNamesystem: supergroup          = supergroup

// :: INFO namenode.FSNamesystem: isPermissionEnabled = true

// :: INFO namenode.FSNamesystem: HA Enabled: false

// :: INFO namenode.FSNamesystem: Append Enabled: true

// :: INFO util.GSet: Computing capacity for map INodeMap

// :: INFO util.GSet: VM type       = -bit

// :: INFO util.GSet: 1.0% max memory  MB = 8.9 MB

// :: INFO util.GSet: capacity      = ^ =  entries

// :: INFO namenode.FSDirectory: ACLs enabled? false

// :: INFO namenode.FSDirectory: XAttrs enabled? true

// :: INFO namenode.FSDirectory: Maximum size of an xattr:

// :: INFO namenode.NameNode: Caching file names occuring more than  times

// :: INFO util.GSet: Computing capacity for map cachedBlocks

// :: INFO util.GSet: VM type       = -bit

// :: INFO util.GSet: 0.25% max memory  MB = 2.2 MB

// :: INFO util.GSet: capacity      = ^ =  entries

// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033

// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes =

// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     =

// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets =

// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users =

// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = ,,

// :: INFO namenode.FSNamesystem: Retry cache on namenode is enabled

// :: INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is  millis

// :: INFO util.GSet: Computing capacity for map NameNodeRetryCache

// :: INFO util.GSet: VM type       = -bit

// :: INFO util.GSet: 0.029999999329447746% max memory  MB = 273.1 KB

// :: INFO util.GSet: capacity      = ^ =  entries

// :: INFO namenode.FSImage: Allocated new BlockPoolId: BP--172.16.30.101-

// :: INFO common.Storage: Storage directory /home/yinzhengjie/hadoop/dfs/name has been successfully formatted.

// :: INFO namenode.FSImageFormatProtobuf: Saving image file /home/yinzhengjie/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression

// :: INFO namenode.FSImageFormatProtobuf: Image file /home/yinzhengjie/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size  bytes saved in  seconds.

// :: INFO namenode.NNStorageRetentionManager: Going to retain  images with txid >=

// :: INFO util.ExitUtil: Exiting with status

// :: INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at s101/172.16.30.101

************************************************************/

[yinzhengjie@s101 ~]$ echo $?

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ hdfs namenode -format

4>.启动hadoop

[yinzhengjie@s101 ~]$ start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

Starting namenodes on [s101]

s101: starting namenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-namenode-s101.out

s102: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s102.out

s104: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s104.out

s103: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s103.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-secondarynamenode-s101.out

starting yarn daemons

starting resourcemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-resourcemanager-s101.out

s103: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s103.out

s104: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s104.out

s102: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s102.out

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ start-all.sh

5>.用自定义脚本验证NameNode和DataNode是否已经正常启动

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 NameNode

 ResourceManager

 Jps

 SecondaryNameNode

命令执行成功

============= s102 jps ============

 NodeManager

 DataNode

 Jps

命令执行成功

============= s103 jps ============

 NodeManager

 DataNode

 Jps

命令执行成功

============= s104 jps ============

 Jps

 NodeManager

 DataNode

命令执行成功

[yinzhengjie@s101 ~]$

6>.上传文件到服务器

[yinzhengjie@s101 ~]$ netstat -tnlp | grep "java"

(Not all processes could be identified, non-owned process info

 will not be shown, you would have to be root to see it all.)

tcp               0.0.0.0:           0.0.0.0:*               LISTEN      /java

tcp               172.16.30.101:      0.0.0.0:*               LISTEN      /java

tcp               0.0.0.0:           0.0.0.0:*               LISTEN      /java

tcp6              172.16.30.101:      :::*                    LISTEN      /java

tcp6              172.16.30.101:      :::*                    LISTEN      /java

tcp6              172.16.30.101:      :::*                    LISTEN      /java

tcp6              172.16.30.101:      :::*                    LISTEN      /java

tcp6              172.16.30.101:      :::*                    LISTEN      /java

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ hdfs dfs -put xcall.sh xrsync.sh /

[yinzhengjie@s101 ~]$ hdfs dfs -ls  /

Found  items

-rw-r--r--    yinzhengjie supergroup         -- : /xcall.sh

-rw-r--r--    yinzhengjie supergroup         -- : /xrsync.sh

[yinzhengjie@s101 ~]$

7>.客户端检查是否可用正常访问Hadoop的UI界面

五.Hadoop常用参数介绍

　　Hadoop有很多参数，其默认配置大多数仅适用于standalone模式，虽然大多情况下在完全分布式(Fully distributed)模式中也没有问题，但距最优化的运行模式去相去甚远。在生产环境中通常需要调整的参数如下：

>.dfs.name.dir

　　作用：NameNode节点用于存储HDFS元数据的本地目录，官方建议为/home/hadoop/dfs/name；

>.dfs.data.dir

　　作用：DataNode节点用于存储HDFS文件数据块的本地目录，官方建议为/home/hadoop/dfs/data；

>.mapred.system.dir

　　作用：HDFS中用于存储共享的MapReduce系统文件的目录，官方建议为/hadoop/mapred/system；

>.mapred.local.dir

　　作用：TaskNode节点用于存储临时数据的本地文件目录；

>.mapred.tasktracker.{map|reduce}.tarks.maximum

　　作用：在TaskTracker上可同时运行的的map或reduce任务的最大数目；

>.hadoop.tmp.dir

　　作用： Hadoop临时目录；

>.mapred.child.java.opts

　　作用：每个子任务可申请使用的heap大小；官方建议为-Xmx512m；

>.mapred.reduce.tasks

　　作用：每任务的reduce数量；

　　上述参数中的大多数都可以接受多个以逗号分隔的目录，尤其是对于dfs.name.dir来说，多个目录还可以达到冗余的目的；而对于拥有多块磁盘的DataNode，为其dfs.data.dir指定多个值可以存储数据于多个磁盘，并能通过并行加速I/O操作。为mapred.local.dir指定多个眼光也能起到一定的加速作用。

　　此外，hadoop.tmp.dir对于不同的用户来说其路径是不相同的，事实上，应该尽量避免让此路径依赖用户属性，比如可以放在一个公共位置让所有用户都可以方便地访问到。在Linux系统下，hadoop.tmp.dir的默认路径指向了/tmp，这的确是一个公共位置，但/tmp目录所在的文件系统大多数都有使用配额，而且空间也通常比较有限，因此，故此此路径殊非理想之所在。建议将其指向一个有着足够空间的文件系统上的目录。

　　默认配置中，Hadoop可以在每个TaskTracker上运行四个任务(两个map任务，两个reduce任务)，这可以通过mapred.tasktracker.{map|reduce}.tarks.maximum进行配置，通常建议其个数为与CPU核心数目相同或者为CPU核心数目的2倍，但其最佳值通常依赖于诸多因素，而在CPU密集型的应用场景中也不应该将其最大数目设置得过高。除了CPU之外，还应该考虑每个任务所能够使用的的heap空间大小所带来的影响；默认情况下，Hadoop为每个任务指定了200MB的heap空间上限，由于每个job可能会申请使用很大的heap，因此，过高的设定可能会带来意外的结果。

　　每个MapReduce任务可以通过mapred.reduce.tasks配置其运行的reduce任务数，通常也应该为其指定一个在多数场景下都能工作良好的默认值，比如Hadoop默认将此数目指定为1个，这对大多数任务来讲都有着不错的性能表现。而实际使用中，一般建议将此值设定为当前Hadoop集群可以运行的reduce任务总数的0.95倍或1.75倍。0.95这个因子意味着Hadoop集群可以立即启动所有的reduce任务并在map任务完成时接收数据并进行处理，而1.75则意味着先启动部分reduce任务，执行速度快的节点上的reduce完成后可再启动一个新的reduce任务，而速度慢的节点则无须执行此类操作。这会带来较为理想的负载均衡效果。

Hadoop部署方式-完全分布式(Fully-Distributed Mode)的更多相关文章

  1. Hadoop部署方式-伪分布式(Pseudo-Distributed Mode)

    Hadoop部署方式-伪分布式(Pseudo-Distributed Mode) 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.下载相应的jdk和Hadoop安装包 JDK:h ...

  2. Hadoop部署方式-本地模式(Local (Standalone) Mode)

    Hadoop部署方式-本地模式(Local (Standalone) Mode) 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. Hadoop总共有三种运行方式.本地模式(Local ...

  3. Hadoop部署方式-高可用集群部署(High Availability)

    版权声明:原创作品,谢绝转载!否则将追究法律责任. 本篇博客的高可用集群是建立在完全分布式基础之上的,详情请参考:https://www.cnblogs.com/yinzhengjie/p/90651 ...

  4. Hadoop学习------Hadoop安装方式之(三)：分布式部署

    这里为了方便直接将单机部署过的虚拟机直接克隆,当然也可以不这样做,一个个手工部署. 创建完整克隆——>下一步——>安装位置.等待一段时间即可. 我这边用了三台虚拟机,分别起名master, ...

  5. Apache Spark探秘：三种分布式部署方式比较

    转自:链接地址: http://dongxicheng.org/framework-on-yarn/apache-spark-comparing-three-deploying-ways/     目 ...

  6. Hadoop 2.6.0分布式部署參考手冊

    Hadoop 2.6.0分布式部署參考手冊 关于本參考手冊的word文档.能够到例如以下地址下载:http://download.csdn.net/detail/u012875880/8291493 ...

  7. Hadoop生态圈-zookeeper完全分布式部署

    Hadoop生态圈-zookeeper完全分布式部署 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 本篇博客部署是建立在Hadoop高可用基础之上的,关于Hadoop高可用部署请参 ...

  8. Hadoop学习------Hadoop安装方式之(二)：伪分布部署

    要想发挥Hadoop分布式.并行处理的优势,还须以分布式模式来部署运行Hadoop.单机模式是指Hadoop在单个节点上以单个进程的方式运行,伪分布模式是指在单个节点上运行NameNode.DataN ...

  9. Hadoop生态圈-phoenix完全分布式部署以及常用命令介绍

    Hadoop生态圈-phoenix完全分布式部署 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. phoenix只是一个插件,我们可以用hive给hbase套上一个JDBC壳,但是你 ...

随机推荐

  1. Scrum Meeting 11.1

    成员 今日任务 明日计划 用时 徐越 学习利用servlet上传下载文件 代码迁移 4h 赵庶宏 数据库的连接及代码学习 数据库连接 2h 武鑫 设计界面;尝试写一些初步的代码,独立完成一些简单界面 ...

  2. No.1010_第七次团队会议

    渺茫的前景 今天大家都很失望,一来昨天的问题还是继续存在着,仍然没有完成.二来,我们看了一下其余几个组的界面,对自己有些难过. 我们组确实存在人手少的问题,这几天我还因为挑战杯的事情缺席了两天,感觉内 ...

  3. 1001 A+B

    代码链接 PDF链接 首先要说的是这道题的难点是如何把数字输出加入逗号,毕竟数据范围并没有超过Long.当然这个难点也不是问题,将数字转为字符串,C中就有这样的函数,然后再用 %3==0 这样来控制输 ...

  4. wcf的DataContractAttribute与DataMenmberAttribute

    文章:序列化和反序列化的几种方式(DataContractSerializer)(二) 介绍了序列化控制细节.哪些字段可以序列化,序列化后这些字段的名字.

  5. OSG学习：裁剪变换（2）

    接着上一篇博客说. 还有一种裁剪的方法:osg::Scissor类. 这个类封装了OpenGL中的glScissor()函数. 该类主要用于设置一个视口裁剪平面举行.设置裁剪平面举行的函数如下: vo ...

  6. 解决getOutputStream() has alerady been called for this response

    在用tomcat启动一个web项目(SpringBoot)的时候报错: getOutputStream() has alerady been called for this response 但是如果 ...

  7. linux安全配置学习

    参考摘自https://www.cnblogs.com/hiccup/p/4300963.html 1.关闭icmp请求 #vm虚拟机是130地址,通过echo 1 > /proc/sys/ne ...

  8. 虚拟机VMware中的CentOS字符界面和图形界面切换

    在虚拟机中安装CentOS后展示的界面是图形用户界面,想切换到命令模式,于是查了linux下切换模式的方法,可是按了ctrl+alt+f1.f2....f7都是没用,后来发现是因为按键冲突的原因 问题 ...

  9. （二）Jmeter各部件的作用

    JMeter主要组件介绍 1.测试计划(Test Plan)是使用 JMeter 进行测试的起点,它是其它 JMeter 测试元件的容器. 2.线程组(Thread Group)代表一定数量的并发用户 ...

  10. linux 搭建epel本地库,并定时同步

    1.安装rsyncyum -y install rsync.x86_64 2.同步epel至本地#http://mirrors.ustc.edu.cn/status/ 获取镜像库rsync路径mkdi ...