CentOS 7安装Hadoop集群

准备三台虚拟机，ip分别为192.168.220.10（master）、192.168.220.11（slave1）、192.168.220.12（slave2）

准备好jdk-6u45-linux-x64.bin和hadoop-1.2.1-bin.tar.gz，放在/usr/local/src/目录下

安装JDK（每台虚拟机都安装）

1.进入到/usr/local/src/目录，执行./jdk-6u45-linux-x64.bin

2.修改~/.bashrc，在文件末尾增加三行

export JAVA_HOME=/usr/local/src/jdk1.6.0_45
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin

　　

3.使环境变量生效，执行source ~/.bashrc

安装Hadoop

在192.168.220.10机器上安装hadoop

1.进入到/usr/local/src/目录，解压hadoop-1.2.1-bin.tar.gz，执行tar -zxf hadoop-1.2.1-bin.tar.gz

2.修改配置文件

masters文件

master

　　

slaves文件

slave1
slave2

　　

core-site.xml文件

<configuration>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/usr/local/src/hadoop-1.2.1/tmp</value>
	</property>
	<property>
		<name>fs.default.name</name>
		<value>hdfs://192.168.220.10:9000</value>
	</property>
</configuration>

　　

mapred-site.xml文件

<configuration>
	<property>
		<name>mapred.job.tracker</name>
		<value>http://192.168.220.10:9001</value>
	</property>
</configuration>

　　

hdfs-site.xml文件

<configuration>
	<property>
		<name>dfs.replication</name>
		<value>3</value>
	</property>
</configuration>

　　

hadoop-env.sh文件，在后面添加一行

export JAVA_HOME=/usr/local/src/jdk1.6.0_45

　　

3.将/usr/local/src/hadoop-1.2.1目录拷贝到192.168.220.11、192.168.220.12机器上

配置hostname

配置192.168.220.10的主机名为master

1.执行hostname master

2.修改/etc/hostname文件

master

　　

修改192.168.220.11的主机名为slave1，修改192.168.220.12的主机名为slave2

配置host文件

三台机器的/etc/hosts文件末尾添加以下代码

192.168.220.10    master
192.168.220.11    slave1
192.168.220.12    slave2

关闭防火墙和SELinux

分别在每台机器上执行

systemctl stop firewalld.service
setenforce 

配置SSH

1.在192.168.220.10上执行ssh-keygen，在~目录下新增.ssh目录，目录中的文件为id_rsa，id_rsa.pub

2.将id_rsa.pub拷贝为authorized_keys

cp id_rsa.pub authorized_keys

3.在192.168.220.11和192.168.220.12上分别执行ssh-keygen

4.将192.168.220.11和192.168.220.12上id_rsa.pub的内容分别拷贝到192.168.220.10的authorized_keys文件中，如下：

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9mGRhFOdcoHw9GUnKQmqThNKpsyah93Dtq/d8RICGWIHDRJ3GXd0sEcb743ejwbuCMmtlhheXcU0FuyA6Cm0jvMyvDfaPKArtxl6KT7Z93uC0VDCXDRomueux81HAIVjc7ZqlXwVeYs1LITxEeJykKlFOXvK7JexWhWGdMMADwxbFMbaNsZ9EwRxcFLFtNg65FQ+u8CIV9KR3D02kemwLCsP+xiRcgs+wirQPm5JM+2cJoLsVQBz3Hk335IsEhc1Xb9Cralo8Tt8gh/ho8K/1pVjvyW1b0LkP9HGNdwVYD9wkWdEJRkryLXBEXpjk4xu+riF+N4rOzJD root@master
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn79fdfR/NjzPVD3NPj1vBBfQdVOrv7jeb4UJCOsd7xioPRiz8gOQnOmhu5C+GchbyGA+tg5pXwnNJTOO2wn32U4lOPndW0okN/wqyN4vgq/taJi7JgY/8rneBiGaIIdNIy/pAGlMwb53Qn766adetMhsxYMD2l4uxmbVVjzCRb8QP5EsAYTmmFOODzJsPm70uF3j1Q8zGavYg0wFSYR/yECQns4DBSuBJNxdGY6PskBXqurahwi5yaR3vWV1Ix4wtB6BYuQomEnGdzOSfrBMZ/yc5tXo0xmEfY7wFkize6z9Pm2E3oDoMR18YkwT1Cz6fHikVILA9cldtL root@slave1
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCydYCvASzCZggks4hMqOcYSGLO2eAvocWezNOMwspTfpJ105Jumb/vf5h6cRZeckq56IvhSV6t6mytk4pZoZjjZPSmWvCwLtMRMPShNbA3BYtj5V3WRKV8ZcMrNdD//U7iHHoJm57vI/m+XO42YSYjPw7JDkb8Ij9b6zgI3fyvbSSYeXb451PlyJLHdxIzRMAaZDSbAML9e7EO8VJB9Wf9bXpow4+VeP33it3kgMNUlHQtyqduSwYGxVVtGsUTJkxnuRsbWeeA1/pp8MNFKUgBTMALTVHByglgZqwGcbblJxsG832PIZNRECIFqorm6odftjnT4DR7/0yR root@slave2

5.将192.168.220.10的authorized_keys文件拷贝到192.168.220.11、192.168.220.12机器上

配置完成后三台机器可以互相访问不用密码

好了，到这里Hadoop集群就配置完了，让我们来使用下吧

在192.168.220.10上格式化namenode

进入到/usr/local/src/hadoop-1.2.1/bin目录，执行

./hadoop namenode -format

出现

// :: INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master/192.168.220.10
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.2.1
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG:   java = 1.6.0_45
************************************************************/
// :: INFO util.GSet: Computing capacity for map BlocksMap
// :: INFO util.GSet: VM type       = -bit
// :: INFO util.GSet: 2.0% max memory =
// :: INFO util.GSet: capacity      = ^ =  entries
// :: INFO util.GSet: recommended=, actual=
// :: INFO namenode.FSNamesystem: fsOwner=root
// :: INFO namenode.FSNamesystem: supergroup=supergroup
// :: INFO namenode.FSNamesystem: isPermissionEnabled=true
// :: INFO namenode.FSNamesystem: dfs.block.invalidate.limit=
// :: INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval= min(s), accessTokenLifetime= min(s)
// :: INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length =
// :: INFO namenode.NameNode: Caching file names occuring more than  times
// :: INFO common.Storage: Image file /usr/local/src/hadoop-1.2./tmp/dfs/name/current/fsimage of size  bytes saved in  seconds.
// :: INFO namenode.FSEditLog: closing edit log: position=, editlog=/usr/local/src/hadoop-1.2./tmp/dfs/name/current/edits
// :: INFO namenode.FSEditLog: close success: truncate to , editlog=/usr/local/src/hadoop-1.2./tmp/dfs/name/current/edits
// :: INFO common.Storage: Storage directory /usr/local/src/hadoop-1.2./tmp/dfs/name has been successfully formatted.
// :: INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.220.10
************************************************************/

用jps查看进程

[root@master bin]# jps
 JobTracker
 NameNode
 SecondaryNameNode
 Jps

在192.168.220.11查看进程

 DataNode
 Jps
 TaskTracker

在192.168.220.12查看进程

 DataNode
 TaskTracker
 Jps

测试下：

执行./hadoop fs -ls /

drwxr-xr-x   - root supergroup           -- : /usr

上传文件，执行./hadoop fs -put /root/w.txt /

查看文件执行./hadoop fs -cat /w.txt，显示

ddd

成功

注：

问题：上传文件时出现以下错误

[root@master bin]# ./hadoop fs -put /root/a.txt /
// :: WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /a.txt could only be replicated to  nodes, instead of
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:)
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
    at java.lang.reflect.Method.invoke(Method.java:)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)
    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:)
 
    at org.apache.hadoop.ipc.Client.call(Client.java:)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:)
    at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
    at java.lang.reflect.Method.invoke(Method.java:)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:)
    at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$(DFSClient.java:)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:)
 
// :: WARN hdfs.DFSClient: Error Recovery for null bad datanode[] nodes == null
// :: WARN hdfs.DFSClient: Could not get block locations. Source file "/a.txt" - Aborting...
put: java.io.IOException: File /a.txt could only be replicated to  nodes, instead of
// :: ERROR hdfs.DFSClient: Failed to close file /a.txt
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /a.txt could only be replicated to  nodes, instead of
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:)
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
    at java.lang.reflect.Method.invoke(Method.java:)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)
    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:)
 
    at org.apache.hadoop.ipc.Client.call(Client.java:)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:)
    at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
    at java.lang.reflect.Method.invoke(Method.java:)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:)
    at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$(DFSClient.java:)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:)

解决方法：关闭防火墙