Linux 搭建Hadoop集群错误锦集

一、Hadoop集群配置好后，执行start-dfs.sh后报错，一堆permission denied

zf sbin $ ./start-dfs.sh
Starting namenodes on [master]
master: chown: changing ownership of '/home/zf/hadoop/hadoop-2.9.1/logs': Operation not permitted
master: starting namenode, logging to /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-namenode-master.out
master: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 159: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-namenode-master.out: Permission denied
master: head: cannot open '/home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-namenode-master.out' for reading: No such file or directory
master: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 177: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-namenode-master.out: Permission denied
master: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 178: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-namenode-master.out: Permission denied
slave-1: chown: changing ownership of '/home/zf/hadoop/hadoop-2.9.1/logs': Operation not permitted
slave-1: starting datanode, logging to /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-1.out
slave-1: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 159: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-1.out: Permission denied
slave-2: chown: changing ownership of '/home/zf/hadoop/hadoop-2.9.1/logs': Operation not permitted
slave-2: starting datanode, logging to /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-2.out
slave-2: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 159: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-2.out: Permission denied
slave-1: head: cannot open '/home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-1.out' for reading: No such file or directory
slave-1: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 177: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-1.out: Permission denied
slave-1: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 178: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-1.out: Permission denied
slave-2: head: cannot open '/home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-2.out' for reading: No such file or directory
slave-2: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 177: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-2.out: Permission denied
slave-2: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 178: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-2.out: Permission denied
Starting secondary namenodes [slave-1]
slave-1: chown: changing ownership of '/home/zf/hadoop/hadoop-2.9.1/logs': Operation not permitted
slave-1: starting secondarynamenode, logging to /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-secondarynamenode-slave-1.out
slave-1: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 159: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-secondarynamenode-slave-1.out: Permission denied
slave-1: head: cannot open '/home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-secondarynamenode-slave-1.out' for reading: No such file or directory
slave-1: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 177: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-secondarynamenode-slave-1.out: Permission denied
slave-1: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 178: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-secondarynamenode-slave-1.out: Permission denied

解决方案：对hadoop安装目录执行命令：sudo chmod a+w * 
对文件敞开权限

二、执行./start-dfs.sh 和 ./start-yarn.sh 后master主机上jps 无法启动NameNode和ResourceManager，但slave-1 和slave-2 DataNode启动正常。

查看logs，错误如下：

**************namenode.log****************************

-- ::, ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.net.BindException: Port in use: master:
    at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:)
    at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:)
    at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:)
    at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:)
Caused by: java.net.BindException: Cannot assign requested address
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:)
    at sun.nio.ch.Net.bind(Net.java:)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:)
    at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:)
    at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:)
    at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:)
    ...  more
-- ::, INFO org.apache.hadoop.util.ExitUtil: Exiting with status : java.net.BindException: Port in use: master:
-- ::, INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/182.61.39.233
************************************************************/

*************secondnamemode.log****************************
-- ::, INFO org.apache.hadoop.http.HttpServer2: HttpServer.start() threw a non Bind IOException
java.net.BindException: Port in use: master:
    at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:)
    at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:)
    at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:)
    at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:)
    at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.startInfoServer(SecondaryNameNode.java:)
    at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:)
Caused by: java.net.BindException: Cannot assign requested address
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:)
    at sun.nio.ch.Net.bind(Net.java:)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:)
    at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:)
    at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:)
    at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:)
    ...  more
-- ::, FATAL org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Failed to start secondary namenode
java.net.BindException: Port in use: master:
    at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:)
    at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:)
    at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:)
    at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:)
    at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.startInfoServer(SecondaryNameNode.java:)
    at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:)
Caused by: java.net.BindException: Cannot assign requested address
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:)
    at sun.nio.ch.Net.bind(Net.java:)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:)
    at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:)
    at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:)
    at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:)
    ...  more

可见主要原因就是：

1、 java.net.BindException: Port in use: master:50070/9001

2、Caused by: java.net.BindException: Cannot assign requested address

端口被占用是直接原因，但起因是不能分配所需的地址，跟地址有关的就联想到 /etc/hosts文件

原来我的是：

127.0.0.1 localhost localhost.localdomain
182.xx.xx.33 master (外网IP， 主机名）
106.xx.xx.72 slave-
106.xx.xx.73 slave-

后来把外网IP全部换成内网IP后就可以了

127.0.0.1 localhost localhost.localdomain
172.xx.x.2 master
172.xx.x.5 slave-
172.xx.x.6 slave-

原因我也不明，我三台主机都是使用百度云服务器。

三、Hadoop 配置好后，发现DataNode没有启动，很大原因是因为NameNode 格式化时与上次的残留有冲突，所以要先删除上次的残留。或者每次使用完后及时Hadoop关闭，重新开启后不要重新格式化NameNode。