Ubuntu 14.10 下Hadoop 错误集

1 FATAL org.apache.hadoop.ha.ZKFailoverController: Unable to start failover controller. Parent znode does not exist.

　　这个错误导致启动不了DFSZKFailoverController，从而不能选举出Active Node，导致了Hadoop两个NameNode都是Standby，我是这样做的

　　停掉Hadoop所有进程，然后重新格式化Zookeeper

hdfs zkfc -formatZK

2 紧接着上个问题，再重新格式化过zookeeper之后，发现yarn启动不了了

2015-08-05 19:00:33,718 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore: Exception while executing a ZK operation.
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
    at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:949)
    at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:915)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:937)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:934)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1076)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1095)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doMultiWithRetries(ZKRMStateStore.java:934)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doMultiWithRetries(ZKRMStateStore.java:948)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.removeRMDTMasterKeyState(ZKRMStateStore.java:844)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.removeRMDTMasterKey(RMStateStore.java:733)
    at org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager.removeStoredMasterKey(RMDelegationTokenSecretManager.java:99)
    at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.removeExpiredKeys(AbstractDelegationTokenSecretManager.java:371)
    at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.rollMasterKey(AbstractDelegationTokenSecretManager.java:348)
    at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover.run(AbstractDelegationTokenSecretManager.java:646)
    at java.lang.Thread.run(Thread.java:745)
2015-08-05 19:00:33,718 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore: Maxed out ZK retries. Giving up!
2015-08-05 19:00:33,718 INFO org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager: removing master key with keyID 55
2015-08-05 19:00:33,738 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore: Exception while executing a ZK operation.
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
    at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:949)
    at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:915)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:937)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:934)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1076)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1095)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doMultiWithRetries(ZKRMStateStore.java:934)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doMultiWithRetries(ZKRMStateStore.java:948)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.removeRMDTMasterKeyState(ZKRMStateStore.java:844)
    at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.removeRMDTMasterKey(RMStateStore.java:733)
    at org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager.removeStoredMasterKey(RMDelegationTokenSecretManager.java:99)
    at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.removeExpiredKeys(AbstractDelegationTokenSecretManager.java:371)
    at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.rollMasterKey(AbstractDelegationTokenSecretManager.java:348)
    at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover.run(AbstractDelegationTokenSecretManager.java:646)
    at java.lang.Thread.run(Thread.java:745)

但是这个问题，估计还是因为zookeeper造成的，但是我没任何修改，从新启动yarn。。。竟然成功了

start-yarn.sh

3 NameNode启动不了

java.lang.IllegalArgumentException: Unable to construct journal, qjournal://spark-1421-0000:8485;spark-1421-0003:8485;spark-1421-0004:8485;spark-1421-0005:8485;spark-1421-0006:8485/hadoop-journal
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1593)
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:276)
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initSharedJournalsForRead(FSEditLog.java:254)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:776)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:621)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1020)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:739)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:536)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:595)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:762)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:746)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1438)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1504)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1591)
    ... 13 more
Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel
    at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel.<init>(IPCLoggerChannel.java:146)
    at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$1.createLogger(IPCLoggerChannel.java:156)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:367)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:149)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:116)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:105)
    ... 18 more

我这里的错误关键是：

tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel

后来排查，发现是因为：com.google.common.base.Stopwatch这个类造成的，因为这个类在guava.jar中，但是这个jar包在Hadoop 中是11.0.2，但是在java_home/jre/lib/ext中还有一个是18.*，那么造成了版本不一致，所以我的解决办法是：删掉java_home/jre/lib/ext中的guava.jar

4 Hadoop启动报Error: JAVA_HOME is not set and could not be found解决办法

是因为etc/hadoop配置文件hadoop-env.sh中Javahome没有配置，重新配置JAVA_HOME的绝对路径后即可

5 Hadoop操作文件报Permission denied: user=dr.who, access=READ_EXECUTE, inode="/tmp":root:supergroup:drwx------

修改一下权限

hdfs dfs -chmod -R 755 /tmp