hadoop的一个节点unused了。然后重启启动hadoop的服务，仍有有一个hadoop的节点起不来。多次重启hadoop和杀进程之后，发现hadoop的master和slave节点上的状态在切换，没有达到同步起停;当master起来的时候，slave节点上的hadoop就unused,  当slave节点上的hadoop状态为running的时候，master节点上的hadoop节点的状态就是unused.主从没法同步起停。

在数据库入库的时候，日志里报如下错误：

这个时候突然想到了hadoop的主从站点之后的关系是不是没有同步好，文件进入了安全模式。

进入到hadoop的bin目录下，执行退出安全模式的命令，第一次是在hadoop服务停了 之后，执行退出安全模式的命令显示没有连接成功。

然后启动hadoop，这个时候hadoop下面的都运行正常了，执行退出安全模式。

退出安全模式命令： hadoop dfsadmin -safemode leave

[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.library.path=/opt/oracle/app/client/oracle/product/11.2.0/inoc/lib:/lib:/usr/lib:/usr/java/packages/lib/a
md64:/usr/lib64:/lib64:/lib:/usr/lib]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.io.tmpdir=/tmp]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.compiler=<NA>]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.name=Linux]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.arch=amd64]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.version=2.6.32.59-0.9-default]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.name=acrosspm]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.home=/home/acrosspm]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.dir=/opt/netwatcher/pm4h2/work/conf/pmpadmin]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Initiating client connection, connectString=10.215.133.36:15248 sessionTimeout=180000 watcher=hconnection]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [The identifier of this process is 35646@pmapp]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Opening socket connection to server pmweb.site/10.215.133.36:15248. Will not attempt to authenticate
using SASL (unknown error)]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Socket connection established to pmweb.site/10.215.133.36:15248, initiating session]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Session establishment complete on server pmweb.site/10.215.133.36:15248, sessionid = 0x35cdf7a17c400
28, negotiated timeout = 180000]
[2017-06-25 18:04:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:05:31] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:06:31] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:07:31] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:08:31] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:09:31] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:10:34] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:10:54] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:11:56] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:12:59] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:13:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:14:59] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:16:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:17:02] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:18:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:19:05] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:20:05] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:20:25] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:21:30] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:22:33] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:23:33] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:24:36] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:24:56] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:26:01] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:27:10] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:28:13] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:28:33] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:29:38] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:30:41] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:31:41] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:32:44] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:33:47] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:34:50] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:35:50] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:36:50] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:37:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:38:59] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:39:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:40:59] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:41:59] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:42:59] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:44:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:44:04] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 0 of 10 failed; retrying after sleep of 1000]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:05] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 1 of 10 failed; retrying after sleep of 1006]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:06] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 2 of 10 failed; retrying after sleep of 1005]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:07] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 3 of 10 failed; retrying after sleep of 2014]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:09] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 4 of 10 failed; retrying after sleep of 2015]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:11] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 5 of 10 failed; retrying after sleep of 4034]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:15] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 6 of 10 failed; retrying after sleep of 4007]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:19] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 7 of 10 failed; retrying after sleep of 8075]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:27] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 8 of 10 failed; retrying after sleep of 16070]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:43] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 9 of 10 failed; no more retrying.]

hadoop节点挂死的一次分析报表。的更多相关文章

  1. I2C 挂死，SDA一直为低问题分析【转】

    转自:https://blog.csdn.net/winitz/article/details/72460775 版权声明:本文为博主原创文章,未经博主允许不得转载. https://blog.csd ...

  2. 记一次 .NET WPF布草管理系统 挂死分析

    一:背景 1. 讲故事 这几天看的 dump 有点多,有点伤神伤脑,晚上做梦都是dump,今天早上头晕晕的到公司就听到背后同事抱怨他负责的WPF程序挂死了,然后测试的小姑娘也跟着抱怨...嗨,也不知道 ...

  3. 记一次 .NET 某上市工业智造 CPU+内存+挂死 三高分析

    一:背景 1. 讲故事 上个月有位朋友加wx告知他的程序有挂死现象,询问如何进一步分析,截图如下: 看这位朋友还是有一定的分析基础,可能玩的少,缺乏一定的分析经验,当我简单分析之后,我发现这个dump ...

  4. 记一次 .NET 某纺织工厂 MES系统 API 挂死分析

    一:背景 1. 讲故事 这个月中旬,有位朋友加我wx求助他的程序线程占有率很高,寻求如何解决,截图如下: 说实话,和不同行业的程序员聊天还是蛮有意思的,广交朋友,也能扩大自己的圈子,朋友说他因为这个b ...

  5. MySQL 连接为什么挂死了？

    摘要:本次分享的是一次关于 MySQL 高可用问题的定位过程,其中曲折颇多但问题本身却比较有些代表性,遂将其记录以供参考. 一.背景 近期由测试反馈的问题有点多,其中关于系统可靠性测试提出的问题令人感 ...

  6. MySQL 连接为什么挂死了

    声明:本文为博主原创文章,由于已授权部分平台发表该文章(知乎.云社区),可能造成发布时间方面的困扰. 一.背景 近期由测试反馈的问题有点多,其中关于系统可靠性测试提出的问题令人感到头疼,一来这类问题有 ...

  7. 应用程序出现挂死，.NET Runtime at IP 791F7E06 (79140000) with exit code 80131506.

    工具出现挂死问题 1.问题描述 工具出现挂死问题,巡检IIS发现以下异常日志 现网系统日志: 事件类型:    错误 事件来源:    .NET Runtime 描述: Application: Di ...

  8. 关于用strace工具定位vrrpd进程有时会挂死的bug

    只做工作总结备忘之用. 正在烧镜像,稍总结一下进来改bug遇到的问题. 一个项目里要用到L3 switch的nat,vrrp功能,但实地测试中偶然出现write file挂死的情况,但不是必现.交付在 ...

  9. IIC挂死问题解决过程

    0.环境:arm CPU 带有IIC控制器作为slave端,带有调试串口. 1.bug表现:IIC slave 在系统启动后概率挂死,导致master无法detect到slave. 猜测1:认为IIC ...

随机推荐

  1. cocos2d-x 3.0 CCTime 的改动

    在cocos3.0 中,去掉了CCTimer这个类. 我们须要把之前的代码改动为 定义时间的变量是 struct  timeval  start 1.获取时间的方式是: gettimeofday(&a ...

  2. iOS:ShareSDk的分享

    使用分享类的SDK其实有很多,例如友盟.ShareSDK等等,参照他们的文档集成起来并不是很难,可能出的一些问题也就是配置文件的问题,这里我个人使用了ShareSDK分享,具体操作可出现的问题如下: ...

  3. 深度增强学习--DQN的变形

    DQN的变形 double DQN prioritised replay dueling DQN

  4. POJ 1511 链式前向星+SPFA

    #include<iostream> #include<cstdio> #include<cstdlib> using namespace std; const i ...

  5. MyEclipse 2014配置Maven

    1 配置maven (1)下载apache-maven-3.3.3和mvnRespo放在某个路径下,我这里是放在D盘根目录下. (2)修改D:\apache-maven-3.3.3\conf\sett ...

  6. db2 v9.5迁移至v10.5，及遇重名节点数据库无法创建db的解决办法

    同系统同版本可以使用备份恢复,本文前提是不同系统不同版本,使用db2move命令. 1.db2move db db_name export 此处注意,先建个目录放文件,因为文件比较多,如果上来直接ex ...

  7. 【ACM】最小公倍数

    http://acm.hdu.edu.cn/game/entry/problem/show.php?chapterid=2&sectionid=1&problemid=1 #inclu ...

  8. Android模糊演示样例-RenderScript-附效果图与代码

    本文链接    http://blog.csdn.net/xiaodongrush/article/details/31031411 參考链接    Android高级模糊技术    http://s ...

  9. Android中Native和H5交互

    1.概述 时至今日,H5的跨平台性越发凸显优势,一套代码适配android.ios,既能减少开发成本,又便于更新与维护.但是native的性能体验也确实更佳,尤其体现在复杂界面和频繁变化的界面上.事实 ...

  10. 细数Python的特殊方法一：集合类

    1. 特殊方法汇总列表     2. 仿集合类 __len__:返回对象的长度 __getitem__:获得某一项的值 __setitem__:设置莫一项的值 __delitem__删除某一项 __c ...