hadoop的一个节点unused了。然后重启启动hadoop的服务，仍有有一个hadoop的节点起不来。多次重启hadoop和杀进程之后，发现hadoop的master和slave节点上的状态在切换，没有达到同步起停;当master起来的时候，slave节点上的hadoop就unused,  当slave节点上的hadoop状态为running的时候，master节点上的hadoop节点的状态就是unused.主从没法同步起停。

在数据库入库的时候，日志里报如下错误：

这个时候突然想到了hadoop的主从站点之后的关系是不是没有同步好，文件进入了安全模式。

进入到hadoop的bin目录下，执行退出安全模式的命令，第一次是在hadoop服务停了 之后，执行退出安全模式的命令显示没有连接成功。

然后启动hadoop，这个时候hadoop下面的都运行正常了，执行退出安全模式。

退出安全模式命令： hadoop dfsadmin -safemode leave

[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.library.path=/opt/oracle/app/client/oracle/product/11.2.0/inoc/lib:/lib:/usr/lib:/usr/java/packages/lib/a
md64:/usr/lib64:/lib64:/lib:/usr/lib]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.io.tmpdir=/tmp]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.compiler=<NA>]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.name=Linux]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.arch=amd64]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.version=2.6.32.59-0.9-default]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.name=acrosspm]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.home=/home/acrosspm]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.dir=/opt/netwatcher/pm4h2/work/conf/pmpadmin]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Initiating client connection, connectString=10.215.133.36:15248 sessionTimeout=180000 watcher=hconnection]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [The identifier of this process is 35646@pmapp]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Opening socket connection to server pmweb.site/10.215.133.36:15248. Will not attempt to authenticate
using SASL (unknown error)]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Socket connection established to pmweb.site/10.215.133.36:15248, initiating session]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Session establishment complete on server pmweb.site/10.215.133.36:15248, sessionid = 0x35cdf7a17c400
28, negotiated timeout = 180000]
[2017-06-25 18:04:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:05:31] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:06:31] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:07:31] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:08:31] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:09:31] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:10:34] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:10:54] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:11:56] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:12:59] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:13:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:14:59] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:16:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:17:02] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:18:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:19:05] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:20:05] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:20:25] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:21:30] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:22:33] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:23:33] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:24:36] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:24:56] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:26:01] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:27:10] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:28:13] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:28:33] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:29:38] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:30:41] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:31:41] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:32:44] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:33:47] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:34:50] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:35:50] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:36:50] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:37:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:38:59] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:39:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:40:59] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:41:59] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:42:59] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:44:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]
[2017-06-25 18:44:04] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 0 of 10 failed; retrying after sleep of 1000]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:05] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 1 of 10 failed; retrying after sleep of 1006]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:06] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 2 of 10 failed; retrying after sleep of 1005]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:07] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 3 of 10 failed; retrying after sleep of 2014]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:09] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 4 of 10 failed; retrying after sleep of 2015]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:11] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 5 of 10 failed; retrying after sleep of 4034]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:15] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 6 of 10 failed; retrying after sleep of 4007]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:19] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 7 of 10 failed; retrying after sleep of 8075]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:27] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 8 of 10 failed; retrying after sleep of 16070]
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:43] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 9 of 10 failed; no more retrying.]

hadoop节点挂死的一次分析报表。的更多相关文章

  1. I2C 挂死，SDA一直为低问题分析【转】

    转自:https://blog.csdn.net/winitz/article/details/72460775 版权声明:本文为博主原创文章,未经博主允许不得转载. https://blog.csd ...

  2. 记一次 .NET WPF布草管理系统 挂死分析

    一:背景 1. 讲故事 这几天看的 dump 有点多,有点伤神伤脑,晚上做梦都是dump,今天早上头晕晕的到公司就听到背后同事抱怨他负责的WPF程序挂死了,然后测试的小姑娘也跟着抱怨...嗨,也不知道 ...

  3. 记一次 .NET 某上市工业智造 CPU+内存+挂死 三高分析

    一:背景 1. 讲故事 上个月有位朋友加wx告知他的程序有挂死现象,询问如何进一步分析,截图如下: 看这位朋友还是有一定的分析基础,可能玩的少,缺乏一定的分析经验,当我简单分析之后,我发现这个dump ...

  4. 记一次 .NET 某纺织工厂 MES系统 API 挂死分析

    一:背景 1. 讲故事 这个月中旬,有位朋友加我wx求助他的程序线程占有率很高,寻求如何解决,截图如下: 说实话,和不同行业的程序员聊天还是蛮有意思的,广交朋友,也能扩大自己的圈子,朋友说他因为这个b ...

  5. MySQL 连接为什么挂死了？

    摘要:本次分享的是一次关于 MySQL 高可用问题的定位过程,其中曲折颇多但问题本身却比较有些代表性,遂将其记录以供参考. 一.背景 近期由测试反馈的问题有点多,其中关于系统可靠性测试提出的问题令人感 ...

  6. MySQL 连接为什么挂死了

    声明:本文为博主原创文章,由于已授权部分平台发表该文章(知乎.云社区),可能造成发布时间方面的困扰. 一.背景 近期由测试反馈的问题有点多,其中关于系统可靠性测试提出的问题令人感到头疼,一来这类问题有 ...

  7. 应用程序出现挂死，.NET Runtime at IP 791F7E06 (79140000) with exit code 80131506.

    工具出现挂死问题 1.问题描述 工具出现挂死问题,巡检IIS发现以下异常日志 现网系统日志: 事件类型:    错误 事件来源:    .NET Runtime 描述: Application: Di ...

  8. 关于用strace工具定位vrrpd进程有时会挂死的bug

    只做工作总结备忘之用. 正在烧镜像,稍总结一下进来改bug遇到的问题. 一个项目里要用到L3 switch的nat,vrrp功能,但实地测试中偶然出现write file挂死的情况,但不是必现.交付在 ...

  9. IIC挂死问题解决过程

    0.环境:arm CPU 带有IIC控制器作为slave端,带有调试串口. 1.bug表现:IIC slave 在系统启动后概率挂死,导致master无法detect到slave. 猜测1:认为IIC ...

随机推荐

  1. Java数据库编程——事务

    我们可以将一组语句构建成一个事务(transaction).当所有语句都顺利执行之后,事务可以提交(commit).否则,如果其中某个语句遇到错误,那么事务将被回滚,就好像没有任何语句被执行过一样. ...

  2. MySQL建表时，日期时间类型选择

    MySQL(5.5)所支持的日期时间类型有:DATETIME. TIMESTAMP.DATE.TIME.YEAR. 几种类型比较如下: 日期时间类型 占用空间 日期格式 最小值 最大值 零值表示  D ...

  3. Nios II uCLinux/Linux启动分析

    1. 说明 本文采用的Linux源码版本来自Altera公司FTP.不考虑zImage生成的Compress过程.因为zImage是内核binary文件经过gzip 压缩,并在头部添加解压缩代码实现的 ...

  4. KMP字符串模式匹配详解（zz)

    刚看到位兄弟也贴了份KMP算法说明,但本人觉得说的不是很详细,当初我在看这个算法的时候也看的头晕昏昏的,我贴的这份也是网上找的.且听详细分解: KMP字符串模式匹配详解 来自CSDN     A_B_ ...

  5. SQL Server 根据表名取得 表主键

    exec sp_primary_keys_rowset N'表名',NULL

  6. Android-经常涉及到的权限

    Android中配置权限的方法: 在AndroidMainFest.xml中加上以下代码 Android中一些经常涉及到的权限: 添加WiFi以及访问网络的权限: <uses-permissio ...

  7. 【原创】Android自定义适配器的使用方法

    比如说我们已经得到了数据,想在一个listview或者在其他的控件中显示的,并且我们显示想要自己设计样式来显示的话就要用到自定义适配器了,下面让我们结合代码讲一下具体的使用方法: 代码会有注释的哦: ...

  8. PHP多文件上传代码练习

    HTML表单: <html> <head><title>upload file</title> <meta http-equiv="Co ...

  9. js 因加入form导致两个table之间出现空白问题

    在<FORM>中加CSS <table> ....... </table> <form style="padding:0; margin:0;&qu ...

  10. docker基本

    安装(centos): Docker 运行在 CentOS 7 上,要求系统为64位.系统内核版本为 3.10 以上.Docker 运行在 CentOS-6.5 或更高的版本的 CentOS 上,要求 ...