hbase0.92 数据修复
现有集群环境；8台region server、1台master server
早晨重启某一region server后，该server的region全部下线未恢复。
重启前
hbase(main):031:0> status 'simple'
7 live servers
ec2d-cntprocdn-03.mypna.com:60020 1556378718902
requestsPerSecond=0, numberOfOnlineRegions=5, usedHeapMB=737, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556378795513
requestsPerSecond=0, numberOfOnlineRegions=9, usedHeapMB=960, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556378718227
requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=225, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556378767123
requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=1013, maxHeapMB=4480
ec2d-cntprocdn-01.mypna.com:60020 1556379153372
requestsPerSecond=0, numberOfOnlineRegions=394, usedHeapMB=1136, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556379080369
requestsPerSecond=0, numberOfOnlineRegions=11, usedHeapMB=1751, maxHeapMB=4480
ec2d-cntprocdn-07.mypna.com:60020 1556587728098
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=143, maxHeapMB=4480
0 dead servers
Aggregate load: 0, regions: 442
将ec2d-cntprocdn-01节点重启后发现该节点热region server并未如期上线
hbase(main):009:0> status 'simple'
7 live servers
ec2d-cntprocdn-03.mypna.com:60020 1556378718902
requestsPerSecond=9, numberOfOnlineRegions=5, usedHeapMB=762, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556378795513
requestsPerSecond=0, numberOfOnlineRegions=9, usedHeapMB=981, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556378718227
requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=244, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556378767123
requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=1031, maxHeapMB=4480
ec2d-cntprocdn-01.mypna.com:60020 1556588642407
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=126, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556379080369
requestsPerSecond=0, numberOfOnlineRegions=11, usedHeapMB=1771, maxHeapMB=4480
ec2d-cntprocdn-07.mypna.com:60020 1556587728098
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=160, maxHeapMB=4480
0 dead servers
Aggregate load: 9, regions: 48
通过 status 'detailed'查看详细region信息
hbase(main):032:0> status 'detailed'
version 0.92.1
3 regionsInTransition
.META.,,1.1028785192 state=OPENING, ts=1556588320882, server=null
vendor_source_rf_10066,72003300000564249015,1544548417172.3fb0ccc6a9adc3f56e1f526f0d111672. state=PENDING_OPEN, ts=1556588026138, server=null
vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OPENING, ts=1556587702216, server=null
master coprocessors: []
发现元数据表'.META.' region无法正常上线
于是常识重启集群后再次查看状态，发现所有server的region下线
hbase(main):025:0> status 'simple'
7 live servers
ec2d-cntprocdn-07.mypna.com:60020 1556589413648
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556589532168
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556589462880
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556589822614
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-03.mypna.com:60020 1556589365893
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=131, maxHeapMB=4480
ec2d-cntprocdn-01.mypna.com:60020 1556589914551
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556589364969
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=138, maxHeapMB=4480
0 dead servers
Aggregate load: 0, regions: 0
查看表内容直接报“-ROOT-”表下线
base(main):034:0> scan 'stats_validation'
ROW COLUMN+CELL ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) hbase(main):035:0> scan '.META.'
ROW COLUMN+CELL ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
执行hbase hbck，发现也是-ROOT-,,0 offline
$ hbase hbck Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=, exceptions:
Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:)
at org.apache.hadoop.hbase.client.MetaScanner.access$(MetaScanner.java:)
at org.apache.hadoop.hbase.client.MetaScanner$.connect(MetaScanner.java:)
at org.apache.hadoop.hbase.client.MetaScanner$.connect(MetaScanner.java:)
at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.getMetaEntries(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.doWork(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:)
其中'.META.'和'-ROOT-'表的原理和关系可以参考
https://songlee24.github.io/2015/08/15/root-and-meta-table-structure/
修复过程
停止hbase集群
$ stop-hbase.sh
执行hbase元数据修复，报错可以先不用管
$ hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair RROR: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
// :: ERROR util.HBaseFsck: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
java.io.FileNotFoundException: File does not exist: /hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea/.regioninfo
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:)
// :: ERROR util.HBaseFsck: Bailed out due to:
org.apache.hadoop.hbase.util.HBaseFsck$RegionInfoLoadException: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:)
启动集群，查看master的日志输出
Mon Apr :: EDT Starting master on ec2d-cntprocnn-
core file size (blocks, -c)
data seg size (kbytes, -d) unlimited
scheduling priority (-e)
file size (blocks, -f) unlimited
pending signals (-i)
max locked memory (kbytes, -l)
max memory size (kbytes, -m) unlimited
open files (-n)
pipe size ( bytes, -p)
POSIX message queues (bytes, -q)
real-time priority (-r)
stack size (kbytes, -s)
cpu time (seconds, -t) unlimited
max user processes (-u)
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
-- ::, WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configurati
on.
-- ::, ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry
-- ::, WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configurati
on.
-- ::, ERROR org.mortbay.log: Error starting handlers
java.lang.NoSuchMethodError: org.slf4j.helpers.MessageFormatter.format(Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
at org.slf4j.impl.Log4jLoggerAdapter.warn(Log4jLoggerAdapter.java:)
at org.mortbay.log.Slf4jLog.warn(Slf4jLog.java:)
at org.mortbay.log.Log.warn(Log.java:)
at org.mortbay.component.AbstractLifeCycle.setFailed(AbstractLifeCycle.java:)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:)
at org.mortbay.jetty.Server.doStart(Server.java:)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:)
at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:)
at java.lang.Thread.run(Thread.java:)
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting on ec2d-cntprocdn-.mypna.com,, with ec2d-cntprocdn-.mypna.com,,
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 7eeecbe911ce6add0487d89ba53b7dcc has null regionLocation. But its table vendor_source_rf_10066 isn't in ENABLING state.
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 3fb0ccc6a9adc3f56e1f526f0d111672 has null regionLocation. But its table vendor_source_rf_10066 isn't in ENABLING state.
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 3be02d44bcba07121d9f29494d66d959 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_poi_10066,,.3be02d44bcba07121d9f29494d66d959. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region e29ae55c52ac2d6644a117fdb0e9af19 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.e29ae55c52ac2d6644a117fdb0e9af19. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region c0dbd03ee61162038fc46f7c9f964585 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.c0dbd03ee61162038fc46f7c9f964585. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 96f2f63ed9de87542a8f643704bb0934 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.96f2f63ed9de87542a8f643704bb0934. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region aa2892d5c917027afcc68c7261879765 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.aa2892d5c917027afcc68c7261879765. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 1e32343c54316020cb96f24bf6ce5cbe from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.1e32343c54316020cb96f24bf6ce5cbe. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 5f0e9542c55245ef429557447a03883e from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.5f0e9542c55245ef429557447a03883e. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 3105e4d146648aff31262166669cd170 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.3105e4d146648aff31262166669cd170. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region b65d05bdbcf0938e281a0df1667d2ed5 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.b65d05bdbcf0938e281a0df1667d2ed5. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 230fef99d67d51aa66a0e7234e7b0766 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.230fef99d67d51aa66a0e7234e7b0766. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but re
gion was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 00c4a8cd6ed22995dadc8d6b7e901175 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.00c4a8cd6ed22995dadc8d6b7e901175. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region bd7401a936fb21c910f662b02e254190 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.bd7401a936fb21c910f662b02e254190. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 30711b0a58809200c20aa18d545aa94d from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.30711b0a58809200c20aa18d545aa94d. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 1d4459d84c02de3b0f194f194a183569 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.1d4459d84c02de3b0f194f194a183569. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 46768cc418862ae474f2190bdc7bada5 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.46768cc418862ae474f2190bdc7bada5. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 9ad69996a4668c6cf93592dddc2d2752 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.9ad69996a4668c6cf93592dddc2d2752. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 2f1752fa24c76301bca2d63478b65381 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.2f1752fa24c76301bca2d63478b65381. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region dfd32ba7ea53cc76abf277e9eda8547e from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.dfd32ba7ea53cc76abf277e9eda8547e. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 965b2b3bb59e0b23cfebabd59c03da12 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.965b2b3bb59e0b23cfebabd59c03da12. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ec2d-cntprocdn-.mypna.com,, due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:)
at java.lang.reflect.Constructor.newInstance(Constructor.java:)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughter(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughters(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.HMaster.fixupDaughters(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:)
at $Proxy11.openRegion(Unknown Source)
at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
... more
-- ::, ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ec2d-cntprocdn-.mypna.com,, due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:)
at java.lang.reflect.Constructor.newInstance(Constructor.java:)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughter(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughters(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.HMaster.fixupDaughters(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:)
at $Proxy11.openRegion(Unknown Source)
at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
再次查看region信息
hbase(main):023:0* status 'simple'
7 live servers
ec2d-cntprocdn-01.mypna.com:60020 1556591551521
requestsPerSecond=0, numberOfOnlineRegions=393, usedHeapMB=1828, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556591001078
requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=1688, maxHeapMB=4480
ec2d-cntprocdn-03.mypna.com:60020 1556591001666
requestsPerSecond=0, numberOfOnlineRegions=4, usedHeapMB=341, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556591062296
requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=387, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556591098649
requestsPerSecond=0, numberOfOnlineRegions=10, usedHeapMB=363, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556591459397
requestsPerSecond=0, numberOfOnlineRegions=12, usedHeapMB=425, maxHeapMB=4480
ec2d-cntprocdn-07.mypna.com:60020 1556591049767
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=315, maxHeapMB=4480
0 dead servers
Aggregate load: 0, regions: 442
hbase0.92 数据修复的更多相关文章
- nutch-2.2.1 hadoop-1.2.1 hbase-0.92.1 集群部署
国内私募机构九鼎控股打造APP,来就送 20元现金领取地址:http://jdb.jiudingcapital.com/phone.html 内部邀请码:C8E245J (不写邀请码,没有现金送) 国 ...
- hbase-0.92.1表备份还原
原表结构和数据 hbase(main):021:0* describe 'test' DESCRIPTION ENABLED {NAME => ', TTL = true > ', COM ...
- nutch-2.2.1 hadoop-1.2.1 hbase-0.92.1 集群部署(实用)
原文地址: http://www.cnblogs.com/i80386/p/3540389.html 参考网站:http://blog.csdn.net/weijonathan/article/det ...
- hbase-0.92.1集群部署
环境 主机名 IP地址 角色 安装目录 sht-sgmhadoopnn-01 172.16.101.55 NameNode.SecondaryNameNode. JobTracker.HMaster ...
- 详解SQL Server数据修复命令DBCC的使用
严重级别为 21 表示可能存在数据损坏. 可能的原因包括损坏的页链.损坏的 IAM 或该对象的 sys.objects目录视图中存在无效条目. 这些错误通常由硬件或磁盘设备驱动程序故障而引起. MS ...
- 某企业用友U8+中勒索病毒后数据修复及重新实施过程记录
近期某客户中了勒索病毒,虽然前期多次提醒客户注意异地备份,但始终未执行,导致悲剧. 经过几天的努力,该客户信息系统已基本恢复正常运行,现将相关过程记录如下,作为警示. 方案抉择 交赎金解密:风险过高, ...
- 通用高效的数据修复方法：Row level repair
导读:随着大数据的进一步发展,NoSQL 数据库系统迅速发展并得到了广泛的应用.其中,Apache Cassandra 是最广泛使用的数据库之一.对于 Cassandra 的优化是大家研究的热点,而 ...
- Python 爬取 热词并进行分类数据分析-[数据修复]
日期:2020.02.01 博客期:140 星期六 [本博客的代码如若要使用,请在下方评论区留言,之后再用(就是跟我说一声)] 所有相关跳转: a.[简单准备] b.[云图制作+数据导入] c.[拓扑 ...
- mongodb数据修复宝典
mongodb数据修复宝典 1. 版本信息 mongodb版本: 3.6.14 2.mongodb日志内容: 2019-10-04T10:41:38.558+0000 E STORAGE [inita ...
随机推荐
- Java ThreadPoolExecutor线程池原理及源码分析
一.源码分析(基于JDK1.6) ThreadExecutorPool是使用最多的线程池组件,了解它的原始资料最好是从从设计者(Doug Lea)的口中知道它的来龙去脉.在Jdk1.6中,Thread ...
- 在Linux-PC上建立kdump调试环境
kdump就是kernel dump的简称,它是从DDR中直接获取的linux内核数据(系统代码/数据).分析kdump是定位内核panic问题的有效手段之一,同时,通过kdump研究内核数据结构,也 ...
- matlab 在柱状图上 显示数字
x = rand(100,1);[n,y] = hist(x);bar(y,n);for i = 1:length(y)text(y(i),n(i)+0.5,num2str(n(i)));end
- Java多线程-新特征-信号量Semaphore
简介信号量(Semaphore),有时被称为信号灯,是在多线程环境下使用的一种设施, 它负责协调各个线程, 以保证它们能够正确.合理的使用公共资源. 概念Semaphore分为单值和多值两种,前者只能 ...
- mysql中的自定义函数
创建不带参数的自定义函数: 使用: 创建带参数的自定义函数: 使用: 创建具有复合结构的函数体的自定义函数:
- window - BOM对象
Window 对象 Window 对象表示浏览器中打开的窗口. 如果文档包含框架(frame 或 iframe 标签),浏览器会为 HTML 文档创建一个 window 对象,并为每个框架创建一个额外 ...
- solrCloud选举leader的逻辑分析
First call *setup(ElectionContext) to ensure the election process is in it'd. Next calljoinElecti ...
- Resin 的watchdog(看门狗)介绍和resin负载均衡实现
为了稳定和安全,Resin使用一个独立的watchdog进程来启动和监视Resin服务器.watchdog连续你检测Resin服务器的状态,如果其没有反应或者迟钝,将会重启Resin服务器进程.大多数 ...
- resin3.X那些事之resin.conf
[经验总结]resin那些事之resin.conf ----by johnson 话说与resin打交道很久了,却从未系统了解过.resin一听火了,说:你老兄当真与我打交道很久了?工具.流程如此发达 ...
- 【HDU5861】Road
题意 有n个村庄排成一排,有n-1条路将他们连在一起.每条路开放一天都会花费一定数量的钱.你可以选择打开或者关上任意条路在任意一天,但是每条路只能打开和关闭一次.我们知道m天的运输计划.每天都有一辆马 ...