hbase0.92 数据修复
现有集群环境；8台region server、1台master server
早晨重启某一region server后，该server的region全部下线未恢复。
重启前
hbase(main):031:0> status 'simple'
7 live servers
ec2d-cntprocdn-03.mypna.com:60020 1556378718902
requestsPerSecond=0, numberOfOnlineRegions=5, usedHeapMB=737, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556378795513
requestsPerSecond=0, numberOfOnlineRegions=9, usedHeapMB=960, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556378718227
requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=225, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556378767123
requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=1013, maxHeapMB=4480
ec2d-cntprocdn-01.mypna.com:60020 1556379153372
requestsPerSecond=0, numberOfOnlineRegions=394, usedHeapMB=1136, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556379080369
requestsPerSecond=0, numberOfOnlineRegions=11, usedHeapMB=1751, maxHeapMB=4480
ec2d-cntprocdn-07.mypna.com:60020 1556587728098
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=143, maxHeapMB=4480
0 dead servers
Aggregate load: 0, regions: 442
将ec2d-cntprocdn-01节点重启后发现该节点热region server并未如期上线
hbase(main):009:0> status 'simple'
7 live servers
ec2d-cntprocdn-03.mypna.com:60020 1556378718902
requestsPerSecond=9, numberOfOnlineRegions=5, usedHeapMB=762, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556378795513
requestsPerSecond=0, numberOfOnlineRegions=9, usedHeapMB=981, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556378718227
requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=244, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556378767123
requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=1031, maxHeapMB=4480
ec2d-cntprocdn-01.mypna.com:60020 1556588642407
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=126, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556379080369
requestsPerSecond=0, numberOfOnlineRegions=11, usedHeapMB=1771, maxHeapMB=4480
ec2d-cntprocdn-07.mypna.com:60020 1556587728098
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=160, maxHeapMB=4480
0 dead servers
Aggregate load: 9, regions: 48
通过 status 'detailed'查看详细region信息
hbase(main):032:0> status 'detailed'
version 0.92.1
3 regionsInTransition
.META.,,1.1028785192 state=OPENING, ts=1556588320882, server=null
vendor_source_rf_10066,72003300000564249015,1544548417172.3fb0ccc6a9adc3f56e1f526f0d111672. state=PENDING_OPEN, ts=1556588026138, server=null
vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OPENING, ts=1556587702216, server=null
master coprocessors: []
发现元数据表'.META.' region无法正常上线
于是常识重启集群后再次查看状态，发现所有server的region下线
hbase(main):025:0> status 'simple'
7 live servers
ec2d-cntprocdn-07.mypna.com:60020 1556589413648
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556589532168
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556589462880
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556589822614
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-03.mypna.com:60020 1556589365893
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=131, maxHeapMB=4480
ec2d-cntprocdn-01.mypna.com:60020 1556589914551
requestsPerSecond=0, numberOfOnlineRegions=, usedHeapMB=130, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556589364969
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=138, maxHeapMB=4480
0 dead servers
Aggregate load: 0, regions: 0
查看表内容直接报“-ROOT-”表下线
base(main):034:0> scan 'stats_validation'
ROW COLUMN+CELL ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) hbase(main):035:0> scan '.META.'
ROW COLUMN+CELL ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
执行hbase hbck，发现也是-ROOT-,,0 offline
$ hbase hbck Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=, exceptions:
Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) Mon Apr :: EDT , org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:)
at org.apache.hadoop.hbase.client.MetaScanner.access$(MetaScanner.java:)
at org.apache.hadoop.hbase.client.MetaScanner$.connect(MetaScanner.java:)
at org.apache.hadoop.hbase.client.MetaScanner$.connect(MetaScanner.java:)
at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.getMetaEntries(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.doWork(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:)
其中'.META.'和'-ROOT-'表的原理和关系可以参考
https://songlee24.github.io/2015/08/15/root-and-meta-table-structure/
修复过程
停止hbase集群
$ stop-hbase.sh
执行hbase元数据修复，报错可以先不用管
$ hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair RROR: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
// :: ERROR util.HBaseFsck: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
java.io.FileNotFoundException: File does not exist: /hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea/.regioninfo
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:)
// :: ERROR util.HBaseFsck: Bailed out due to:
org.apache.hadoop.hbase.util.HBaseFsck$RegionInfoLoadException: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:)
at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:)
启动集群，查看master的日志输出
Mon Apr :: EDT Starting master on ec2d-cntprocnn-
core file size (blocks, -c)
data seg size (kbytes, -d) unlimited
scheduling priority (-e)
file size (blocks, -f) unlimited
pending signals (-i)
max locked memory (kbytes, -l)
max memory size (kbytes, -m) unlimited
open files (-n)
pipe size ( bytes, -p)
POSIX message queues (bytes, -q)
real-time priority (-r)
stack size (kbytes, -s)
cpu time (seconds, -t) unlimited
max user processes (-u)
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
-- ::, WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configurati
on.
-- ::, ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry
-- ::, WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configurati
on.
-- ::, ERROR org.mortbay.log: Error starting handlers
java.lang.NoSuchMethodError: org.slf4j.helpers.MessageFormatter.format(Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
at org.slf4j.impl.Log4jLoggerAdapter.warn(Log4jLoggerAdapter.java:)
at org.mortbay.log.Slf4jLog.warn(Slf4jLog.java:)
at org.mortbay.log.Log.warn(Log.java:)
at org.mortbay.component.AbstractLifeCycle.setFailed(AbstractLifeCycle.java:)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:)
at org.mortbay.jetty.Server.doStart(Server.java:)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:)
at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:)
at java.lang.Thread.run(Thread.java:)
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting on ec2d-cntprocdn-.mypna.com,, with ec2d-cntprocdn-.mypna.com,,
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 7eeecbe911ce6add0487d89ba53b7dcc has null regionLocation. But its table vendor_source_rf_10066 isn't in ENABLING state.
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 3fb0ccc6a9adc3f56e1f526f0d111672 has null regionLocation. But its table vendor_source_rf_10066 isn't in ENABLING state.
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 3be02d44bcba07121d9f29494d66d959 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_poi_10066,,.3be02d44bcba07121d9f29494d66d959. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region e29ae55c52ac2d6644a117fdb0e9af19 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.e29ae55c52ac2d6644a117fdb0e9af19. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region c0dbd03ee61162038fc46f7c9f964585 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.c0dbd03ee61162038fc46f7c9f964585. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 96f2f63ed9de87542a8f643704bb0934 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.96f2f63ed9de87542a8f643704bb0934. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region aa2892d5c917027afcc68c7261879765 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.aa2892d5c917027afcc68c7261879765. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 1e32343c54316020cb96f24bf6ce5cbe from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.1e32343c54316020cb96f24bf6ce5cbe. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 5f0e9542c55245ef429557447a03883e from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.5f0e9542c55245ef429557447a03883e. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 3105e4d146648aff31262166669cd170 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.3105e4d146648aff31262166669cd170. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region b65d05bdbcf0938e281a0df1667d2ed5 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.b65d05bdbcf0938e281a0df1667d2ed5. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 230fef99d67d51aa66a0e7234e7b0766 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.230fef99d67d51aa66a0e7234e7b0766. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but re
gion was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 00c4a8cd6ed22995dadc8d6b7e901175 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.00c4a8cd6ed22995dadc8d6b7e901175. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region bd7401a936fb21c910f662b02e254190 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.bd7401a936fb21c910f662b02e254190. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 30711b0a58809200c20aa18d545aa94d from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.30711b0a58809200c20aa18d545aa94d. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 1d4459d84c02de3b0f194f194a183569 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.1d4459d84c02de3b0f194f194a183569. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 46768cc418862ae474f2190bdc7bada5 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.46768cc418862ae474f2190bdc7bada5. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 9ad69996a4668c6cf93592dddc2d2752 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.9ad69996a4668c6cf93592dddc2d2752. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 2f1752fa24c76301bca2d63478b65381 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.2f1752fa24c76301bca2d63478b65381. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region dfd32ba7ea53cc76abf277e9eda8547e from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.dfd32ba7ea53cc76abf277e9eda8547e. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=, server=null and not in PENDING_OPEN or OPENING
-- ::, WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 965b2b3bb59e0b23cfebabd59c03da12 from server ec2d-cntprocdn-.mypna.com,, but region was in the state vendor_source_rf_10066,,.965b2b3bb59e0b23cfebabd59c03da12. state=OPEN, ts=, server=ec2d-cntprocdn-.mypna.com,, and not in expected PENDING_OPEN or OPENING states
-- ::, ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ec2d-cntprocdn-.mypna.com,, due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:)
at java.lang.reflect.Constructor.newInstance(Constructor.java:)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughter(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughters(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.HMaster.fixupDaughters(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:)
at $Proxy11.openRegion(Unknown Source)
at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
... more
-- ::, ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ec2d-cntprocdn-.mypna.com,, due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:)
at java.lang.reflect.Constructor.newInstance(Constructor.java:)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughter(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughters(ServerShutdownHandler.java:)
at org.apache.hadoop.hbase.master.HMaster.fixupDaughters(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,,.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:)
at $Proxy11.openRegion(Unknown Source)
at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:)
再次查看region信息
hbase(main):023:0* status 'simple'
7 live servers
ec2d-cntprocdn-01.mypna.com:60020 1556591551521
requestsPerSecond=0, numberOfOnlineRegions=393, usedHeapMB=1828, maxHeapMB=4480
ec2d-cntprocdn-04.mypna.com:60020 1556591001078
requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=1688, maxHeapMB=4480
ec2d-cntprocdn-03.mypna.com:60020 1556591001666
requestsPerSecond=0, numberOfOnlineRegions=4, usedHeapMB=341, maxHeapMB=4480
ec2d-cntprocdn-06.mypna.com:60020 1556591062296
requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=387, maxHeapMB=4480
ec2d-cntprocdn-05.mypna.com:60020 1556591098649
requestsPerSecond=0, numberOfOnlineRegions=10, usedHeapMB=363, maxHeapMB=4480
ec2d-cntprocdn-02.mypna.com:60020 1556591459397
requestsPerSecond=0, numberOfOnlineRegions=12, usedHeapMB=425, maxHeapMB=4480
ec2d-cntprocdn-07.mypna.com:60020 1556591049767
requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=315, maxHeapMB=4480
0 dead servers
Aggregate load: 0, regions: 442
hbase0.92 数据修复的更多相关文章
- nutch-2.2.1 hadoop-1.2.1 hbase-0.92.1 集群部署
国内私募机构九鼎控股打造APP,来就送 20元现金领取地址:http://jdb.jiudingcapital.com/phone.html 内部邀请码:C8E245J (不写邀请码,没有现金送) 国 ...
- hbase-0.92.1表备份还原
原表结构和数据 hbase(main):021:0* describe 'test' DESCRIPTION ENABLED {NAME => ', TTL = true > ', COM ...
- nutch-2.2.1 hadoop-1.2.1 hbase-0.92.1 集群部署(实用)
原文地址: http://www.cnblogs.com/i80386/p/3540389.html 参考网站:http://blog.csdn.net/weijonathan/article/det ...
- hbase-0.92.1集群部署
环境 主机名 IP地址 角色 安装目录 sht-sgmhadoopnn-01 172.16.101.55 NameNode.SecondaryNameNode. JobTracker.HMaster ...
- 详解SQL Server数据修复命令DBCC的使用
严重级别为 21 表示可能存在数据损坏. 可能的原因包括损坏的页链.损坏的 IAM 或该对象的 sys.objects目录视图中存在无效条目. 这些错误通常由硬件或磁盘设备驱动程序故障而引起. MS ...
- 某企业用友U8+中勒索病毒后数据修复及重新实施过程记录
近期某客户中了勒索病毒,虽然前期多次提醒客户注意异地备份,但始终未执行,导致悲剧. 经过几天的努力,该客户信息系统已基本恢复正常运行,现将相关过程记录如下,作为警示. 方案抉择 交赎金解密:风险过高, ...
- 通用高效的数据修复方法：Row level repair
导读:随着大数据的进一步发展,NoSQL 数据库系统迅速发展并得到了广泛的应用.其中,Apache Cassandra 是最广泛使用的数据库之一.对于 Cassandra 的优化是大家研究的热点,而 ...
- Python 爬取 热词并进行分类数据分析-[数据修复]
日期:2020.02.01 博客期:140 星期六 [本博客的代码如若要使用,请在下方评论区留言,之后再用(就是跟我说一声)] 所有相关跳转: a.[简单准备] b.[云图制作+数据导入] c.[拓扑 ...
- mongodb数据修复宝典
mongodb数据修复宝典 1. 版本信息 mongodb版本: 3.6.14 2.mongodb日志内容: 2019-10-04T10:41:38.558+0000 E STORAGE [inita ...
随机推荐
- Python图片转字符画
PIL安装很麻烦,推荐下载exe直接安装 PIL官网:http://pythonware.com/products/pil/ 但现在里面下载链接访问不了,我把32位和64位版本上传到博客园以供下载 P ...
- Cable master（二分-求可行解）
Inhabitants of the Wonderland have decided to hold a regional programming contest. The Judging Commi ...
- springMVC第一天
这些是springMVC3.2所用到的jar包 web.xml配置 <?xml version="1.0" encoding="UTF-8"?> & ...
- wince驱动开发入门
因为课题前期调研没做好,用的CPU板卡和数据采集卡来自两个部门.加上买的是裸板,自己定制的OS,技术支持不爱搭理.所以给的AI板卡的驱动一直装不上,自己在郁闷中寻找答案,就扎进了wince驱动的知识库 ...
- 搭建J2EE开发平台-Eclipse+MySql+tomcat
搭建J2EE开发平台-Eclipse+MySql+tomcat 分类: ·Java 2010-10-10 15:45 2596人阅读 评论(3) 收藏 举报 mysql平台eclipsetomcatj ...
- Bell数和Stirling数
前面说到了Catalan数,现在来了一个Bell数和Stirling数.什么是Bell数,什么是Stirling数呢?两者的关系如何,有用于解决什么算法问题呢? Bell数是以Bell这个人命名的,组 ...
- 无锁的同步策略——CAS操作详解
目录 1. 从乐观锁和悲观锁谈起 2. CAS详解 2.1 CAS指令 2.3 Java中的CAS指令 2.4 CAS结合失败重试机制进行并发控制 3. CAS操作的优势和劣势 3.1 CAS相比独占 ...
- WordPress 4.1去掉侧边栏“功能”小工具中WordPress.Org
打开wp-includes/default-widgets.php,注释掉第398到第403行 echo apply_filters( 'widget_meta_poweredby', sprintf ...
- HTML中禁用表中控件的两种方法与区别
在网页的制作过程中,我们会经常使用到表单.但是有时候我们会希望表单上的控件是不可修改的,比如在修改密码的网页中,显示用户名的文本框就应该是不可修改状态的. 在html中有两种禁用的方法,他们分别是: ...
- Ionic01 简单介绍、环境搭建、创建项目、项目结构、创建组件、创建页面、子页面跳转
1 Ionic 基本介绍 Ionic 是一款基于 Angular.Cordova 的强大的 HTML5 移动应用开发框架 , 可以快速创建一个跨平台的移动应用.可以快速开发移动 App.移动端 WEB ...