【转】一次HBase问题的解决过程（Status: INCONSISTENT）
【From】https://www.cnblogs.com/quchunhui/p/9583746.html
==版本信息==
HBase：2.7.1
Storm：1.0.1
RocketMQ：3.4.6（阿里版）
==问题描述==
2018年9月3号晚上23点左右，例行巡检系统运行状况时，
发现Storm消费RocketMQ入HBase的过程中，RocketMQ的TPS极低，甚至为0了。
（Storm从RocketMQ消费消息自后，读写HBase）
==问题定位==
查看了Storm的日志，发现有如下异常信息：
- 2018-09-03 20:29:43.696 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] java.net.SocketTimeoutException: callTimeout=1200000, callDuration=1536845: row '97390002332830' on table 'DBN_ANE' at region=DBN_ANE,97,1507795766296.69724c5bd4e82bae4a3ec8a95c8f9c65., hostname=dscn33,16020,1529215379398, seqNum=17296637
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.HTable.get(HTable.java:862)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.HTable.get(HTable.java:828)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at com.dscn.topology.BasicBolt.execute(BasicBolt.java:125)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.daemon.executor$fn__5030$tuple_action_fn__5032.invoke(executor.clj:729)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.daemon.executor$mk_task_receiver$fn__4951.invoke(executor.clj:461)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.disruptor$clojure_handler$reify__4465.onEvent(disruptor.clj:40)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:482)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:460)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.daemon.executor$fn__5030$fn__5043$fn__5096.invoke(executor.clj:848)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at clojure.lang.AFn.run(AFn.java:22)
- 2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at java.lang.Thread.run(Thread.java:745)
- 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] Caused by: java.io.IOException: HRegionInfo was null in DBN_ANE, row=keyvalues={DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf./info:seqnumDuringOpen/1535968873207/Put/vlen=8/seqid=0, DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf./info:server/1535968873207/Put/vlen=12/seqid=0, DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf./info:serverstartcode/1535968873207/Put/vlen=8/seqid=0}
- 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1289)
- 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
- 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1152)
- 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1136)
- 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:957)
- 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
- 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:75)
- 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
- 2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] ... 14 more
然后查看HBase集群日志，里面也出现错误，日志如下：
- 2018-09-03 17:58:57,603 WARN [MemStoreFlusher.0] regionserver.HStore: Failed flushing store file, retrying num=0
- java.io.IOException: All datanodes 10.11.2.31:50010 are bad. Aborting...
- at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137)
- at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
- at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
- 2018-09-03 17:58:57,606 INFO [regionserver/dscn33/10.11.2.31:16020.logRoller] wal.FSHLog: Slow sync cost: 16261 ms, current pipeline: [10.11.2.31:50010]
- 2018-09-03 17:58:57,606 ERROR [regionserver/dscn33/10.11.2.31:16020.logRoller] wal.FSHLog: Failed close of WAL writer hdfs://ns/hbase/WALs/dscn33,16020,1529215379398/dscn33%2C16020%2C1529215379398.default.1535968646203, unflushedEntries=30
- org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=41639867, requesting roll of WAL
- at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1615)
- at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:833)
- at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:699)
- at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
- at java.lang.Thread.run(Thread.java:745)
- Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=41639867, requesting roll of WAL
- at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1898)
- at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1750)
- at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672)
- at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
- ... 1 more
- Caused by: java.io.IOException: All datanodes 10.11.2.31:50010 are bad. Aborting...
- at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137)
- at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
- at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
- 2018-09-03 17:58:41,358 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher: Cache flush failed for region DBN_JBD,30270769371,1515408107652.fca3d8ee774898302a377535e3f0ccac.
- org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=41639867, requesting roll of WAL
- at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1898)
- at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1750)
- at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672)
- at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
- at java.lang.Thread.run(Thread.java:745)
- Caused by: java.io.IOException: All datanodes 10.11.2.31:50010 are bad. Aborting...
- at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137)
- at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
- at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
检查HBase集群状态（hbase hbck），发现异常，定位为HBase集群问题：
- [root@dscn12 ~]# hbase hbck
- SLF4J: Class path contains multiple SLF4J bindings.
- SLF4J: Found binding in [jar:file:/home/hadmin/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
- SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
- HBaseFsck command line options:
- 2018-09-04 00:01:54,217 INFO [main] util.HBaseFsck: Launching hbck
- 2018-09-04 00:01:54,456 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x57576994 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
- 2018-09-04 00:01:54,471 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
- 2018-09-04 00:01:54,471 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=dscn12
- 2018-09-04 00:01:54,471 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_73
- 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
- 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_73/jre
- 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/home/hadmin/hbase/conf:/usr/java/default/lib/tools.jar:/home/hadmin/hbase:/home/hadmin/hbase/lib/activation-1.1.jar:/home/hadmin/hbase/lib/antisamy-1.4.3.jar:/home/hadmin/hbase/lib/aopalliance-1.0.jar:/home/hadmin/hbase/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadmin/hbase/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadmin/hbase/lib/api-asn1-api-1.0.0-M20.jar:/home/hadmin/hbase/lib/api-util-1.0.0-M20.jar:/home/hadmin/hbase/lib/asm-3.1.jar:/home/hadmin/hbase/lib/avro-1.7.4.jar:/home/hadmin/hbase/lib/batik-css-1.7.jar:/home/hadmin/hbase/lib/batik-ext-1.7.jar:/home/hadmin/hbase/lib/batik-util-1.7.jar:/home/hadmin/hbase/lib/bsh-core-2.0b4.jar:/home/hadmin/hbase/lib/commons-beanutils-1.7.0.jar:/home/hadmin/hbase/lib/commons-beanutils-core-1.7.0.jar:/home/hadmin/hbase/lib/commons-cli-1.2.jar:/home/hadmin/hbase/lib/commons-codec-1.9.jar:/home/hadmin/hbase/lib/commons-collections-3.2.2.jar:/home/hadmin/hbase/lib/commons-compress-1.4.1.jar:/home/hadmin/hbase/lib/commons-configuration-1.6.jar:/home/hadmin/hbase/lib/commons-daemon-1.0.13.jar:/home/hadmin/hbase/lib/commons-digester-1.8.jar:/home/hadmin/hbase/lib/commons-el-1.0.jar:/home/hadmin/hbase/lib/commons-fileupload-1.2.jar:/home/hadmin/hbase/lib/commons-httpclient-3.1.jar:/home/hadmin/hbase/lib/commons-io-2.4.jar:/home/hadmin/hbase/lib/commons-lang-2.6.jar:/home/hadmin/hbase/lib/commons-logging-1.2.jar:/home/hadmin/hbase/lib/commons-math-2.2.jar:/home/hadmin/hbase/lib/commons-math3-3.1.1.jar:/home/hadmin/hbase/lib/commons-net-3.1.jar:/home/hadmin/hbase/lib/disruptor-3.3.0.jar:/home/hadmin/hbase/lib/esapi-2.1.0.jar:/home/hadmin/hbase/lib/findbugs-annotations-1.3.9-1.jar:/home/hadmin/hbase/lib/guava-12.0.1.jar:/home/hadmin/hbase/lib/guice-3.0.jar:/home/hadmin/hbase/lib/guice-servlet-3.0.jar:/home/hadmin/hbase/lib/hadoop-annotations-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-auth-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-client-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-common-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-hdfs-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-app-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-common-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-core-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-api-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-client-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-common-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-server-common-2.5.1.jar:/home/hadmin/hbase/lib/hbase-annotations-1.2.1.jar:/home/hadmin/hbase/lib/hbase-annotations-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-client-1.2.1.jar:/home/hadmin/hbase/lib/hbase-common-1.2.1.jar:/home/hadmin/hbase/lib/hbase-common-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-examples-1.2.1.jar:/home/hadmin/hbase/lib/hbase-external-blockcache-1.2.1.jar:/home/hadmin/hbase/lib/hbase-hadoop2-compat-1.2.1.jar:/home/hadmin/hbase/lib/hbase-hadoop-compat-1.2.1.jar:/home/hadmin/hbase/lib/hbase-it-1.2.1.jar:/home/hadmin/hbase/lib/hbase-it-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-prefix-tree-1.2.1.jar:/home/hadmin/hbase/lib/hbase-procedure-1.2.1.jar:/home/hadmin/hbase/lib/hbase-protocol-1.2.1.jar:/home/hadmin/hbase/lib/hbase-resource-bundle-1.2.1.jar:/home/hadmin/hbase/lib/hbase-rest-1.2.1.jar:/home/hadmin/hbase/lib/hbase-server-1.2.1.jar:/home/hadmin/hbase/lib/hbase-server-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-shell-1.2.1.jar:/home/hadmin/hbase/lib/hbase-thrift-1.2.1.jar:/home/hadmin/hbase/lib/htrace-core-3.1.0-incubating.jar:/home/hadmin/hbase/lib/httpclient-4.2.5.jar:/home/hadmin/hbase/lib/httpcore-4.4.1.jar:/home/hadmin/hbase/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hbase/lib/jackson-jaxrs-1.9.13.jar:/home/hadmin/hbase/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hbase/lib/jackson-xc-1.9.13.jar:/home/hadmin/hbase/lib/jamon-runtime-2.4.1.jar:/home/hadmin/hbase/lib/jasper-compiler-5.5.23.jar:/home/hadmin/hbase/lib/jasper-runtime-5.5.23.jar:/home/hadmin/hbase/lib/javax.inject-1.jar:/home/hadmin/hbase/lib/java-xmlbuilder-0.4.jar:/home/hadmin/hbase/lib/jaxb-api-2.2.2.jar:/home/hadmin/hbase/lib/jaxb-impl-2.2.3-1.jar:/home/hadmin/hbase/lib/jcodings-1.0.8.jar:/home/hadmin/hbase/lib/jersey-client-1.9.jar:/home/hadmin/hbase/lib/jersey-core-1.9.jar:/home/hadmin/hbase/lib/jersey-guice-1.9.jar:/home/hadmin/hbase/lib/jersey-json-1.9.jar:/home/hadmin/hbase/lib/jersey-server-1.9.jar:/home/hadmin/hbase/lib/jets3t-0.9.0.jar:/home/hadmin/hbase/lib/jettison-1.3.3.jar:/home/hadmin/hbase/lib/jetty-6.1.26.jar:/home/hadmin/hbase/lib/jetty-sslengine-6.1.26.jar:/home/hadmin/hbase/lib/jetty-util-6.1.26.jar:/home/hadmin/hbase/lib/joni-2.1.2.jar:/home/hadmin/hbase/lib/jruby-complete-1.6.8.jar:/home/hadmin/hbase/lib/jsch-0.1.42.jar:/home/hadmin/hbase/lib/jsp-2.1-6.1.14.jar:/home/hadmin/hbase/lib/jsp-api-2.1-6.1.14.jar:/home/hadmin/hbase/lib/jsr305-1.3.9.jar:/home/hadmin/hbase/lib/junit-4.12.jar:/home/hadmin/hbase/lib/leveldbjni-all-1.8.jar:/home/hadmin/hbase/lib/libthrift-0.9.3.jar:/home/hadmin/hbase/lib/log4j-1.2.17.jar:/home/hadmin/hbase/lib/metrics-core-2.2.0.jar:/home/hadmin/hbase/lib/nekohtml-1.9.12.jar:/home/hadmin/hbase/lib/netty-all-4.0.23.Final.jar:/home/hadmin/hbase/lib/paranamer-2.3.jar:/home/hadmin/hbase/lib/protobuf-java-2.5.0.jar:/home/hadmin/hbase/lib/servlet-api-2.5-6.1.14.jar:/home/hadmin/hbase/lib/servlet-api-2.5.jar:/home/hadmin/hbase/lib/slf4j-api-1.7.7.jar:/home/hadmin/hbase/lib/slf4j-log4j12-1.7.5.jar:/home/hadmin/hbase/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hbase/lib/spymemcached-2.11.6.jar:/home/hadmin/hbase/lib/xalan-2.7.0.jar:/home/hadmin/hbase/lib/xml-apis-1.3.03.jar:/home/hadmin/hbase/lib/xml-apis-ext-1.3.04.jar:/home/hadmin/hbase/lib/xmlenc-0.52.jar:/home/hadmin/hbase/lib/xom-1.2.5.jar:/home/hadmin/hbase/lib/xz-1.0.jar:/home/hadmin/hbase/lib/zookeeper-3.4.6.jar:/home/hadmin/hadoop/etc/hadoop:/home/hadmin/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/hadoop-annotations-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadmin/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadmin/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadmin/hadoop/share/hadoop/common/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/junit-4.11.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/home/hadmin/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadmin/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-common-2.7.1-tests.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-nfs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/hdfs:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.1-tests.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1-tests.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.1.jar:.::/home/hadmin/hadoop/contrib/capacity-scheduler/*.jar:/home/hadmin/hadoop/bin
- 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/home/hadmin/hadoop/lib/native
- 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
- 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
- 2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
- 2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
- 2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-229.el7.x86_64
- 2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root
- 2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/root
- 2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/root
- 2018-09-04 00:01:54,474 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x575769940x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
- 2018-09-04 00:01:54,499 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
- 2018-09-04 00:01:54,506 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
- 2018-09-04 00:01:54,564 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bd9a, negotiated timeout = 40000
- Version: 1.2.1
- Number of live region servers: 25
- Number of dead region servers: 0
- Master: dscn11,16000,1529215352324
- Number of backup masters: 1
- Average load: 305.88
- Number of requests: 2507
- Number of regions: 7647
- Number of regions in transition: 0
- 2018-09-04 00:01:56,473 INFO [main] util.HBaseFsck: Loading regionsinfo from the hbase:meta table
- ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
- ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
- Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 2
- 2018-09-04 00:01:57,276 INFO [main] util.HBaseFsck: getHTableDescriptors == tableNames => [DBN_BSHTIndex, DBN_TTKDIndex, SJYB_UPS, SJYB_ANE, DBN_SUR, DBN_CAEIndex, SJYB_BSHT, DBN_STOIndex, DBN_YTO, DBN_TTKD, DBN_FEDEX, SJYB_ZJS, DBN_ANEIndex, DBN_TNTIndex, SJYB_TNT, SJYB_DHL, DBN_ZJS, DBN_KJIndex, BOX, DBN_DEPPONIndex, DBN_ZGYZIndex, DBN_YUNDA, hbase:namespace, DBN_TNT, DBN_FEDEXIndex, DBN_YTOIndex, DBN_PJIndex, SJYB_EMS, SJYB_KJ, SJYB_ZGYZ, DBN_JBD, DBN_DEPPON, SJYB_KYE, SJYB_SUR, SJYB_PJ, SJYB_UC, DBN_SURIndex, DBN_EMS, DBN_STO, SJYB_TTKD, SJYB_SF, SJYB_JBD, SJYB_APEX, DBN_ZTOIndex, DBN_APEX, DBN_PJ, DBN_UPS, DBN_ZJSIndex, DBN_UPSIndex, DBN_UC, DBN_GTO, DBN_BSHT, DBN_EMSIndex, DBN_APEXIndex, SJYB_CAE, DBN_ZTO, DBN_CAE, DBN_SF, DBN_UCIndex, DBN_SFIndex, SJYB_YUNDA, SJYB_FEDEX, SJYB_STO, DBN_JBDIndex, DBN_ANE, DBN_DHLIndex, DBN_KYE, DBN_KJ, SJYB_GTO, DBN_ZGYZ, SJYB_YTO, SJYB_DEPPON, DBN_YUNDAIndex, SJYB_ZTO, DBN_DHL]
- 2018-09-04 00:01:57,280 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x51e8e6e6 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
- 2018-09-04 00:01:57,280 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x51e8e6e60x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
- 2018-09-04 00:01:57,281 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
- 2018-09-04 00:01:57,281 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
- 2018-09-04 00:01:57,283 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bd9c, negotiated timeout = 40000
- 2018-09-04 00:01:57,296 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
- 2018-09-04 00:01:57,296 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bd9c
- 2018-09-04 00:01:57,297 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bd9c closed
- 2018-09-04 00:01:57,298 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
- Number of Tables: 75
- 2018-09-04 00:01:57,308 INFO [main] util.HBaseFsck: Loading region directories from HDFS
- ..............................................................................................................................................................................................................................................................................................................
- 2018-09-04 00:01:59,904 INFO [main] util.HBaseFsck: Loading region information from HDFS
- ............................................................................
- 2018-09-04 00:02:02,282 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x7792d851 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
- 2018-09-04 00:02:02,282 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x7792d8510x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
- 2018-09-04 00:02:02,286 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
- 2018-09-04 00:02:02,287 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
- 2018-09-04 00:02:02,289 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda1, negotiated timeout = 40000
- 2018-09-04 00:02:02,303 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
- 2018-09-04 00:02:02,303 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
- 2018-09-04 00:02:02,304 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
- 2018-09-04 00:02:02,305 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
- 2018-09-04 00:02:02,306 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda2, negotiated timeout = 40000
- 2018-09-04 00:02:02,352 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda2 closed
- 2018-09-04 00:02:02,352 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
- 2018-09-04 00:02:02,352 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bda1
- 2018-09-04 00:02:02,354 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda1 closed
- 2018-09-04 00:02:02,354 INFO [main] util.HBaseFsck: Checking and fixing region consistency
- 2018-09-04 00:02:02,354 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
- ERROR: Region { meta => null, hdfs => hdfs://ns/hbase/data/default/DBN_ANE/95a27c1ae2df5db14fa66d13966a8daf, deployed => dscn33,16020,1535968743109;DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf., replicaId => 0 } not in META, but deployed on dscn33,16020,1535968743109
- ERROR: Region { meta => null, hdfs => hdfs://ns/hbase/data/default/DBN_ANE/e051ccd37678ff95fdd7652106b1fd5b, deployed => dscn33,16020,1535968743109;DBN_ANE,97,1535968864242.e051ccd37678ff95fdd7652106b1fd5b., replicaId => 0 } not in META, but deployed on dscn33,16020,1535968743109
- 2018-09-04 00:02:02,733 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
- ERROR: There is a hole in the region chain between 97 and 97581111303371. You need to create a new .regioninfo and region dir in hdfs to plug the hole.
- ERROR: Found inconsistency in table DBN_ANE
- 2018-09-04 00:02:02,876 INFO [main] util.HBaseFsck: Computing mapping of all store files
- ...................................................................................................................................................................................................................................................................................................................................................................................
- 2018-09-04 00:02:10,477 INFO [main] util.HBaseFsck: Validating mapping using HDFS state
- 2018-09-04 00:02:10,662 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
- 2018-09-04 00:02:10,662 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
- 2018-09-04 00:02:10,664 INFO [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn11/10.11.2.4:2181. Will not attempt to authenticate using SASL (unknown error)
- 2018-09-04 00:02:10,665 INFO [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Socket connection established to dscn11/10.11.2.4:2181, initiating session
- 2018-09-04 00:02:10,666 INFO [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn11/10.11.2.4:2181, sessionid = 0x656ba3c1b1c04f, negotiated timeout = 40000
- 2018-09-04 00:02:10,714 INFO [main] zookeeper.ZooKeeper: Session: 0x656ba3c1b1c04f closed
- 2018-09-04 00:02:10,714 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
- 2018-09-04 00:02:10,714 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
- 2018-09-04 00:02:10,715 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
- 2018-09-04 00:02:10,717 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
- 2018-09-04 00:02:10,717 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
- 2018-09-04 00:02:10,719 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda7, negotiated timeout = 40000
- 2018-09-04 00:02:10,752 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda7 closed
- Summary:
- Table DBN_ANE is okay.
- Number of regions: 147
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_CAE is okay.
- Number of regions: 100
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_DHL is okay.
- Number of regions: 100
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn35,16020,1534221202496
- Table DBN_EMS is okay.
- Number of regions: 226
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,15342212024962018-09-04 00:02:10,754 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
- Table DBN_GTO is okay.
- Number of regions: 51
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn35,16020,1534221202496
- Table DBN_JBD is okay.
- Number of regions: 238
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_KYE is okay.
- Number of regions: 51
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_STO is okay.
- Number of regions: 440
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_SUR is okay.
- Number of regions: 100
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_TNT is okay.
- Number of regions: 100
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_UPS is okay.
- Number of regions: 100
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_YTO is okay.
- Number of regions: 638
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_ZJS is okay.
- Number of regions: 102
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_ZTO is okay.
- Number of regions: 867
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table SJYB_ANE is okay.
- Number of regions: 43
- Deployed on: dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table SJYB_CAE is okay.
- Number of regions: 21
- Deployed on: dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table SJYB_DHL is okay.
- Number of regions: 21
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table SJYB_EMS is okay.
- Number of regions: 90
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table SJYB_GTO is okay.
- Number of regions: 21
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496
- Table SJYB_JBD is okay.
- Number of regions: 46
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn35,16020,1534221202496
- Table SJYB_KYE is okay.
- Number of regions: 21
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn31,16020,1529215376767 dscn35,16020,1534221202496
- Table SJYB_STO is okay.
- Number of regions: 111
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table SJYB_SUR is okay.
- Number of regions: 21
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn35,16020,1534221202496
- Table SJYB_TNT is okay.
- Number of regions: 21
- Deployed on: dscn11,16020,1535969281236 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn26,16020,1535018986330 dscn31,16020,1529215376767 dscn34,16020,1535675408409
- Table SJYB_UPS is okay.
- Number of regions: 21
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn29,16020,1535540080040 dscn31,16020,1529215376767
- Table SJYB_YTO is okay.
- Number of regions: 111
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table SJYB_ZJS is okay.
- Number of regions: 21
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table SJYB_ZTO is okay.
- Number of regions: 137
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_TNTIndex is okay.
- Number of regions: 1
- Deployed on: dscn34,16020,1535675408409
- Table SJYB_KJ is okay.
- Number of regions: 21
- Deployed on: dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn34,16020,1535675408409
- Table SJYB_PJ is okay.
- Number of regions: 33
- Deployed on: dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496
- Table SJYB_SF is okay.
- Number of regions: 92
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table SJYB_UC is okay.
- Number of regions: 36
- Deployed on: dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109
- Table DBN_DEPPONIndex is okay.
- Number of regions: 1
- Deployed on: dscn20,16020,1529215363444
- Table DBN_SURIndex is okay.
- Number of regions: 1
- Deployed on: dscn14,16020,1535088098764
- Table DBN_BSHTIndex is okay.
- Number of regions: 2
- Deployed on: dscn31,16020,1529215376767
- Table DBN_APEX is okay.
- Number of regions: 200
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_BSHT is okay.
- Number of regions: 439
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_TTKD is okay.
- Number of regions: 267
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_ZGYZ is okay.
- Number of regions: 302
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table hbase:meta is okay.
- Number of regions: 1
- Deployed on: dscn35,16020,1534221202496
- Table DBN_ANEIndex is okay.
- Number of regions: 1
- Deployed on: dscn13,16020,1529215824290
- Table DBN_DEPPON is okay.
- Number of regions: 145
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_DHLIndex is okay.
- Number of regions: 1
- Deployed on: dscn26,16020,1535018986330
- Table DBN_SFIndex is okay.
- Number of regions: 2
- Deployed on: dscn17,16020,1534932216753
- Table DBN_YTOIndex is okay.
- Number of regions: 2
- Deployed on: dscn30,16020,1533181074642
- Table DBN_FEDEX is okay.
- Number of regions: 100
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_YUNDA is okay.
- Number of regions: 499
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table SJYB_APEX is okay.
- Number of regions: 21
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn32,16020,1535798049804
- Table SJYB_BSHT is okay.
- Number of regions: 118
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table SJYB_TTKD is okay.
- Number of regions: 86
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table SJYB_ZGYZ is okay.
- Number of regions: 38
- Deployed on: dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409
- Table DBN_CAEIndex is okay.
- Number of regions: 1
- Deployed on: dscn20,16020,1529215363444
- Table BOX is okay.
- Number of regions: 141
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_JBDIndex is okay.
- Number of regions: 2
- Deployed on: dscn14,16020,1535088098764
- Table DBN_TTKDIndex is okay.
- Number of regions: 1
- Deployed on: dscn20,16020,1529215363444
- Table hbase:namespace is okay.
- Number of regions: 1
- Deployed on: dscn12,16020,1533367433799
- Table DBN_KJIndex is okay.
- Number of regions: 1
- Deployed on: dscn21,16020,1529215364453
- Table DBN_ZJSIndex is okay.
- Number of regions: 1
- Deployed on: dscn18,16020,1535022020180
- Table DBN_UCIndex is okay.
- Number of regions: 1
- Deployed on: dscn14,16020,1535088098764
- Table DBN_PJIndex is okay.
- Number of regions: 1
- Deployed on: dscn18,16020,1535022020180
- Table DBN_FEDEXIndex is okay.
- Number of regions: 2
- Deployed on: dscn26,16020,1535018986330
- Table DBN_ZGYZIndex is okay.
- Number of regions: 1
- Deployed on: dscn32,16020,1535798049804
- Table DBN_ZTOIndex is okay.
- Number of regions: 2
- Deployed on: dscn23,16020,1529215366710 dscn33,16020,1535968743109
- Table SJYB_DEPPON is okay.
- Number of regions: 33
- Deployed on: dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496
- Table DBN_APEXIndex is okay.
- Number of regions: 1
- Deployed on: dscn16,16020,1534933465007
- Table DBN_EMSIndex is okay.
- Number of regions: 2
- Deployed on: dscn15,16020,1529371666615
- Table DBN_YUNDAIndex is okay.
- Number of regions: 2
- Deployed on: dscn13,16020,1529215824290
- Table SJYB_FEDEX is okay.
- Number of regions: 21
- Deployed on: dscn11,16020,1535969281236 dscn13,16020,1529215824290 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409
- Table DBN_STOIndex is okay.
- Number of regions: 2
- Deployed on: dscn24,16020,1535104303466
- Table SJYB_YUNDA is okay.
- Number of regions: 102
- Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_KJ is okay.
- Number of regions: 119
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_PJ is okay.
- Number of regions: 164
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_SF is okay.
- Number of regions: 457
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_UC is okay.
- Number of regions: 210
- Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
- Table DBN_UPSIndex is okay.
- Number of regions: 2
- Deployed on: dscn15,16020,1529371666615
- 5 inconsistencies detected.
- Status: INCONSISTENT
- 2018-09-04 00:02:10,770 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
- 2018-09-04 00:02:10,771 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bd9a
- 2018-09-04 00:02:10,772 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bd9a closed
- 2018-09-04 00:02:10,773 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
- [root@dscn12 ~]#
==原因分析==
从hbase hbck检查结果中可以看到以下关键信息
- ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
- ERROR: There is a hole in the region chain between 97 and 97581111303371. You need to create a new .regioninfo and region dir in hdfs to plug the hole.
错误提示了hbase meta数据存在问题，缺失的rowkey范围边界，也就是hole（Region之间有空隙，正常的Region之间是无交叉、无缝隙的）。
导致HBase整个集群状态出现问题，影响Storm的读写处理。
==解决过程==
以前有位大神给了在线hbck修复的运维命令
- 前提：HDFS fsck确保hbase根目录下文件没有损坏丢失，如果有，则先进行corrupt block移除。
- 切记：一定要在所有Region都上线之后再修复，否则修复之后可能出现重复Region。
- 步骤1. hbase hbck 检查输出所以ERROR信息，每个ERROR都会说明错误信息。
- 步骤2. hbase hbck -fixTableOrphans 先修复tableinfo缺失问题，根据内存cache或者hdfs table 目录结构，重新生成tableinfo文件。
- 步骤3. hbase hbck -fixHdfsOrphans 修复regioninfo缺失问题，根据region目录下的hfile重新生成regioninfo文件。
- 步骤4. hbase hbck -fixHdfsOverlaps 修复region重叠问题，merge重叠的region为一个region目录，并从新生成一个regioninfo。
- 步骤5. hbase hbck -fixHdfsHoles 修复region缺失，利用缺失的rowkey范围边界，生成新的region目录以及regioninfo填补这个空洞。
- 步骤6. hbase hbck -fixMeta 修复meta表信息，利用regioninfo信息，重新生成对应meta row填写到meta表中，并为其填写默认的分配regionserver。
- 步骤7. hbase hbck -fixAssignments 把这些offline的region触发上线，当region开始重新open 上线的时候，会被重新分配到真实的RegionServer上 , 并更新meta表上对应的行信息。
我尝试执行了两个命令：
- hbase hbck -fixHdfsHoles
- hbase hbck -fixAssignments -fixMete
重新运行hbase hbck之后，集群恢复了正常。
==经验总结==
hbase hbck检查发现集群不健康的时候，
尝试确诊不健康的原因（有hole？regioninfo缺失？或是tableinfo缺失？）
选择适合的在线修复命令修复HBase，
修复之后，再次执行hbase hbck命令，
当检查结果为下面这种状态时，hbase为健康状态：
（HBase集群不需要重启，因为是在线修复命令）
--END--
【转】一次HBase问题的解决过程（Status: INCONSISTENT）的更多相关文章
- 一次HBase问题的解决过程（Status: INCONSISTENT）
==版本信息== HBase:2.7.1 Storm:1.0.1 RocketMQ:3.4.6(阿里版) ==问题描述== 2018年9月3号晚上23点左右,例行巡检系统运行状况时, 发现Storm消 ...
- .NET平台开源项目速览(7)关于NoSQL数据库LiteDB的分页查询解决过程
在文章:这些.NET开源项目你知道吗?让.NET开源来得更加猛烈些吧!(第二辑) 与 .NET平台开源项目速览(3)小巧轻量级NoSQL文件数据库LiteDB中,介绍了LiteDB的基本使用情况以及部 ...
- 启动Tomcat一闪而过——分析及解决过程
启动Tomcat一闪而过--分析及解决过程 嗯,昨天将有关JDK的知识稍微整理了一下,现在稍微整理一下有关Tomcat的! 1:Tomcat是什么? Tomcat是当今世界上使用最为广泛的.开源免费的 ...
- 这几天帮一个朋友解决了一点小问题（RF的有些小问题及解决过程）
最近涉猎自动化太少了,以至于都不经常更新了.最近一个朋友在做移动端自动化的时候遇到了一些小问题来找我解决.本人也不是很精通,只是接触的时间长了一点了.下面就是一些问题和解决过程: 1.她刚过来的时候, ...
- nginx负载下站点错误响应会导致其他节点重复响应问题的解决过程
目录 前言 问题来了 问题又来了 问题分析 困惑 转机 后续 前言: 这是我上周工作过程中的一次解决问题的过程.解决的是nginx负载下站点错误响应导致其他节点重复响应. 我在整理这个记叙文时,在给这 ...
- 再记一次w3wp占用CPU过高的解决过程（Dictionary和线程安全）
在此之前项目有发生过两次类似的状况,都得以解决,但最近又会发现偶尔CPU会跑满,虽然之前使用过WinDbg解决过两次问题但人的记忆是不可靠的,今天处理同样问题的时候还是遇到了一些障碍,这一次希望可以记 ...
- 记录linux /bin被误删除的解决过程
1.事因: 执行shell测试时,shell中rm -rf $path/* 变量$path为空,结果执行的命令是rm -rf / 事发时及时ctrl+c中断,导致只有/bin /boot目录删除 2. ...
- 【故障处理】ORA-30012的解决过程
[故障处理]ORA-30012的解决过程 1 BLOG文档结构图 2 前言部分 2.1 导读和注意事项 各位技术爱好者,看完本文后,你可以掌握如下的技能,也可以学到一些其它你所不知道的知识,~O ...
- 记VS2013并行编译导致出错的解决过程
接前一篇,电脑换了新的,系统是64bit的win8系统,先安装了SQLServer2012,再安装VS2010旗舰版,Stop!为什么还是2010?因为2010太经典了,以至于公司的项目还在用它写项目 ...
随机推荐
- 【转载】softmax的log似然代价函数（求导过程）
全文转载自:softmax的log似然代价函数(公式求导) 在人工神经网络(ANN)中,Softmax通常被用作输出层的激活函数.这不仅是因为它的效果好,而且因为它使得ANN的输出值更易于理解.同时, ...
- 51Nod 1831 PN表
打出PN表来 发现合数除16,34,289都是赢 质数除2,17都是输 #include<bits/stdc++.h> using namespace std; bool prime(in ...
- python操作mysql代码讲解（及其实用，未来测试工作主要操作数据库的方法）
pymsql pymsql是Python中操作MySQL的模块,其使用方法和MySQLdb几乎相同. 下载安装 1 pip3 install pymysql 使用操作 1.执行SQL 1 2 3 4 ...
- MySQL 8下忘密码后重置密码
解决方案:1):设置mysql为无密码启动 (修改MySQL的登录设置:vi /etc/my.cnf 在[mysqld]的段中加上一句:skip-grant-table) 2):重新启动mys ...
- maven 项目下 Maven Dependencies 下列表为空
问题如题,如下图: 解决: 选中 Maven Dependencies ,右键 属性 如下图: 把 resolve dependencies from workspace projects 这 ...
- Junit(手动/自动)加载
ssm中测试service层数据 Junit手动加载配置文件 package com.oukele.bookshop_ssm.service; import org.junit.After; impo ...
- C#静态变量 总结
在C#程序中,没有全局变量的概念,这意味着所有的成员变量只有该类的实例才能操作这些数据,这起到了“信息隐藏”的作用.但有些时候,这样做却不是个明智的选择. 假设我们要定义一个图书类,要求该类能保存图书 ...
- 《剑指offer》算法题第八天
今日题目(对应书上第39~42题): 数组中出现次数超过一半的数字 最小的k个数(top k,重点!) 数据流中的中位数 连续子数组的最大和 今天的题目都比较经典,特别是第2题. 1. 数组中出现次数 ...
- html5原生js拖拽上传（golang版）
一次只能传一个文件,需在main.go同级目录中建一个upload文件夹 main.go package main import ( "fmt" "io" &q ...
- jupyter工具
国内源安装: pip install jupyter -i http://pypi.douban.com/simple --trusted-host pypi.douban.com pip --def ...