一次HBase问题的解决过程（Status: INCONSISTENT）
==版本信息==
HBase：2.7.1
Storm：1.0.1
RocketMQ：3.4.6（阿里版）
==问题描述==
2018年9月3号晚上23点左右，例行巡检系统运行状况时，
发现Storm消费RocketMQ入HBase的过程中，RocketMQ的TPS极低，甚至为0了。
（Storm从RocketMQ消费消息自后，读写HBase）
==问题定位==
查看了Storm的日志，发现有如下异常信息：
-- ::43.696 STDIO Thread--BasicBolt-executor[ ] [ERROR] java.net.SocketTimeoutException: callTimeout=, callDuration=: row '' on table 'DBN_ANE' at region=DBN_ANE,,.69724c5bd4e82bae4a3ec8a95c8f9c65., hostname=dscn33,,, seqNum=
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.hadoop.hbase.client.HTable.get(HTable.java:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.hadoop.hbase.client.HTable.get(HTable.java:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at com.dscn.topology.BasicBolt.execute(BasicBolt.java:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.storm.daemon.executor$fn__5030$tuple_action_fn__5032.invoke(executor.clj:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.storm.daemon.executor$mk_task_receiver$fn__4951.invoke(executor.clj:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.storm.disruptor$clojure_handler$reify__4465.onEvent(disruptor.clj:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.storm.daemon.executor$fn__5030$fn__5043$fn__5096.invoke(executor.clj:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at clojure.lang.AFn.run(AFn.java:)
-- ::43.697 STDIO Thread--BasicBolt-executor[ ] [ERROR] at java.lang.Thread.run(Thread.java:)
-- ::43.698 STDIO Thread--BasicBolt-executor[ ] [ERROR] Caused by: java.io.IOException: HRegionInfo was null in DBN_ANE, row=keyvalues={DBN_ANE,,.95a27c1ae2df5db14fa66d13966a8daf./info:seqnumDuringOpen//Put/vlen=/seqid=, DBN_ANE,,.95a27c1ae2df5db14fa66d13966a8daf./info:server//Put/vlen=/seqid=, DBN_ANE,,.95a27c1ae2df5db14fa66d13966a8daf./info:serverstartcode//Put/vlen=/seqid=}
-- ::43.698 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:)
-- ::43.698 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:)
-- ::43.698 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:)
-- ::43.698 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:)
-- ::43.698 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:)
-- ::43.698 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:)
-- ::43.698 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:)
-- ::43.698 STDIO Thread--BasicBolt-executor[ ] [ERROR] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:)
-- ::43.698 STDIO Thread--BasicBolt-executor[ ] [ERROR] ... more
然后查看HBase集群日志，里面也出现错误，日志如下：
-- ::, WARN [MemStoreFlusher.] regionserver.HStore: Failed flushing store file, retrying num=
java.io.IOException: All datanodes 10.11.2.31: are bad. Aborting...
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:)
-- ::, INFO [regionserver/dscn33/10.11.2.31:.logRoller] wal.FSHLog: Slow sync cost: ms, current pipeline: [10.11.2.31:]
-- ::, ERROR [regionserver/dscn33/10.11.2.31:.logRoller] wal.FSHLog: Failed close of WAL writer hdfs://ns/hbase/WALs/dscn33,16020,1529215379398/dscn33%2C16020%2C1529215379398.default.1535968646203, unflushedEntries=30
org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=, requesting roll of WAL
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=, requesting roll of WAL
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
... more
Caused by: java.io.IOException: All datanodes 10.11.2.31: are bad. Aborting...
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:)
-- ::, ERROR [MemStoreFlusher.] regionserver.MemStoreFlusher: Cache flush failed for region DBN_JBD,,.fca3d8ee774898302a377535e3f0ccac.
org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=, requesting roll of WAL
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: java.io.IOException: All datanodes 10.11.2.31: are bad. Aborting...
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:)
检查HBase集群状态（hbase hbck），发现异常，定位为HBase集群问题：
[root@dscn12 ~]# hbase hbck
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadmin/hbase/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBaseFsck command line options:
-- ::, INFO [main] util.HBaseFsck: Launching hbck
-- ::, INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x57576994 connecting to ZooKeeper ensemble=dscn11:,dscn12:,dscn13:
-- ::, INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.-, built on // : GMT
-- ::, INFO [main] zookeeper.ZooKeeper: Client environment:host.name=dscn12
-- ::, INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_73
-- ::, INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
-- ::, INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1..0_73/jre
-- ::, INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/home/hadmin/hbase/conf:/usr/java/default/lib/tools.jar:/home/hadmin/hbase:/home/hadmin/hbase/lib/activation-1.1.jar:/home/hadmin/hbase/lib/antisamy-1.4..jar:/home/hadmin/hbase/lib/aopalliance-1.0.jar:/home/hadmin/hbase/lib/apacheds-i18n-2.0.-M15.jar:/home/hadmin/hbase/lib/apacheds-kerberos-codec-2.0.-M15.jar:/home/hadmin/hbase/lib/api-asn1-api-1.0.-M20.jar:/home/hadmin/hbase/lib/api-util-1.0.-M20.jar:/home/hadmin/hbase/lib/asm-3.1.jar:/home/hadmin/hbase/lib/avro-1.7..jar:/home/hadmin/hbase/lib/batik-css-1.7.jar:/home/hadmin/hbase/lib/batik-ext-1.7.jar:/home/hadmin/hbase/lib/batik-util-1.7.jar:/home/hadmin/hbase/lib/bsh-core-.0b4.jar:/home/hadmin/hbase/lib/commons-beanutils-1.7..jar:/home/hadmin/hbase/lib/commons-beanutils-core-1.7..jar:/home/hadmin/hbase/lib/commons-cli-1.2.jar:/home/hadmin/hbase/lib/commons-codec-1.9.jar:/home/hadmin/hbase/lib/commons-collections-3.2..jar:/home/hadmin/hbase/lib/commons-compress-1.4..jar:/home/hadmin/hbase/lib/commons-configuration-1.6.jar:/home/hadmin/hbase/lib/commons-daemon-1.0..jar:/home/hadmin/hbase/lib/commons-digester-1.8.jar:/home/hadmin/hbase/lib/commons-el-1.0.jar:/home/hadmin/hbase/lib/commons-fileupload-1.2.jar:/home/hadmin/hbase/lib/commons-httpclient-3.1.jar:/home/hadmin/hbase/lib/commons-io-2.4.jar:/home/hadmin/hbase/lib/commons-lang-2.6.jar:/home/hadmin/hbase/lib/commons-logging-1.2.jar:/home/hadmin/hbase/lib/commons-math-2.2.jar:/home/hadmin/hbase/lib/commons-math3-3.1..jar:/home/hadmin/hbase/lib/commons-net-3.1.jar:/home/hadmin/hbase/lib/disruptor-3.3..jar:/home/hadmin/hbase/lib/esapi-2.1..jar:/home/hadmin/hbase/lib/findbugs-annotations-1.3.-.jar:/home/hadmin/hbase/lib/guava-12.0..jar:/home/hadmin/hbase/lib/guice-3.0.jar:/home/hadmin/hbase/lib/guice-servlet-3.0.jar:/home/hadmin/hbase/lib/hadoop-annotations-2.5..jar:/home/hadmin/hbase/lib/hadoop-auth-2.5..jar:/home/hadmin/hbase/lib/hadoop-client-2.5..jar:/home/hadmin/hbase/lib/hadoop-common-2.5..jar:/home/hadmin/hbase/lib/hadoop-hdfs-2.5..jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-app-2.5..jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-common-2.5..jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-core-2.5..jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-jobclient-2.5..jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-shuffle-2.5..jar:/home/hadmin/hbase/lib/hadoop-yarn-api-2.5..jar:/home/hadmin/hbase/lib/hadoop-yarn-client-2.5..jar:/home/hadmin/hbase/lib/hadoop-yarn-common-2.5..jar:/home/hadmin/hbase/lib/hadoop-yarn-server-common-2.5..jar:/home/hadmin/hbase/lib/hbase-annotations-1.2..jar:/home/hadmin/hbase/lib/hbase-annotations-1.2.-tests.jar:/home/hadmin/hbase/lib/hbase-client-1.2..jar:/home/hadmin/hbase/lib/hbase-common-1.2..jar:/home/hadmin/hbase/lib/hbase-common-1.2.-tests.jar:/home/hadmin/hbase/lib/hbase-examples-1.2..jar:/home/hadmin/hbase/lib/hbase-external-blockcache-1.2..jar:/home/hadmin/hbase/lib/hbase-hadoop2-compat-1.2..jar:/home/hadmin/hbase/lib/hbase-hadoop-compat-1.2..jar:/home/hadmin/hbase/lib/hbase-it-1.2..jar:/home/hadmin/hbase/lib/hbase-it-1.2.-tests.jar:/home/hadmin/hbase/lib/hbase-prefix-tree-1.2..jar:/home/hadmin/hbase/lib/hbase-procedure-1.2..jar:/home/hadmin/hbase/lib/hbase-protocol-1.2..jar:/home/hadmin/hbase/lib/hbase-resource-bundle-1.2..jar:/home/hadmin/hbase/lib/hbase-rest-1.2..jar:/home/hadmin/hbase/lib/hbase-server-1.2..jar:/home/hadmin/hbase/lib/hbase-server-1.2.-tests.jar:/home/hadmin/hbase/lib/hbase-shell-1.2..jar:/home/hadmin/hbase/lib/hbase-thrift-1.2..jar:/home/hadmin/hbase/lib/htrace-core-3.1.-incubating.jar:/home/hadmin/hbase/lib/httpclient-4.2..jar:/home/hadmin/hbase/lib/httpcore-4.4..jar:/home/hadmin/hbase/lib/jackson-core-asl-1.9..jar:/home/hadmin/hbase/lib/jackson-jaxrs-1.9..jar:/home/hadmin/hbase/lib/jackson-mapper-asl-1.9..jar:/home/hadmin/hbase/lib/jackson-xc-1.9..jar:/home/hadmin/hbase/lib/jamon-runtime-2.4..jar:/home/hadmin/hbase/lib/jasper-compiler-5.5..jar:/home/hadmin/hbase/lib/jasper-runtime-5.5..jar:/home/hadmin/hbase/lib/javax.inject-.jar:/home/hadmin/hbase/lib/java-xmlbuilder-0.4.jar:/home/hadmin/hbase/lib/jaxb-api-2.2..jar:/home/hadmin/hbase/lib/jaxb-impl-2.2.-.jar:/home/hadmin/hbase/lib/jcodings-1.0..jar:/home/hadmin/hbase/lib/jersey-client-1.9.jar:/home/hadmin/hbase/lib/jersey-core-1.9.jar:/home/hadmin/hbase/lib/jersey-guice-1.9.jar:/home/hadmin/hbase/lib/jersey-json-1.9.jar:/home/hadmin/hbase/lib/jersey-server-1.9.jar:/home/hadmin/hbase/lib/jets3t-0.9..jar:/home/hadmin/hbase/lib/jettison-1.3..jar:/home/hadmin/hbase/lib/jetty-6.1..jar:/home/hadmin/hbase/lib/jetty-sslengine-6.1..jar:/home/hadmin/hbase/lib/jetty-util-6.1..jar:/home/hadmin/hbase/lib/joni-2.1..jar:/home/hadmin/hbase/lib/jruby-complete-1.6..jar:/home/hadmin/hbase/lib/jsch-0.1..jar:/home/hadmin/hbase/lib/jsp-2.1-6.1..jar:/home/hadmin/hbase/lib/jsp-api-2.1-6.1..jar:/home/hadmin/hbase/lib/jsr305-1.3..jar:/home/hadmin/hbase/lib/junit-4.12.jar:/home/hadmin/hbase/lib/leveldbjni-all-1.8.jar:/home/hadmin/hbase/lib/libthrift-0.9..jar:/home/hadmin/hbase/lib/log4j-1.2..jar:/home/hadmin/hbase/lib/metrics-core-2.2..jar:/home/hadmin/hbase/lib/nekohtml-1.9..jar:/home/hadmin/hbase/lib/netty-all-4.0..Final.jar:/home/hadmin/hbase/lib/paranamer-2.3.jar:/home/hadmin/hbase/lib/protobuf-java-2.5..jar:/home/hadmin/hbase/lib/servlet-api-2.5-6.1..jar:/home/hadmin/hbase/lib/servlet-api-2.5.jar:/home/hadmin/hbase/lib/slf4j-api-1.7..jar:/home/hadmin/hbase/lib/slf4j-log4j12-1.7..jar:/home/hadmin/hbase/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hbase/lib/spymemcached-2.11..jar:/home/hadmin/hbase/lib/xalan-2.7..jar:/home/hadmin/hbase/lib/xml-apis-1.3..jar:/home/hadmin/hbase/lib/xml-apis-ext-1.3..jar:/home/hadmin/hbase/lib/xmlenc-0.52.jar:/home/hadmin/hbase/lib/xom-1.2..jar:/home/hadmin/hbase/lib/xz-1.0.jar:/home/hadmin/hbase/lib/zookeeper-3.4..jar:/home/hadmin/hadoop/etc/hadoop:/home/hadmin/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-client-2.7..jar:/home/hadmin/hadoop/share/hadoop/common/lib/gson-2.2..jar:/home/hadmin/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9..jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.-.jar:/home/hadmin/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.-M15.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/httpclient-4.2..jar:/home/hadmin/hadoop/share/hadoop/common/lib/log4j-1.2..jar:/home/hadmin/hadoop/share/hadoop/common/lib/htrace-core-3.1.-incubating.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jets3t-0.9..jar:/home/hadmin/hadoop/share/hadoop/common/lib/zookeeper-3.4..jar:/home/hadmin/hadoop/share/hadoop/common/lib/hadoop-auth-2.7..jar:/home/hadmin/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-api-1.7..jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9..jar:/home/hadmin/hadoop/share/hadoop/common/lib/avro-1.7..jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-recipes-2.7..jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jetty-util-6.1..jar:/home/hadmin/hadoop/share/hadoop/common/lib/protobuf-java-2.5..jar:/home/hadmin/hadoop/share/hadoop/common/lib/hadoop-annotations-2.7..jar:/home/hadmin/hadoop/share/hadoop/common/lib/netty-3.6..Final.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadmin/hadoop/share/hadoop/common/lib/guava-11.0..jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-compress-1.4..jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsch-0.1..jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-beanutils-1.7..jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/api-util-1.0.-M20.jar:/home/hadmin/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.-M20.jar:/home/hadmin/hadoop/share/hadoop/common/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8..jar:/home/hadmin/hadoop/share/hadoop/common/lib/stax-api-1.0-.jar:/home/hadmin/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-xc-1.9..jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-logging-1.1..jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsr305-3.0..jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-collections-3.2..jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-math3-3.1..jar:/home/hadmin/hadoop/share/hadoop/common/lib/jetty-6.1..jar:/home/hadmin/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jaxb-api-2.2..jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/junit-4.11.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9..jar:/home/hadmin/hadoop/share/hadoop/common/lib/mockito-all-1.8..jar:/home/hadmin/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/httpcore-4.2..jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-framework-2.7..jar:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7..jar:/home/hadmin/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.-M15.jar:/home/hadmin/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-common-2.7..jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-common-2.7.-tests.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-nfs-2.7..jar:/home/hadmin/hadoop/share/hadoop/hdfs:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3..jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/log4j-1.2..jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/htrace-core-3.1.-incubating.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0..jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9..jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1..jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5..jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9..jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/netty-3.6..Final.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/guava-11.0..jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/netty-all-4.0..Final.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1..jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jsr305-3.0..jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jetty-6.1..jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9..jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.-tests.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7..jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.-.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/log4j-1.2..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/zookeeper-3.4..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.-tests.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jetty-util-6.1..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/netty-3.6..Final.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guava-11.0..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-compress-1.4..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/stax-api-1.0-.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-logging-1.1..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jsr305-3.0..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-collections-3.2..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jetty-6.1..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9..jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/javax.inject-.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.7..jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7..jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7..jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.7..jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7..jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.7..jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7..jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.7..jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7..jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7..jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.7..jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7..jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/log4j-1.2..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/avro-1.7..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.7..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/netty-3.6..Final.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/javax.inject-.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7..jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.-tests.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7..jar:.::/home/hadmin/hadoop/contrib/capacity-scheduler/*.jar:/home/hadmin/hadoop/bin
2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/home/hadmin/hadoop/lib/native
2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-229.el7.x86_64
2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root
2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/root
2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/root
2018-09-04 00:01:54,474 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x575769940x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:01:54,499 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:01:54,506 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:01:54,564 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bd9a, negotiated timeout = 40000
Version: 1.2.1
Number of live region servers: 25
Number of dead region servers: 0
Master: dscn11,16000,1529215352324
Number of backup masters: 1
Average load: 305.88
Number of requests: 2507
Number of regions: 7647
Number of regions in transition: 0
2018-09-04 00:01:56,473 INFO [main] util.HBaseFsck: Loading regionsinfo from the hbase:meta table
ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 2
2018-09-04 00:01:57,276 INFO [main] util.HBaseFsck: getHTableDescriptors == tableNames => [DBN_BSHTIndex, DBN_TTKDIndex, SJYB_UPS, SJYB_ANE, DBN_SUR, DBN_CAEIndex, SJYB_BSHT, DBN_STOIndex, DBN_YTO, DBN_TTKD, DBN_FEDEX, SJYB_ZJS, DBN_ANEIndex, DBN_TNTIndex, SJYB_TNT, SJYB_DHL, DBN_ZJS, DBN_KJIndex, BOX, DBN_DEPPONIndex, DBN_ZGYZIndex, DBN_YUNDA, hbase:namespace, DBN_TNT, DBN_FEDEXIndex, DBN_YTOIndex, DBN_PJIndex, SJYB_EMS, SJYB_KJ, SJYB_ZGYZ, DBN_JBD, DBN_DEPPON, SJYB_KYE, SJYB_SUR, SJYB_PJ, SJYB_UC, DBN_SURIndex, DBN_EMS, DBN_STO, SJYB_TTKD, SJYB_SF, SJYB_JBD, SJYB_APEX, DBN_ZTOIndex, DBN_APEX, DBN_PJ, DBN_UPS, DBN_ZJSIndex, DBN_UPSIndex, DBN_UC, DBN_GTO, DBN_BSHT, DBN_EMSIndex, DBN_APEXIndex, SJYB_CAE, DBN_ZTO, DBN_CAE, DBN_SF, DBN_UCIndex, DBN_SFIndex, SJYB_YUNDA, SJYB_FEDEX, SJYB_STO, DBN_JBDIndex, DBN_ANE, DBN_DHLIndex, DBN_KYE, DBN_KJ, SJYB_GTO, DBN_ZGYZ, SJYB_YTO, SJYB_DEPPON, DBN_YUNDAIndex, SJYB_ZTO, DBN_DHL]
2018-09-04 00:01:57,280 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x51e8e6e6 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:01:57,280 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x51e8e6e60x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:01:57,281 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:01:57,281 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:01:57,283 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bd9c, negotiated timeout = 40000
2018-09-04 00:01:57,296 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2018-09-04 00:01:57,296 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bd9c
2018-09-04 00:01:57,297 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bd9c closed
2018-09-04 00:01:57,298 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
Number of Tables: 75
2018-09-04 00:01:57,308 INFO [main] util.HBaseFsck: Loading region directories from HDFS
..............................................................................................................................................................................................................................................................................................................
2018-09-04 00:01:59,904 INFO [main] util.HBaseFsck: Loading region information from HDFS
............................................................................
2018-09-04 00:02:02,282 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x7792d851 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:02:02,282 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x7792d8510x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:02:02,286 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:02:02,287 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:02:02,289 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda1, negotiated timeout = 40000
2018-09-04 00:02:02,303 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:02:02,303 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:02:02,304 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:02:02,305 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:02:02,306 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda2, negotiated timeout = 40000
2018-09-04 00:02:02,352 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda2 closed
2018-09-04 00:02:02,352 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2018-09-04 00:02:02,352 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bda1
2018-09-04 00:02:02,354 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda1 closed
2018-09-04 00:02:02,354 INFO [main] util.HBaseFsck: Checking and fixing region consistency
2018-09-04 00:02:02,354 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
ERROR: Region { meta => null, hdfs => hdfs://ns/hbase/data/default/DBN_ANE/95a27c1ae2df5db14fa66d13966a8daf, deployed => dscn33,16020,1535968743109;DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf., replicaId => 0 } not in META, but deployed on dscn33,16020,1535968743109
ERROR: Region { meta => null, hdfs => hdfs://ns/hbase/data/default/DBN_ANE/e051ccd37678ff95fdd7652106b1fd5b, deployed => dscn33,16020,1535968743109;DBN_ANE,97,1535968864242.e051ccd37678ff95fdd7652106b1fd5b., replicaId => 0 } not in META, but deployed on dscn33,16020,1535968743109
2018-09-04 00:02:02,733 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: There is a hole in the region chain between 97 and 97581111303371. You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table DBN_ANE
2018-09-04 00:02:02,876 INFO [main] util.HBaseFsck: Computing mapping of all store files
...................................................................................................................................................................................................................................................................................................................................................................................
2018-09-04 00:02:10,477 INFO [main] util.HBaseFsck: Validating mapping using HDFS state
2018-09-04 00:02:10,662 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:02:10,662 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:02:10,664 INFO [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn11/10.11.2.4:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:02:10,665 INFO [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Socket connection established to dscn11/10.11.2.4:2181, initiating session
2018-09-04 00:02:10,666 INFO [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn11/10.11.2.4:2181, sessionid = 0x656ba3c1b1c04f, negotiated timeout = 40000
2018-09-04 00:02:10,714 INFO [main] zookeeper.ZooKeeper: Session: 0x656ba3c1b1c04f closed
2018-09-04 00:02:10,714 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2018-09-04 00:02:10,714 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:02:10,715 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:02:10,717 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:02:10,717 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:02:10,719 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda7, negotiated timeout = 40000
2018-09-04 00:02:10,752 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda7 closed
Summary:
Table DBN_ANE is okay.
Number of regions: 147
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_CAE is okay.
Number of regions: 100
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_DHL is okay.
Number of regions: 100
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn35,16020,1534221202496
Table DBN_EMS is okay.
Number of regions: 226
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,15342212024962018-09-04 00:02:10,754 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down Table DBN_GTO is okay.
Number of regions: 51
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn35,16020,1534221202496
Table DBN_JBD is okay.
Number of regions: 238
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_KYE is okay.
Number of regions: 51
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_STO is okay.
Number of regions: 440
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_SUR is okay.
Number of regions: 100
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_TNT is okay.
Number of regions: 100
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_UPS is okay.
Number of regions: 100
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_YTO is okay.
Number of regions: 638
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_ZJS is okay.
Number of regions: 102
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_ZTO is okay.
Number of regions: 867
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_ANE is okay.
Number of regions: 43
Deployed on: dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_CAE is okay.
Number of regions: 21
Deployed on: dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_DHL is okay.
Number of regions: 21
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_EMS is okay.
Number of regions: 90
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_GTO is okay.
Number of regions: 21
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496
Table SJYB_JBD is okay.
Number of regions: 46
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn35,16020,1534221202496
Table SJYB_KYE is okay.
Number of regions: 21
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn31,16020,1529215376767 dscn35,16020,1534221202496
Table SJYB_STO is okay.
Number of regions: 111
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_SUR is okay.
Number of regions: 21
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn35,16020,1534221202496
Table SJYB_TNT is okay.
Number of regions: 21
Deployed on: dscn11,16020,1535969281236 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn26,16020,1535018986330 dscn31,16020,1529215376767 dscn34,16020,1535675408409
Table SJYB_UPS is okay.
Number of regions: 21
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn29,16020,1535540080040 dscn31,16020,1529215376767
Table SJYB_YTO is okay.
Number of regions: 111
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_ZJS is okay.
Number of regions: 21
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_ZTO is okay.
Number of regions: 137
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_TNTIndex is okay.
Number of regions: 1
Deployed on: dscn34,16020,1535675408409
Table SJYB_KJ is okay.
Number of regions: 21
Deployed on: dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn34,16020,1535675408409
Table SJYB_PJ is okay.
Number of regions: 33
Deployed on: dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496
Table SJYB_SF is okay.
Number of regions: 92
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_UC is okay.
Number of regions: 36
Deployed on: dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109
Table DBN_DEPPONIndex is okay.
Number of regions: 1
Deployed on: dscn20,16020,1529215363444
Table DBN_SURIndex is okay.
Number of regions: 1
Deployed on: dscn14,16020,1535088098764
Table DBN_BSHTIndex is okay.
Number of regions: 2
Deployed on: dscn31,16020,1529215376767
Table DBN_APEX is okay.
Number of regions: 200
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_BSHT is okay.
Number of regions: 439
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_TTKD is okay.
Number of regions: 267
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_ZGYZ is okay.
Number of regions: 302
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table hbase:meta is okay.
Number of regions: 1
Deployed on: dscn35,16020,1534221202496
Table DBN_ANEIndex is okay.
Number of regions: 1
Deployed on: dscn13,16020,1529215824290
Table DBN_DEPPON is okay.
Number of regions: 145
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_DHLIndex is okay.
Number of regions: 1
Deployed on: dscn26,16020,1535018986330
Table DBN_SFIndex is okay.
Number of regions: 2
Deployed on: dscn17,16020,1534932216753
Table DBN_YTOIndex is okay.
Number of regions: 2
Deployed on: dscn30,16020,1533181074642
Table DBN_FEDEX is okay.
Number of regions: 100
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_YUNDA is okay.
Number of regions: 499
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_APEX is okay.
Number of regions: 21
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn32,16020,1535798049804
Table SJYB_BSHT is okay.
Number of regions: 118
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_TTKD is okay.
Number of regions: 86
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_ZGYZ is okay.
Number of regions: 38
Deployed on: dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409
Table DBN_CAEIndex is okay.
Number of regions: 1
Deployed on: dscn20,16020,1529215363444
Table BOX is okay.
Number of regions: 141
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_JBDIndex is okay.
Number of regions: 2
Deployed on: dscn14,16020,1535088098764
Table DBN_TTKDIndex is okay.
Number of regions: 1
Deployed on: dscn20,16020,1529215363444
Table hbase:namespace is okay.
Number of regions: 1
Deployed on: dscn12,16020,1533367433799
Table DBN_KJIndex is okay.
Number of regions: 1
Deployed on: dscn21,16020,1529215364453
Table DBN_ZJSIndex is okay.
Number of regions: 1
Deployed on: dscn18,16020,1535022020180
Table DBN_UCIndex is okay.
Number of regions: 1
Deployed on: dscn14,16020,1535088098764
Table DBN_PJIndex is okay.
Number of regions: 1
Deployed on: dscn18,16020,1535022020180
Table DBN_FEDEXIndex is okay.
Number of regions: 2
Deployed on: dscn26,16020,1535018986330
Table DBN_ZGYZIndex is okay.
Number of regions: 1
Deployed on: dscn32,16020,1535798049804
Table DBN_ZTOIndex is okay.
Number of regions: 2
Deployed on: dscn23,16020,1529215366710 dscn33,16020,1535968743109
Table SJYB_DEPPON is okay.
Number of regions: 33
Deployed on: dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496
Table DBN_APEXIndex is okay.
Number of regions: 1
Deployed on: dscn16,16020,1534933465007
Table DBN_EMSIndex is okay.
Number of regions: 2
Deployed on: dscn15,16020,1529371666615
Table DBN_YUNDAIndex is okay.
Number of regions: 2
Deployed on: dscn13,16020,1529215824290
Table SJYB_FEDEX is okay.
Number of regions: 21
Deployed on: dscn11,16020,1535969281236 dscn13,16020,1529215824290 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409
Table DBN_STOIndex is okay.
Number of regions: 2
Deployed on: dscn24,16020,1535104303466
Table SJYB_YUNDA is okay.
Number of regions: 102
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_KJ is okay.
Number of regions: 119
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_PJ is okay.
Number of regions: 164
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_SF is okay.
Number of regions: 457
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_UC is okay.
Number of regions: 210
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_UPSIndex is okay.
Number of regions: 2
Deployed on: dscn15,16020,1529371666615
5 inconsistencies detected.
Status: INCONSISTENT
2018-09-04 00:02:10,770 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2018-09-04 00:02:10,771 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bd9a
2018-09-04 00:02:10,772 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bd9a closed
2018-09-04 00:02:10,773 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
[root@dscn12 ~]#
==原因分析==
从hbase hbck检查结果中可以看到以下关键信息
ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
错误提示了hbase meta数据存在问题，缺失的rowkey范围边界，也就是hole（Region之间有空隙，正常的Region之间是无交叉、无缝隙的）。
导致HBase整个集群状态出现问题，影响Storm的读写处理。
==解决过程==
以前有位大神给了在线hbck修复的运维命令
前提：HDFS fsck确保hbase根目录下文件没有损坏丢失，如果有，则先进行corrupt block移除。
切记：一定要在所有Region都上线之后再修复，否则修复之后可能出现重复Region。 步骤1. hbase hbck 检查输出所以ERROR信息，每个ERROR都会说明错误信息。
步骤2. hbase hbck -fixTableOrphans 先修复tableinfo缺失问题，根据内存cache或者hdfs table 目录结构，重新生成tableinfo文件。
步骤3. hbase hbck -fixHdfsOrphans 修复regioninfo缺失问题，根据region目录下的hfile重新生成regioninfo文件。
步骤4. hbase hbck -fixHdfsOverlaps 修复region重叠问题，merge重叠的region为一个region目录，并从新生成一个regioninfo。
步骤5. hbase hbck -fixHdfsHoles 修复region缺失，利用缺失的rowkey范围边界，生成新的region目录以及regioninfo填补这个空洞。
步骤6. hbase hbck -fixMeta 修复meta表信息，利用regioninfo信息，重新生成对应meta row填写到meta表中，并为其填写默认的分配regionserver。
步骤7. hbase hbck -fixAssignments 把这些offline的region触发上线，当region开始重新open 上线的时候，会被重新分配到真实的RegionServer上 , 并更新meta表上对应的行信息。
我尝试执行了两个命令：
hbase hbck -fixHdfsHoles
hbase hbck -fixAssignments -fixMete
重新运行hbase hbck之后，集群恢复了正常。
==经验总结==
hbase hbck检查发现集群不健康的时候，
尝试确诊不健康的原因（有hole？regioninfo缺失？或是tableinfo缺失？）
选择适合的在线修复命令修复HBase，
修复之后，再次执行hbase hbck命令，
当检查结果为下面这种状态时，hbase为健康状态：
（HBase集群不需要重启，因为是在线修复命令）
--END--
一次HBase问题的解决过程（Status: INCONSISTENT）的更多相关文章
- 【转】一次HBase问题的解决过程（Status: INCONSISTENT）
[From]https://www.cnblogs.com/quchunhui/p/9583746.html ==版本信息== HBase:2.7.1 Storm:1.0.1 RocketMQ:3.4 ...
- status 404 reading EduClient#getCourseInfoOrder(String)解决过程
UcenterClient#getUserInfoOrder(String) failed and no fallback available.解决过程 报错内容: com.netflix.hystr ...
- 【故障处理】ORA-30012的解决过程
[故障处理]ORA-30012的解决过程 1 BLOG文档结构图 2 前言部分 2.1 导读和注意事项 各位技术爱好者,看完本文后,你可以掌握如下的技能,也可以学到一些其它你所不知道的知识,~O ...
- 无法SSH服务器的解决过程（openssh-daemon is stopped）
公司某台服务器不知为何无法ssh连接上,进入现场查看: 1.执行netstat -atnlp|grep ssh,没有找到ssh端口 2.执行ps aux|grep ssh,没找到相关进程 3.执行se ...
- 转载：oracle11G 已开启监听，但远程连接依旧无监听解决过程
1.连接数据库显示无监听程序,首先查看服务器的oracle监听服务是否开启,服务名称:OracleOraDb11g_home1TNSListener(具体环境中可能不完全一样,但是认准TNSListe ...
- mysql ab主从复制出错及解决过程
一.mysql主从服务器报错描述:Slave_IO_Running=NO,Slave_SQL_Running=YES,Last_Errno=0 mysql slave stop ; mysql sla ...
- oracle11G 已开启监听，但远程连接依旧无监听解决过程
1.连接数据库显示无监听程序,首先查看服务器的oracle监听服务是否开启,服务名称:OracleOraDb11g_home1TNSListener(具体环境中可能不完全一样,但是认准TNSListe ...
- .NET平台开源项目速览(7)关于NoSQL数据库LiteDB的分页查询解决过程
在文章:这些.NET开源项目你知道吗?让.NET开源来得更加猛烈些吧!(第二辑) 与 .NET平台开源项目速览(3)小巧轻量级NoSQL文件数据库LiteDB中,介绍了LiteDB的基本使用情况以及部 ...
- 启动Tomcat一闪而过——分析及解决过程
启动Tomcat一闪而过--分析及解决过程 嗯,昨天将有关JDK的知识稍微整理了一下,现在稍微整理一下有关Tomcat的! 1:Tomcat是什么? Tomcat是当今世界上使用最为广泛的.开源免费的 ...
随机推荐
- 通过cookie验证用户登录
# cookie # cookie# 当你在浏览器登陆时,浏览器记录这个登录信息(服务器响应时发送请求的数据和登录信息),再次访问时 浏览器会将访问请求和缓存的登录信息都发送到服务器, # 服务器通过 ...
- 1.vue和react的区别
1.个人感觉Vue好用,react不咋地呀. 2.(网上搜的)Vue的解决方案适用于小型应用,但对于对于大型应用而言不太适合.
- win10 + Ubuntu 双系统，重装后的引导修复，时间调整和启动项调整
▶ 原先为 win10 + Ubuntu 双系统,使用 grub2 作引导,在重装了 win10 (大作死升到了1803)后系统重写了引导,启动项里找不到 Ubuntu,需要修复. ● 参考[http ...
- 0_Simple__UnifiedMemoryStreams
使用 OpenMP 和 pthreads 两种环境,利用实现统一内存编址,计算基本的矩阵乘法 result = α * A * x + β * result . ▶ 源代码 #include < ...
- MacBook Pro 一月使用体验
从 2013 年开始,就特别想买 MBP,终于在 2015 年的尾巴用上了 MBPR.原本是要在使用一周后写一份使用体验的,但因为懒,现在拖到一个月了,刚好现在也是2016年的一月,就把标题改成一月使 ...
- hbase命名空间
在HBase中,namespace命名空间指对一组表的逻辑分组,类似于数据库,便于对表在业务上划分 HBase系统默认定义了两个缺省的namespace hbase:系统内建表,包括namespace ...
- 5. window.location.href/replace/reload()--页面跳转+替换+刷新
1.window.location=url; window.location 对象用于获得当前页面的地址 (URL),并把浏览器重定向到新的页面. 一.最外层top跳转页面,适合用于iframe框架集 ...
- 解决maven工程 子工程中的一些配置读取进来的问题
方案:在父工程中手动配置一些节点 <build> <!-- 插件 --> <plugins> <plugin> <groupId>org.a ...
- 微信公众号自动回复 node
纯属分享记录: app.js var bodyParser = require('body-parser'); require('body-parser-xml')(bodyParser); var ...
- JS 时间 获取 当天,昨日,本周,上周,本月,上月
调用 setTimeRange (2); function setTimeRange (v) { var fmt = 'YYYY-MM-DD HH:mm'; var now = new Date() ...