背景说明：

　　今天对生产环境hbase增加了节点，下午的时候一个同事反馈，应用程序后台报错，如下：

  1. Tue Feb 26 17:35:35 CST 2019, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68451: row 'SYSTEM.CATALOG,TARGETCUST_DATA,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host-10-191-36-24,16020,1551146724629, seqNum=0
  2.  
  3.         at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)
  4.         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210)
  5.         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
  6.         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
  7.         at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
  8.         at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
  9.         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1275)
  10.         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
  11.         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)
  12.         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
  13.         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:957)
  14.         at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
  15.         at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:506)
  16.         at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:722)
  17.         at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:692)
  18.         at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1769)
  19.         at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1724)
  20.         at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1704)
  21.         at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1301)
  22.         ... 47 more
  23. Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68451: row 'SYSTEM.CATALOG,TARGETCUST_DATA,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host-10-191-36-24,16020,1551146724629, seqNum=0
  24.         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:169)
  25.         at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
  26.         ... 3 more
  27. Caused by: java.net.UnknownHostException: host-10-191-36-24
  28.         at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.<init>(AbstractRpcClient.java:315)
  29.         at org.apache.hadoop.hbase.ipc.AbstractRpcClient.createBlockingRpcChannel(AbstractRpcClient.java:267)
  30.         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getClient(ConnectionManager.java:1639)
  31.         at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:162)
  32.         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.prepare(ScannerCallableWithReplicas.java:376)
  33.         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
  34.         ... 4 more
  35. 2019-02-26 17:35:35 [com.asiainfo.cb2.consumer.ReceiveSMSID]-[ERROR]:33 - sms consumer begin
  36. 2019-02-26 17:35:35 [com.asiainfo.cb2.consumer.ReceiveSMSID]-[ERROR]:38 - sms consumer pushType3

分析：

　　开始的时候，思路还是纠结在，超时的错误java.net.SocketTimeoutException: callTimeout=60000,想着有没有办法能够增加客户端的超时时间，然后还看了datanode的日志，认为是datanode在写数据导磁盘的时候性能问题，新增加hadoop节点，节点间在进行平衡，导致影响了性能，但是，对于以上的分析都没有更好的方法来解决问题。

结果：

　　等到后续，在仔细看错误，发现，Caused by: java.net.UnknownHostException: host-10-191-36-24，找不到主机的异常，才突然明白，应用程序首先连接到zk，然后zk告知region在哪个regionserver上，然后，应用程序再连接到hbase的regionserver上读写数据。

解决：

　　所以，在应用程序的/etc/hosts上配置新增的hbase regionserver节点的hosts解析，再次观察应用程序，该问题解决

文档创建时间：2019年2月27日10:34:38

应用程序连接hbase报错：java.net.SocketTimeoutException: callTimeout=60000的更多相关文章

  1. JAVA Api 调用Hbase报错锦集

    1. 报错 java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/protobuf/generated/MasterProtos$Master ...

  2. eclipse启动报错java.lang.IllegalStateException: LifecycleProcessor not initialized - call 'refresh' befo

    报错: java.lang.IllegalStateException: LifecycleProcessor not initialized - call 'refresh' before invo ...

  3. 开着idea，死机了，关机重启。重启之后，重新打开idea报错java.lang.AssertionError:upexpected content storage modification

    开着idea,死机了,关机重启.重启之后,重新打开idea报错java.lang.AssertionError:upexpected content storage modification. goo ...

  4. Eclipse启动报错Java was started but returned exit code=13

    启动Eclipse的时候报错Java was started but returned exit code=13,这个错误的原因是由于eclipse版本与jdk版本不符导致的,可能你的eclipse是 ...

  5. idea中编译项目报错 java: javacTask: 源版本 1.8 需要目标版本 1.8

    问题如上面所叙: > idea中编译项目报错 java: javacTask: 源版本 1.8 需要目标版本 1.8 解决方案: > Setting->Compiler->Ja ...

  6. 关于SpringMVC项目报错:java.io.FileNotFoundException: Could not open ServletContext resource [/WEB-INF/xxxx.xml]

    关于SpringMVC项目报错:java.io.FileNotFoundException: Could not open ServletContext resource [/WEB-INF/xxxx ...

  7. maven web 项目中启动报错 Java.lang.ClassNotFoundException: org.springframework.web.servlet.DispatcherServlet

    主要原因是maven项目里面的jar包吗,没有导入到项目中 maven web 项目中启动报错 Java.lang.ClassNotFoundException: org.springframewor ...

  8. zookeeper报错java.net.ConnectException: Connection refused: no further information

    zookeeper报错java.net.ConnectException: Connection refused: no further information 这是在linux 启动 https:/ ...

  9. 【原创】大叔问题定位分享（2）spark任务一定几率报错java.lang.NoSuchFieldError: HIVE_MOVE_FILES_THREAD_COUNT

    最近用yarn cluster方式提交spark任务时,有时会报错,报错几率是40%,报错如下: 18/03/15 21:50:36 116 ERROR ApplicationMaster91: Us ...

随机推荐

  1. 编程菜鸟的日记-初学尝试编程-C++ Primer Plus 第4章编程练习5

    #include <iostream>using namespace std;struct CandyBar{ char kind[20]; float weight; int calor ...

  2. 部署前准备--使用Mysql之Django Debug Toolbar安装以及配置

    python -c "import django ;print(django.__path__);" 查看python的全局配置 vi /usr/local/lib/python3 ...

  3. Grok patterns 汇总

    S3_REQUEST_LINE (?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawreques ...

  4. JAVA自学笔记26

    JAVA自学笔记26 1.网络编程 1)用来实现网络互联的不同计算机上运行的程序可以进行数据交换 2)网络模型一般泛指 OSI:(Open System Interconnection)开放系统互联参 ...

  5. I2C总线协议图解

    原帖地址:https://www.cnblogs.com/aaronLinux/p/6218660.html

  6. Netty 中ChannelOption的含义以及使用的场景

    Netty 中ChannelOption的含义以及使用的场景 转自:http://www.cnblogs.com/googlemeoften/p/6082785.html 1.ChannelOptio ...

  7. 十进制转为x进制的递归代码

    十进制转为x进制的递归代码 #include <stdio.h> void fun(int n,int x) { ) return; else { fun(n/x,x); printf(& ...

  8. Springboot整合二 集成 rabbitmq

    1.在application.yml文件中进行RabbitMQ的相关配置先上代码 spring: rabbitmq: host: 192168.21.11 port: username: guest ...

  9. Python分页组件

    分页组件的实现: class Pagination(object): """ 自定义分页 """ def __init__(self,cur ...

  10. Android Studio创建JAR/AAR库

    [时间:2017-09] [状态:Open] [关键词:Android,Android Studio,gradle,jar,aar,library] 0 引言 最近在工作中遇到了升级Android S ...