Flume启动时报错Caused by: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.解决办法(图文详解)
前期博客
Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解)
问题详情
-- ::, (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:)] Shutdown Metric for type: SOURCE, name: fileSource. src.events.accepted ==
-- ::, (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:)] Shutdown Metric for type: SOURCE, name: fileSource. src.events.received ==
-- ::, (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:)] Shutdown Metric for type: SOURCE, name: fileSource. src.open-connection.count ==
-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:)] process failed
java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.
at org.apache.flume.sink.hdfs.BucketWriter.checkAndThrowInterruptedException(BucketWriter.java:)
at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.
at org.apache.flume.sink.hdfs.BucketWriter.checkAndThrowInterruptedException(BucketWriter.java:)
at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:)
... more
这句话,说的很明显。
你的hdfs.calltimeout可能设置得太低或HDFS需要花费太长的时间。
我的是
#---------hdfsSink 相关配置------------------
agent1.sinks.hdfsSink.type = hdfs
# 注意, 我们输出到下面一个子文件夹datax中
agent1.sinks.hdfsSink.hdfs.path = hdfs://master:9000/data/types/%Y%m%d/%{log_type}
agent1.sinks.hdfsSink.hdfs.writeFormat = Text
agent1.sinks.hdfsSink.hdfs.fileType = DataStream
agent1.sinks.hdfsSink.hdfs.callTimeout =
agent1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true #当文件大小为52428800字节时,将临时文件滚动成一个目标文件
agent1.sinks.hdfsSink.hdfs.rollSize =
#events数据达到该数量的时候,将临时文件滚动成目标文件
agent1.sinks.hdfsSink.hdfs.rollCount =
#每隔N s将临时文件滚动成一个目标文件
agent1.sinks.hdfsSink.hdfs.rollInterval = #配置前缀和后缀
agent1.sinks.hdfsSink.hdfs.filePrefix=run
这个大家,可以还改大些。
agent1.sinks.hdfsSink.hdfs.callTimeout =
hdfs.sinks.fs.hdfs.rollInterval=21600
hdfs.sinks.fs.hdfs.rollSize=8589934592
为何这两个设置这么大。超时出现问题正常的设置小一点,根据情况自己调试下。
建议初始值:
hdfs.sinks.fs.hdfs.rollInterval=10
hdfs.sinks.fs.hdfs.rollSize=0
依次:
20
40
80
200
等值。
Flume启动时报错Caused by: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.解决办法(图文详解)的更多相关文章
- 【spring boot】【elasticsearch】spring boot整合elasticsearch,启动报错Caused by: java.lang.IllegalStateException: availableProcessors is already set to [8], rejecting [8
spring boot整合elasticsearch, 启动报错: Caused by: java.lang.IllegalStateException: availableProcessors ], ...
- kettle添加hadoop cluster时报错Caused by: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: hadoop:password@node56:9000
完整报错是: Caused by: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: ...
- 【spring cloud】spring cloud 使用feign调用,1.fallback熔断器不起作用,2.启动报错Caused by: java.lang.ClassNotFoundException: com.netflix.hystrix.contrib.javanica.aop.aspectj.Hystri解决
示例GitHub源码地址:https://github.com/AngelSXD/springcloud 1.首先使用feign调用,需要配置熔断器 2.配置熔断器需要将熔断器注入Bean,熔断器类上 ...
- 整合spring boot时操作数据库时报错Caused by: java.lang.InstantiationException: tk.mybatis.mapper.provider.base.B
原文:https://blog.csdn.net/u__f_o/article/details/82756701 一般出现这种情况,应该是没有扫描到对应的mapper包,即在启动类下配置MapperS ...
- springcloud中config启动时候报错Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'config.info' in value "${config.info}"
-noverify -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dcom.sun.management.jm ...
- SpringBoot启动报:Caused by: java.lang.IllegalArgumentException: At least one JPA metamodel must be present!
使用spring boot对项目改造,启动报错: Caused by: java.lang.IllegalArgumentException: At least one JPA metamodel m ...
- Feign调用远程服务报错:Caused by: java.lang.IllegalStateException: Method getMemberInfo not annotated with HTTP method type (ex. GET, POST)
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'ord ...
- javafx:JavaFX Scene Builder 2.0打开含有第三方jar包的fxml文件报错 Caused by: java.lang.ClassNotFoundException
报错如下: java.io.IOException: javafx.fxml.LoadException: /C:/User.................test.fxml at com.orac ...
- 【原创】大叔问题定位分享(8)提交spark任务报错 Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException
spark 2.1.1 一 问题重现 spark-submit --master local[*] --class app.package.AppClass --jars /jarpath/zkcli ...
随机推荐
- ruby on rails 中render的
Ruby rails页面跳转代码如下: 1.render(:text => string) 2.render(:inline => string, [:type => "r ...
- java 多线程系列基础篇(九)之interrupt()和线程终止方式
1. interrupt()说明 在介绍终止线程的方式之前,有必要先对interrupt()进行了解.关于interrupt(),java的djk文档描述如下:http://docs.oracle.c ...
- 11-15SQLserver基础--数据库之范式理论
数据库的设计理论与思路 在设计数据库的时候,有一个著名的设计理论---范式理论. 1.内容: 第一范式:每一列的数据类型要单一,必须要统一: 第二范式:在设计主键的时候,主键尽量更能体现表中的数据信息 ...
- linux命令-mke2fs
想在磁盘下写东西,必须要先格式化 /////////////////////////////////////////////////////////////////////////////////// ...
- jQuery的AJax异步加载
主要用到load()方法以及getScript()方法,具体以一个例子说明: 在现有html文件中加载一个拟好的片段,以及在片段加载完成之前阻止用户进一步操作的弹出框. 首先是现有html代码,无任何 ...
- centos系统查看本机IP地址
centos系统查看本机IP地址,输入 ifconfig -a查看 centos查询上网公网IP输入 curl ifconfig.me 命令即可查看 centos查询上网网关IP,tracepath ...
- cocos打包到ios与android上音频推荐
首先贴一张官方对于ios与android上音频格式的推荐: 这里只给出了推荐格式,一般我们在实际运用中会使用如下方式: 一.IOS与安卓各一套:音乐:都使用MP3 音效:ios用caf Andr ...
- 【转】PEAR安装、管理及使用
PEAR安装 linux下只要你安装的是PHP 4.3.0以上的版本,默认安装都是支持PEAR的,除非你使用了”--WITHOUT-PEAR”选项,修改PHP.INI文件,在INCLUDE_PAT ...
- MSER
1.注释很全的分析:http://blog.csdn.net/zhaocj/article/details/40742191 2.opencv采用的mser实现方法 * 1. the gray ima ...
- php学习笔记-for循环
for(init;condition;statement) { func(); } for循环的执行逻辑是先执行一次init语句,然后判断condition是否为true,是则执行func(),再执行 ...