今天在本地测试flume的exec  监控文件   分割的问题!!!遇到各种141异常问题!

怀疑是在切割文件的时候超过了监控文本的时间,导致flume异常退出,,,所以增加了keep-alive 时长,,,他的默认值是3秒,,我把它设置为30秒,,,之后运行,,,,他不再异常!!!

解决:设置agent1.channels.<channel_name>.keep-alive = 30

参考文章:问题2,,,,虽然前边的agent,方式可能不一样,但是这个关键的时间是一样的。

-------------------------------------------------以下是原文,原文地址:http://www.tuicool.com/articles/mmm2AvF

flume 问题分析与处理

时间 2014-03-30 11:50:18  CSDN博客
主题 Flume

问题一:

org.apache.flume.EventDeliveryException:Failed to send events

atorg.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:382)

atorg.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

at java.lang.Thread.run(Thread.java:722)

Caused by:org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host:10.95.198.123, port: 44444 }: Failed to send batch

at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:294)

atorg.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:366)

... 3 more

Caused by:org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host:10.95.198.123, port: 44444 }: Avro RPC call returned Status: FAILED

atorg.apache.flume.api.NettyAvroRpcClient.waitForStatusOK(NettyAvroRpcClient.java:370)

分析:

代码分析

try { 
      appendBatch(events, requestTimeout, TimeUnit.MILLISECONDS); 
    } catch (Throwable t) { 
      // we mark as no longer active without trying to clean up resources 
      // client is required to call close() to clean up resources 
      setState(ConnState.DEAD); 
      if (t instanceof Error) { 
        throw (Error) t; 
      } 
      if (t instanceof TimeoutException) { 
        throw new EventDeliveryException(this + ": Failed to send event. " + 
            "RPC request timed out after " + requestTimeout + " ms", t); 
      } 
      throw new EventDeliveryException(this + ": Failed to send batch", t);

请求超时,导致发送event失败

解决:

设置request-timeout长一点,默认20秒

问题二:

org.apache.flume.ChannelException: Unableto put batch on required channel: org.apache.flume.channel.MemoryChannel{name:woStoreSoftWDownloadC2}

atorg.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)

atorg.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:376)

atorg.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:336)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

at java.util.concurrent.FutureTask.run(FutureTask.java:166)

atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

at java.lang.Thread.run(Thread.java:722)

Caused by:org.apache.flume.ChannelException: Space for commit to queue couldn't beacquired Sinks are likely not keeping up with sources, or the buffer size istoo tight

atorg.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)

atorg.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)

atorg.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)

... 8 more

30 Mar 2014 10:16:00,960 ERROR[timedFlushExecService18-0](org.apache.flume.source.ExecSource$ExecRunnable$1.run:322)  - Exception occured when processing eventbatch

org.apache.flume.ChannelException: Unableto put batch on required channel: org.apache.flume.channel.MemoryChannel{name:woStoreSoftWDownloadC2}

atorg.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)

atorg.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:376)

atorg.apache.flume.source.ExecSource$ExecRunnable.access$100(ExecSource.java:249)

at org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:318)

atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

atjava.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)

at

代码分析:

protected void doCommit() throws InterruptedException { 
      int remainingChange = takeList.size() - putList.size(); 
      if(remainingChange < 0) { 
        if(!queueRemaining.tryAcquire(-remainingChange, keepAlive, TimeUnit.SECONDS)) { 
          throw new ChannelException("Space for commit to queue couldn't be acquired" + 
              " Sinks are likely not keeping up with sources, or the buffer size is too tight"); 
        } 
      } 
      int puts = putList.size(); 
      int takes = takeList.size(); 
      synchronized(queueLock) { 
        if(puts > 0 ) { 
          while(!putList.isEmpty()) { 
            if(!queue.offer(putList.removeFirst())) { 
              throw new RuntimeException("Queue add failed, this shouldn't be able to happen"); 
            } 
          } 
        } 
        putList.clear(); 
        takeList.clear(); 
      } 
      queueStored.release(puts); 
      if(remainingChange > 0) { 
        queueRemaining.release(remainingChange); 
      } 
      if (puts > 0) { 
        channelCounter.addToEventPutSuccessCount(puts); 
      } 
      if (takes > 0) { 
        channelCounter.addToEventTakeSuccessCount(takes); 
      } 
      channelCounter.setChannelSize(queue.size()); 
    }

等待keep-alive之后,还是没办法插入event

解决方案:

设置keep-alive(默认3秒) , capacity(100), transactionCapacity(100)大一点

-----------------------------------------------------------------------------------------------

之后,前边的异常少了,,后边的异常是,,,产生消息快,命令消费慢!!!导致管道满!!!!

这个就需要对端加大管道的消费速度,来调整???,,,,,怀疑之,,,然后增大空间和传输空间!

-----------更改结果如下!

stage_nginx.channels.M1.capacity = 1000
stage_nginx.channels.M1.transactionCapacity = 1000
stage_nginx.channels.M1.keep-alive = 30

---------

解决flume运行中的一个异常问题!的更多相关文章

  1. 解决div布局中第一个div的margin-top在浏览器中显示无效果问题。

    原味来源:http://www.hicss.net/do-not-tell-me-you-understand-margin/ 垂直外边距合并问题 别被上面这个名词给吓倒了,简单地说,外边距合并指的是 ...

  2. java程序运行中如果出现异常未被处理,将会被抛到java虚拟机进行处理,程序中断运行后被挂起,在页面输出错误信息(不会输出到console)

    下面的代码中,因为我是使用 for (Iterator<Element> i = el.elements().iterator(); i.hasNext(); ) 迭代器遍历根节点的所有子 ...

  3. 自动添加菜品,加入运行中遇到的异常,生成日志文件...<工作中场景...>

    """ 很弱智的小脚本,记录下.也许以后看到会笑,因为太幼稚或者证明曾经也努力过.so... """ """ ...

  4. java中自定义一个异常类 在某些情况抛出自定的异常 ----------阻断程序

    //=============定义异常类 package org.springblade.flow.engine.errorException; /** * 自定义异常处理写入sap失败 */ pub ...

  5. 解决spark运行中failed to locate the winutils binary in the hadoop binary path的问题

    1.下载hadoop-common-2.2.0-bin并解压到某个目录 https://github.com/srccodes/hadoop-common-2.2.0-bin 2.设置hadoop.h ...

  6. 完美解决在Servlet中出现一个输出中文乱码的问题

    @Override public void doPost(HttpServletRequest reqeust, HttpServletResponse response) throws Servle ...

  7. 一个解决在非UI线程中访问UI 异常的小方法

    写 WPF 的童鞋可能都会碰到 在非UI线程中访问 UI 异常的问题.这是为了防止数据不一致做的安全限制. 子线程中更新UI还要交给主线程更新,引用满天飞,实在是麻烦. 接下来,我们推出一个可以称之为 ...

  8. Emgu CV的一个异常的解决方法

    今年组里有大项目落我头上了,并不能像去年一样回家还能搞搞Cocos2dX,一把老泪流了下来... 回到正题,由于组里需要做一个显示板的自动测试项目,涉及到Computer Vision.不得不说,这才 ...

  9. 让 DolphinScheduler 1.3.4 开启 Netty 日志打印,解决流程实例一直在运行中的问题

    关于新一代大数据任务调度 - Apache DolphinScheduler   Apache DolphinScheduler(incubator) 于 17 年在易观数科立项, 19 年 8 月进 ...

随机推荐

  1. Centos安装FTP服务器和配置

    安装 yum install vsftpd 启动/重启/关闭 /sbin/service vsftpd start /sbin/service vsftpd restart /sbin/service ...

  2. spark0.9分布式安装

    http://blog.csdn.net/myboyliu2007/article/details/18990277 spark安装包:spark-0.9.0-incubating-bin-hadoo ...

  3. (转) Unity3D 使用Texturepacker打包工具制作NGUI(Atlas)图集

    转自:http://www.unitymanual.com/thread-37485-1-1.html 由于NGUI AtlasMaker对打包的优化不好,容易打出很大的图集,很多部分都是浪费的,所有 ...

  4. SharePoint CAML Query小结

    CAML的结构. <View Type="HTML" Name="Summary"> <ViewBody ExpandXML="TR ...

  5. 【Python】使用内置base64模块进行编解码

    代码: import hashlib import base64 hash = hashlib.md5() hash.update('逆火Tu22m'.encode('utf-8')) print(h ...

  6. Android studio 自己定义打包APK名称

    Android Studio打包应用默认生成的apk名称是:app-release.apk .假设我们要让生成的apk名跟我们版本号包名有联系的话.那我们就要自己定义生成的apk名了,要怎么做呢. 我 ...

  7. 线程:主线程、子线程 同步线程、异步线程 单线程、多线程 System.Threading与System.Windows.Threading

    入门-------------------------------------------------------------------------------- 概述与概念    一个C#程序开始 ...

  8. MicrosoftAjax.js

    MicrosoftAjax.js下载 Function.__typeName = "Function"; Function.__class = true; Function.cre ...

  9. setTimeout迭代替换setInterval

    一.它们之间的区别 setTimeout - 仅执行一次 setInterval - 间隔执行     二.为什么推荐用setTimeout替换掉setIntelval?   javascript是异 ...

  10. 微信小程序 - template和include详细描述

    演示 index.wxml <!-- 内部模板 --> <template name="all"> {{a}} {{b}} </template> ...