flume在抽取MySQL数据到kafka时报错，如下

[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.kafka.KafkaSink - Failed to publish events

org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity  full, consider committing more frequently, increasing capacity, or increasing thread count

    at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)

    at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)

    at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)

    at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)

    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)

    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)

    at java.lang.Thread.run(Thread.java:)

[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.

org.apache.flume.EventDeliveryException: Failed to publish events

    at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)

    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)

    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)

    at java.lang.Thread.run(Thread.java:)

Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity  full, consider committing more frequently, increasing capacity, or increasing thread count

    at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)

    at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)

    at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)

    at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)

    ...  more


MemoryChannel的事务的Put队列满

解决:将channel的capacity的值调大即可

a1.channels.ch-.capacity = 
注意：Flume配置每行后边不能加注释，否则会导致配置不生效

org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count的更多相关文章

  1. Put queue for MemoryTransaction of capacity 10000 full, consider committing more frequently, increasing capacity or increasing thread count flume capacity 时间数

    package com.test; import org.apache.http.*;import org.apache.http.entity.ContentType;import org.apac ...

  2. 一个Flume 异常（Put queue for MemoryTransaction of capacity 100 full）的排查和解决思路

    最近在做一个分布式调用链跟踪系统, 在两个地方采用了flume (我使用的flume版本是1.5.0-cdh5.4.4),一个是宿主系统 ,用flume agent进行日志搜集. 一个是从kafka拉 ...

  3. Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法（图文详解）

        前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情 启动agent服务 [hadoop@master flume-1.7.0]$ ...

  4. Apache Flume 学习笔记

    # 从http://flume.apache.org/download.html 下载flume ############################################# # 概述: ...

  5. Apache Flume 1.7.0 发布，日志服务器

    Apache Flume 1.7.0 发布了,Flume 是一个分布式.可靠和高可用的服务,用于收集.聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型.这是一个可靠.容错的服务. 本次更 ...

  6. org.apache.flume.FlumeException: NettyAvroRpcClient { host: xxx.xxx.xxx.xxx, port: 41100 }: RPC

    2014-12-19 01:05:42,141 (lifecycleSupervisor-1-1) [WARN - org.apache.flume.sink.AbstractRpcSink.star ...

  7. Apache Flume日志收集系统简介

    Apache Flume是一个分布式.可靠.可用的系统,用于从大量不同的源有效地收集.聚合.移动大量日志数据进行集中式数据存储. Flume简介 Flume的核心是Agent,Agent中包含Sour ...

  8. Apache Flume 1.7.0 各个模块简介

    Flume简介 Apache Flume是一个分布式.可靠.高可用的日志收集系统,支持各种各样的数据来源,如http,log文件,jms,监听端口数据等等,能将这些数据源的海量日志数据进行高效收集.聚 ...

  9. Flafka: Apache Flume Meets Apache Kafka for Event Processing

    The new integration between Flume and Kafka offers sub-second-latency event processing without the n ...

随机推荐

  1. socket listen/accept

    listen函数 摘要:listen函数使用主动连接套接口变为被连接套接口,使得一个进程可以接受其它进程的请求,从而成为一个服务器进程.在TCP服务器编程中listen函数把进程变为一个服务器,并指定 ...

  2. Android UI开发第二十九篇——Android中五种常用的menu（菜单）

    Android Menu在手机的应用中起着导航的作用,作者总结了5种常用的Menu. 1.左右推出的Menu 前段时间比较流行,我最早是在海豚浏览器中看到的,当时耳目一新.最早使用左右推出菜单的,听说 ...

  3. after

    .pay-type_select-after, .amount-item_select-after { padding: 0; border: @wx-width-one_unit solid @co ...

  4. Delphi中的dll操作

    利用delphi dll wizard进行dll的编写. 创建:保存时改dll名称 library test2; uses SysUtils, Classes, forms, dialogs; {$R ...

  5. pc端监听屏幕实现导航固定定位

    要点:占位符 js,监听屏幕滚动事件,当滚动条距离浏览器顶部的距离 大于 要固定定位开始以下元素的距离,则给要固定元素添加fixed样式. 初始化方法时,要给占位符添加样式 function sort ...

  6. Unity字体文件放Resources和打成ab对比

    情况一:公共字体打成ab的时候,加载A界面的时候加载了font的ab,卸载A和font的ab后,接着加载B界面,加载了font的ab,卸载B和font的ab,这时候字体对应的asset会在内存里有两份 ...

  7. MySQL中锁的类型

    InnoDB存储引擎实现了一下两种标准的行级锁: 共享锁S LOCK 允许事务读一行数据 排他锁 X LOCK 允许事务删除或更新一行数据 如果是一个事务T1斤获得了行r的共享锁,那么另外一个事务T2 ...

  8. [Erlang危机](5.1.3)进程

    原创文章,转载请注明出处:server非业余研究http://blog.csdn.net/erlib 作者Sunface 联系邮箱:cto@188.com Processes Trying to ge ...

  9. DOM 查找标签

    1.直接查找 document.getElementById // 根据ID获取一个标签 document.getElementsByClassName //根据class属性获取 document. ...

  10. （4.14）存储：RAID在数据库存储上的应用

    关键词:(4.14)存储:RAID在数据库存储上的应用 转自:http://blog.51cto.com/qianzhang/1251260 随着单块磁盘在数据安全.性能.容量上呈现出的局限,磁盘阵列 ...