flume在抽取MySQL数据到kafka时报错,如下

[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.kafka.KafkaSink - Failed to publish events
org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
... more
MemoryChannel的事务的Put队列满
解决:将channel的capacity的值调大即可

a1.channels.ch-.capacity = 
注意:Flume配置每行后边不能加注释,否则会导致配置不生效

org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count的更多相关文章

  1. Put queue for MemoryTransaction of capacity 10000 full, consider committing more frequently, increasing capacity or increasing thread count flume capacity 时间数

    package com.test; import org.apache.http.*;import org.apache.http.entity.ContentType;import org.apac ...

  2. 一个Flume 异常(Put queue for MemoryTransaction of capacity 100 full)的排查和解决思路

    最近在做一个分布式调用链跟踪系统, 在两个地方采用了flume (我使用的flume版本是1.5.0-cdh5.4.4),一个是宿主系统 ,用flume agent进行日志搜集. 一个是从kafka拉 ...

  3. Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法(图文详解)

        前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情 启动agent服务 [hadoop@master flume-1.7.0]$ ...

  4. Apache Flume 学习笔记

    # 从http://flume.apache.org/download.html 下载flume ############################################# # 概述: ...

  5. Apache Flume 1.7.0 发布,日志服务器

    Apache Flume 1.7.0 发布了,Flume 是一个分布式.可靠和高可用的服务,用于收集.聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型.这是一个可靠.容错的服务. 本次更 ...

  6. org.apache.flume.FlumeException: NettyAvroRpcClient { host: xxx.xxx.xxx.xxx, port: 41100 }: RPC

    2014-12-19 01:05:42,141 (lifecycleSupervisor-1-1) [WARN - org.apache.flume.sink.AbstractRpcSink.star ...

  7. Apache Flume日志收集系统简介

    Apache Flume是一个分布式.可靠.可用的系统,用于从大量不同的源有效地收集.聚合.移动大量日志数据进行集中式数据存储. Flume简介 Flume的核心是Agent,Agent中包含Sour ...

  8. Apache Flume 1.7.0 各个模块简介

    Flume简介 Apache Flume是一个分布式.可靠.高可用的日志收集系统,支持各种各样的数据来源,如http,log文件,jms,监听端口数据等等,能将这些数据源的海量日志数据进行高效收集.聚 ...

  9. Flafka: Apache Flume Meets Apache Kafka for Event Processing

    The new integration between Flume and Kafka offers sub-second-latency event processing without the n ...

随机推荐

  1. <转> 堆和栈的区别

    一.预备知识—程序的内存分配    一个由C/C++编译的程序占用的内存分为以下几个部分    1.栈区(stack)—   由编译器自动分配释放,存放函数的参数值,局部变量的值等.其操作方式类似于数 ...

  2. 【BZOJ1925】[Sdoi2010]地精部落 组合数+DP

    [BZOJ1925][Sdoi2010]地精部落 Description 传说很久以前,大地上居住着一种神秘的生物:地精. 地精喜欢住在连绵不绝的山脉中.具体地说,一座长度为 N 的山脉 H可分 为从 ...

  3. mysql编译参数详解(./configure)

    1.--prefix=PREFIX:指定程序安装路径: 2.--enable-assembler:使用汇编模式:(文档说明:compiling in x86 (and sparc) versions  ...

  4. <2013 08 17> BucketList of girlfriend

    BucketList of girlfriend 1.出国旅游 2.跟相爱的人结婚,生个健康可爱的孩子 3.说一口流利的英语 4.学素描和水彩 5.买个雅马哈钢琴,偶尔学着弹一首曲子 6.把泪腺堵住 ...

  5. css选择器的权重

    权重会叠加!

  6. Vue.js中css的作用域

    Vue.js中的css的作用域问题: 如果在vue组件下的style中定义样式,效果会作用于整个html页面,如果只想本组件的css样式只作用于本组件的话,在<style>标签里添加sco ...

  7. Python3.6全栈开发实例[021]

    21.给出一个纯数字列表. 请对列表进行排序(升级题). 思路: (1)完成a和b的数据交换. 例如, a = 10, b = 24 交换之后, a = 24, b = 10(2)循环列表. 判断a[ ...

  8. 一、2440裸机点亮led

    从代码開始(先写一个像普通单片机一样的代码): /********led.c************************/ #define GPFCON  (*(volatile unsigned ...

  9. 纯手写wcf代码,wcf入门,wcf基础教程

    1.定义服务协定     =>定义接口 using System.ServiceModel; namespace WcfConsole { /// <summary> /// 定义服 ...

  10. 0403-服务注册与发现-客户端负载均衡-Ribbon的基本使用

    一.概述 问题1.上一篇文章已说明如何注册微服务,但是调用方如何调用,以及如何防止硬编码.即电影微服务调用用户微服务 问题2.用户微服务多个节点,调用服务方如何负载均衡 二.实现负载均衡方式 2.1. ...