org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count
flume在抽取MySQL数据到kafka时报错,如下
[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.kafka.KafkaSink - Failed to publish events
org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
... more
MemoryChannel的事务的Put队列满
解决:将channel的capacity的值调大即可
a1.channels.ch-.capacity =
注意:Flume配置每行后边不能加注释,否则会导致配置不生效
org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count的更多相关文章
- Put queue for MemoryTransaction of capacity 10000 full, consider committing more frequently, increasing capacity or increasing thread count flume capacity 时间数
package com.test; import org.apache.http.*;import org.apache.http.entity.ContentType;import org.apac ...
- 一个Flume 异常(Put queue for MemoryTransaction of capacity 100 full)的排查和解决思路
最近在做一个分布式调用链跟踪系统, 在两个地方采用了flume (我使用的flume版本是1.5.0-cdh5.4.4),一个是宿主系统 ,用flume agent进行日志搜集. 一个是从kafka拉 ...
- Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法(图文详解)
前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情 启动agent服务 [hadoop@master flume-1.7.0]$ ...
- Apache Flume 学习笔记
# 从http://flume.apache.org/download.html 下载flume ############################################# # 概述: ...
- Apache Flume 1.7.0 发布,日志服务器
Apache Flume 1.7.0 发布了,Flume 是一个分布式.可靠和高可用的服务,用于收集.聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型.这是一个可靠.容错的服务. 本次更 ...
- org.apache.flume.FlumeException: NettyAvroRpcClient { host: xxx.xxx.xxx.xxx, port: 41100 }: RPC
2014-12-19 01:05:42,141 (lifecycleSupervisor-1-1) [WARN - org.apache.flume.sink.AbstractRpcSink.star ...
- Apache Flume日志收集系统简介
Apache Flume是一个分布式.可靠.可用的系统,用于从大量不同的源有效地收集.聚合.移动大量日志数据进行集中式数据存储. Flume简介 Flume的核心是Agent,Agent中包含Sour ...
- Apache Flume 1.7.0 各个模块简介
Flume简介 Apache Flume是一个分布式.可靠.高可用的日志收集系统,支持各种各样的数据来源,如http,log文件,jms,监听端口数据等等,能将这些数据源的海量日志数据进行高效收集.聚 ...
- Flafka: Apache Flume Meets Apache Kafka for Event Processing
The new integration between Flume and Kafka offers sub-second-latency event processing without the n ...
随机推荐
- <转> 堆和栈的区别
一.预备知识—程序的内存分配 一个由C/C++编译的程序占用的内存分为以下几个部分 1.栈区(stack)— 由编译器自动分配释放,存放函数的参数值,局部变量的值等.其操作方式类似于数 ...
- 【BZOJ1925】[Sdoi2010]地精部落 组合数+DP
[BZOJ1925][Sdoi2010]地精部落 Description 传说很久以前,大地上居住着一种神秘的生物:地精. 地精喜欢住在连绵不绝的山脉中.具体地说,一座长度为 N 的山脉 H可分 为从 ...
- mysql编译参数详解(./configure)
1.--prefix=PREFIX:指定程序安装路径: 2.--enable-assembler:使用汇编模式:(文档说明:compiling in x86 (and sparc) versions ...
- <2013 08 17> BucketList of girlfriend
BucketList of girlfriend 1.出国旅游 2.跟相爱的人结婚,生个健康可爱的孩子 3.说一口流利的英语 4.学素描和水彩 5.买个雅马哈钢琴,偶尔学着弹一首曲子 6.把泪腺堵住 ...
- css选择器的权重
权重会叠加!
- Vue.js中css的作用域
Vue.js中的css的作用域问题: 如果在vue组件下的style中定义样式,效果会作用于整个html页面,如果只想本组件的css样式只作用于本组件的话,在<style>标签里添加sco ...
- Python3.6全栈开发实例[021]
21.给出一个纯数字列表. 请对列表进行排序(升级题). 思路: (1)完成a和b的数据交换. 例如, a = 10, b = 24 交换之后, a = 24, b = 10(2)循环列表. 判断a[ ...
- 一、2440裸机点亮led
从代码開始(先写一个像普通单片机一样的代码): /********led.c************************/ #define GPFCON (*(volatile unsigned ...
- 纯手写wcf代码,wcf入门,wcf基础教程
1.定义服务协定 =>定义接口 using System.ServiceModel; namespace WcfConsole { /// <summary> /// 定义服 ...
- 0403-服务注册与发现-客户端负载均衡-Ribbon的基本使用
一.概述 问题1.上一篇文章已说明如何注册微服务,但是调用方如何调用,以及如何防止硬编码.即电影微服务调用用户微服务 问题2.用户微服务多个节点,调用服务方如何负载均衡 二.实现负载均衡方式 2.1. ...