flume 整合 kafka:
 
flume:高可用的,高可靠的,分布式的海量日志采集、聚合和传输的系统。
kafka:分布式的流数据平台。
 
flume 采集业务日志,发送到kafka
 
一、安装部署Kafka

Download

1.0.0 is the latest release. The current stable version is 1.0.0.
You can verify your download by following these procedures and using these KEYS.

1.0.0

We build for multiple versions of Scala. This only matters if you are using Scala and you want a version built for the same Scala version you use. Otherwise any version should work (2.11 is recommended).
 
1.解压:tar zxvf kafka_2.11-1.0.0.tgz
 
2.部署目录:mv kafka_2.12-1.0.0 /usr/local/kafka2.12
 
3.启动zookeeper ....
 
4.启动kafka:
#nohup bin/kafka-server-start.sh config/server.properties &
 

5.创建topic:

#bin/kafka-topics.sh --create --zookeeper localhost:2181 --partitions 1 --replication-factor 1 --topic test
Created topic "test".

6.查看topic:

# bin/kafka-topics.sh --list --zookeeper localhost:2181
__consumer_offsets
test

7.测试发送数据

#bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
输入:my test
 
8.测试消费消息:
#bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
 
 
二、安装部署flume
 
flume下载:
 

Download

Apache Flume is distributed under the Apache License, version 2.0
The link in the Mirrors column should display a list of available mirrors with a default selection based on your inferred location. If you do not see that page, try a different browser. The checksum and signature are links to the originals on the main distribution server.
Apache Flume binary (tar.gz) apache-flume-1.8.0-bin.tar.gz apache-flume-1.8.0-bin.tar.gz.md5 apache-flume-1.8.0-bin.tar.gz.sha1 apache-flume-1.8.0-bin.tar.gz.asc
Apache Flume source (tar.gz) apache-flume-1.8.0-src.tar.gz apache-flume-1.8.0-src.tar.gz.md5 apache-flume-1.8.0-src.tar.gz.sha1 apache-flume-1.8.0-src.tar.gz.asc
It is essential that you verify the integrity of the downloaded files using the PGP or MD5 signatures. Please read Verifying Apache HTTP Server Releases for more information on why you should verify our releases.
 
 
2.解压:tar zxvf apache-flume-1.8.0-bin.tar.gz
 
3.设置目录:mv apache-flume-1.8.0-bin /usr/local/flume1.8
 

4.准备工作:

安装java并设置java环境变量,flume环境变量,在`/etc/profile`中加入
 
export JAVA_HOME=/usr/java/jdk1.8.0_65
export FLUME_HOME=/usr/local/flume1.8
export PATH=$PATH:$JAVA_HOME/bin:$FLUME_HOME
 

执行:source /etc/profile 生效变量

5.建立log采集目录:
/tmp/logs/kafka.log
6.配置
拷贝配置模板:
# cp conf/flume-conf.properties.template conf/flume-conf.properties
# cp conf/flume-env.properties.template conf/flume-env.properties
编辑配置如下:
agent.sources = s1                                                                                                                  
agent.channels = c1                                                                                                                 
agent.sinks = k1                                                                                                                    
                                                                                                                                      
agent.sources.s1.type=exec                                     
#日志采集位置                                                                     
agent.sources.s1.command=tail -F /tmp/logs/kafka.log                                                                                
agent.sources.s1.channels=c1                                                                                                        
agent.channels.c1.type=memory                                                                                                       
agent.channels.c1.capacity=10000                                                                                                    
agent.channels.c1.transactionCapacity=100                                                                                           
                                                                                                                                      
agent.sinks.k1.type= org.apache.flume.sink.kafka.KafkaSink    
#kafka 地址                                                                      
agent.sinks.k1.brokerList=localhost:9092   
#kafka topic                                                                                         
agent.sinks.k1.topic=test                                                                                                      
agent.sinks.k1.serializer.class=kafka.serializer.StringEncoder                                                                      
                                                                                                                                      
agent.sinks.k1.channel=c1

功能验证

7.启动服务

# bin/flume-ng agent --conf ./conf/ -f conf/kafka.properties -Dflume.root.logger=DEBUG,console -n agent
运行日志位于logs目录,或者启动时添加-Dflume.root.logger=INFO,console 选项前台启动,输出打印日志,查看具体运行日志,服务异常时查原因。

8.创建测试日志生成:log_producer_test.sh

for((i=0;i<=1000;i++));
do echo "kafka_flume_test-"+$i>>/tmp/logs/kafka.log;
do
 
9.生成日志:
./log_producer_test.sh
 
观察kafka日志消费情况。。。

flume 整合 kafka的更多相关文章

  1. 【Kafka】Flume整合Kafka

    目录 需求 一.Flume下载地址 二.上传解压Flume 三.配置flume.conf 四.启动flume 五.测试整合 需求 实现flume监控某个目录下面的所有文件,然后将文件收集发送到kafk ...

  2. 入门大数据---Flume整合Kafka

    一.背景 先说一下,为什么要使用 Flume + Kafka? 以实时流处理项目为例,由于采集的数据量可能存在峰值和峰谷,假设是一个电商项目,那么峰值通常出现在秒杀时,这时如果直接将 Flume 聚合 ...

  3. flume整合kafka

    # Please paste flume.conf here. Example: # Sources, channels, and sinks are defined per # agent name ...

  4. flume 整合kafka

    背景:系统的数据量越来越大,日志不能再简单的文件的保存,如此日志将会越来越大,也不方便查找与分析,综合考虑下使用了flume来收集日志,收集日志后向kafka传递消息,下面给出具体的配置 # The ...

  5. flume与kafka整合

    flume与kafka整合 前提: flume安装和测试通过,可参考:http://www.cnblogs.com/rwxwsblog/p/5800300.html kafka安装和测试通过,可参考: ...

  6. ambari下的flume和kafka整合

    1.配置flume #扫描指定文件配置 agent.sources = s1 agent.channels = c1 agent.sinks = k1 agent.sources.s1.type=ex ...

  7. Flume和Kafka整合安装

    版本号: RedHat6.5   JDK1.8    flume-1.6.0   kafka_2.11-0.8.2.1 1.flume安装 RedHat6.5安装单机flume1.6:http://b ...

  8. 大数据入门第二十四天——SparkStreaming(二)与flume、kafka整合

    前一篇中数据源采用的是从一个socket中拿数据,有点属于“旁门左道”,正经的是从kafka等消息队列中拿数据! 主要支持的source,由官网得知如下: 获取数据的形式包括推送push和拉取pull ...

  9. flume和kafka整合(转)

    原文链接:Kafka flume 整合 前提 前提是要先把flume和kafka独立的部分先搭建好. 下载插件包 下载flume-kafka-plus:https://github.com/beyon ...

随机推荐

  1. inode的理解

    迫于需要理解sock_init()中的init_inodecache,所以稍微学习了一下inode. 一.inode的定义 文件储存在硬盘上,硬盘的最小存储单位叫做"扇区"(Sec ...

  2. STM32F10X固件库函数——串口清状态位函数分析

    STM32F10X固件库函数——串口清状态位函数分析 最近在测试串口热插拔功能的时候,意外发现STM32F10X的串口库函数中,清理串口状态位函数稍稍有点不解.下面是改函数的源码: /******** ...

  3. canvas背景

    <!doctype html> <html lang="en"> <head> <meta charset="UTF-8&quo ...

  4. react系列(一)JSX语法、组件概念、生命周期介绍

    JSX React中,推出了一种新的语法取名为JSX,它给了JS中写HTML标签的能力,不需要加引号.JSX的语法看起来是一种模板,然而它在编译以后,会转成JS语法,只是书写过程中的语法糖. JSX的 ...

  5. 记录一次没有收集直方图优化器选择全表扫描导致CPU耗尽

    场景:数据库升级第二天,操作系统CPU使用率接近100%. 查看ash报告: 再看TOP SQL 具体SQL: select count(1) as chipinCount, sum(bets) as ...

  6. ARC下IBOutlet用weak还是strong

    原文来自这里. 今天用Xcode5的时候,发现默认的IBoutlet的属性设置为weak——因为Xcode5建立的工程都是ARC的了.但是当时还有点不明白,因为项目的原因,一直没有正式使用过ARC.于 ...

  7. 执行SQL查询导致磁盘耗尽故障演示

            a fellow in IMG wechat group 2 met an error about running out of disk space when using MySQL ...

  8. JS实现继承 JavaScript

    JS实现继承 JavaScript 定义一个父类: // 定义一个动物类 function Animal (name) { // 属性 this.name = name || 'Animal'; // ...

  9. Redis学习笔记(一)

    定义 Redis是一个开源的使用ANSI C语言编写.支持网络.可基于内存亦可持久化的日志型.Key-Value数据库. 从该定义中抽出几个关键信息,以表示Redis的特性: 存储结构:key-val ...

  10. ElasticSearch 集群安装,简单使用

    http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html https://gith ...