flume ng 1.3 安装(转)
http://blog.csdn.net/hijk139/article/details/8308224
业务系统需要收集监控系统日志,想到了hadoop的flume。经过试验,虽说功能不算足够强大,但基本上能够满足功能需求。Flume 是一个分布式、可靠和高可用的服务日志收集工具,能够和hadoop,hive等配置完成日志收集,存储,分析处理等工作,更详细的介绍可以参见apache网站。下面介绍下简单的安装配置方法
1,网上下载flume-ng安装包,分别部署在收集和接收日志文件的服务器上,服务器上需安装jdk 1.6以上,
http://flume.apache.org/download.html
tar -zxvf apache-flume-1.3.0-bin.tar.gz
2, 日志文件接收端端新建conf/flume-conf.properties server端的具体配置如下
从avro source端接收数据,然后写入到HDFS文件系统中
- [flume@ conf]$ cat flume-conf.properties
- agent.sources = avrosrc
- agent.channels = memoryChanne3
- agent.sinks = hdfsSink
- # For each one of the sources, the type is defined
- agent.sources.avrosrc.type = avro
- agent.sources.avrosrc.bind = 172.16.251.1
- agent.sources.avrosrc.port = 44444
- # The channel can be defined as follows.
- agent.sources.avrosrc.channels = memoryChanne3
- # Each channel's type is defined.
- agent.channels.memoryChanne3.type = memory
- agent.channels.memoryChanne3.keep-alive = 10
- agent.channels.memoryChanne3.capacity = 100000
- agent.channels.memoryChanne3.transactionCapacity =100000
- # Each sink's type must be defined
- agent.sinks.hdfsSink.type = hdfs
- agent.sinks.hdfsSink.channel = memoryChanne3
- agent.sinks.hdfsSink.hdfs.path = /logdata/%{hostname}_linux/%Y%m%d_date
- agent.sinks.hdfsSink.hdfs.filePrefix = %{datacenter}_
- agent.sinks.hdfsSink.hdfs.rollInterval = 0
- agent.sinks.hdfsSink.hdfs.rollSize = 4000000
- agent.sinks.hdfsSink.hdfs.rollCount = 0
- agent.sinks.hdfsSink.hdfs.writeFormat = Text
- agent.sinks.hdfsSink.hdfs.fileType = DataStream
- agent.sinks.hdfsSink.hdfs.batchSize = 10
如果flume和hadoop不是同一用户,需要注意相关权限问题
3,日志收集端的conf/flume-conf.properties server文件配置,这里收集二个日志文件到收集端
- agent.sources = tailsource-1 tailsource-2
- agent.channels = memoryChannel-1 memoryChannel-2
- agent.sinks = remotesink remotesink-2
- agent.sources.tailsource-1.type = exec
- agent.sources.tailsource-1.command = tail -F /tmp/linux2.log
- agent.sources.tailsource-1.channels = memoryChannel-1
- agent.sources.tailsource-2.type = exec
- agent.sources.tailsource-2.command = tail -F /tmp/linux2_2.log
- agent.sources.tailsource-2.channels = memoryChannel-2
- agent.sources.tailsource-1.interceptors = host_int timestamp_int inter1
- agent.sources.tailsource-1.interceptors.host_int.type = host
- agent.sources.tailsource-1.interceptors.host_int.hostHeader = hostname
- agent.sources.tailsource-1.interceptors.timestamp_int.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
- #agent.sources.tailsource-1.interceptors = inter1
- agent.sources.tailsource-1.interceptors.inter1.type = static
- agent.sources.tailsource-1.interceptors.inter1.key = datacenter
- agent.sources.tailsource-1.interceptors.inter1.value = BEIJING
- agent.sources.tailsource-2.interceptors = host_int timestamp_int inter1
- agent.sources.tailsource-2.interceptors.host_int.type = host
- agent.sources.tailsource-2.interceptors.host_int.hostHeader = hostname
- agent.sources.tailsource-2.interceptors.timestamp_int.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
- #agent.sources.tailsource-1.interceptors = inter1
- agent.sources.tailsource-2.interceptors.inter1.type = static
- agent.sources.tailsource-2.interceptors.inter1.key = datacenter
- agent.sources.tailsource-2.interceptors.inter1.value = linux2_2
- agent.channels.memoryChannel-1.type = memory
- agent.channels.memoryChannel-1.keep-alive = 10
- agent.channels.memoryChannel-1.capacity = 100000
- agent.channels.memoryChannel-1.transactionCapacity =100000
- agent.channels.memoryChannel-2.type = memory
- agent.channels.memoryChannel-2.keep-alive = 10
- agent.channels.memoryChannel-2.capacity = 100000
- agent.channels.memoryChannel-2.transactionCapacity =100000
- agent.sinks.remotesink.type = avro
- agent.sinks.remotesink.hostname = 172.16.251.1
- agent.sinks.remotesink.port = 44444
- agent.sinks.remotesink.channel = memoryChannel-1
- agent.sinks.remotesink-2.type = avro
- agent.sinks.remotesink-2.hostname = 172.16.251.1
- agent.sinks.remotesink-2.port = 44444
- agent.sinks.remotesink-2.channel = memoryChannel-2
4,后台运行
nohup bin/flume-ng agent -n agent -c conf -f conf/flume-conf.properties >1.log &
查看日志vi flume.log
端口连接情况 netstat -an|grep 44444
[flume@dtydb6 flume-1.4]$ netstat -an|grep 44444
tcp 0 0 ::ffff:172.16.251.1:44444 :::* LISTEN
5,测试方法
可以使用如下类似的脚本,定期向日志文件写入来进行测试
for i in {1..1000000}; do echo "LINUX2 PRESS ************* Flume log rotation $i" >> /tmp/linux3.log; sleep 0.0001; done
参考资料:
http://flume.apache.org/FlumeUserGuide.html
flume ng 1.3 安装(转)的更多相关文章
- flume ng系列之——flume安装
flume版本:1.5.0 1.下载安装包: http://www.apache.org/dyn/closer.cgi/flume/1.5.0/apache-flume-1.5.0-bin.tar.g ...
- Flume 学习笔记之 Flume NG概述及单节点安装
Flume NG概述: Flume NG是一个分布式,高可用,可靠的系统,它能将不同的海量数据收集,移动并存储到一个数据存储系统中.轻量,配置简单,适用于各种日志收集,并支持 Failover和负载均 ...
- Flume NG Getting Started(Flume NG 新手入门指南)
Flume NG Getting Started(Flume NG 新手入门指南)翻译 新手入门 Flume NG是什么? 有什么改变? 获得Flume NG 从源码构建 配置 flume-ng全局选 ...
- 高可用Hadoop平台-Flume NG实战图解篇
1.概述 今天补充一篇关于Flume的博客,前面在讲解高可用的Hadoop平台的时候遗漏了这篇,本篇博客为大家讲述以下内容: Flume NG简述 单点Flume NG搭建.运行 高可用Flume N ...
- 【转】Flume(NG)架构设计要点及配置实践
Flume(NG)架构设计要点及配置实践 Flume NG是一个分布式.可靠.可用的系统,它能够将不同数据源的海量日志数据进行高效收集.聚合.移动,最后存储到一个中心化数据存储系统中.由原来的Fl ...
- Flume NG简介及配置
Flume下载地址:http://apache.fayea.com/flume/ 常用的分布式日志收集系统: Apache Flume. Facebook Scribe. Apache Chukwa ...
- Flume NG 简介及配置实战
Flume 作为 cloudera 开发的实时日志收集系统,受到了业界的认可与广泛应用.Flume 初始的发行版本目前被统称为 Flume OG(original generation),属于 clo ...
- 【Flume NG用户指南】(1)设置
作者:周邦涛(Timen) Email:zhoubangtao@gmail.com 转载请注明出处: http://blog.csdn.net/zhoubangtao/article/details ...
- 分布式实时日志系统(二) 环境搭建之 flume 集群搭建/flume ng资料
最近公司业务数据量越来越大,以前的基于消息队列的日志系统越来越难以满足目前的业务量,表现为消息积压,日志延迟,日志存储日期过短,所以,我们开始着手要重新设计这块,业界已经有了比较成熟的流程,即基于流式 ...
随机推荐
- TOJ 2749 Absent Substrings
描述 Given a string of symbols, it’s natural to look it over and see what substrings are present. In t ...
- NDK编译不同架构的参数
随着Android的蓬勃发展, CPU的架构也越来越多. 早期只支持ARMv5, 截至目前, 支持的架构已达三类七种: ARM(ARMv5,ARMv7 (从2010年起),ARMv8), x86(x8 ...
- bzoj 4543: [POI2014]Hotel加强版
Description 给出一棵树求三元组 \((x,y,z)\,,x<y<z\) 满足三个点两两之间距离相等,求三元组的数量 Solution 考虑暴力 \(DP\) 设 \(f[i][ ...
- PCU
PCU(Peak concurrent users ),互联网术语,应用在网络游戏和其他互联网服务领域,意思是最高同时在线人数 业务系统架构性能提升主要分为两种不同的方式,scale-out(横向扩展 ...
- VB.Net遍历已安装的程序卸载信息
Private Shared Function ProgramExists(ByVal pgName As String) As Boolean Dim temp As String = Nothin ...
- CSP学习之ASN.1编码(一)
一.定义: 是定义抽象数据类型的标准. 是用于描述数据的表示.编码.传输.解码的灵活记法. 它提供一套正式.无歧义和精确的规则,以描述独立于特定计算机硬件的对象结构. 标准的ASN.1编码规则有其基本 ...
- redis的有序集合(Sorted Sets)数据类型
和Sets相比,Sorted Sets增加了一个权重参数score,使得集合中的元素能够按score进行有序排列,比如一个存储全班同学成绩的Sorted Sets,其集合value可以是同学的学号,而 ...
- HDU 5012 骰子旋转(DFS)
http://acm.hdu.edu.cn/showproblem.php?pid=5012 保存骰子的状态,然后用dfs或者bfs搜索 还是再讲一下dfs 我们的目标是找一个与b相同,且转次数最少的 ...
- 静态化HttpClient
实现方法: public class HttpClientHelper { private static HttpClient _client; public static HttpClient Ge ...
- java中的集合和视图
一.集合的概念 何为集合,集合就是相当于一个对象的容器.集合是类似数组的一个作用.既然有了数组,为何还要有集合呢,由于数组对象一旦创建,其大小便不可以更改,我们只能往数组中存放创建时数量的对象.而集合 ...