Hadoop-(Flume)

1. Flume 介绍

1.1. 概述

  • Flume是一个分布式、可靠、和高可用的海量日志采集、聚合和传输的系统。
  • Flume可以采集文件，socket数据包、文件、文件夹、kafka等各种形式源数据，又可以将采集到的数据(下沉sink)输出到HDFS、hbase、hive、kafka等众多外部存储系统中
  • 一般的采集需求，通过对flume的简单配置即可实现
  • Flume针对特殊场景也具备良好的自定义扩展能力， 
    因此，flume可以适用于大部分的日常数据采集场景

1.2. 运行机制

  1. Flume分布式系统中最核心的角色是agent，flume采集系统就是由一个个agent所连接起来形成

  2. 每一个agent相当于一个数据传递员，内部有三个组件：

    1. Source：采集组件，用于跟数据源对接，以获取数据

    2. Sink：下沉组件，用于往下一级agent传递数据或者往最终存储系统传递数据

    3. Channel：传输通道组件，用于从source将数据传递到sink

1.3. Flume 结构图

简单结构

单个 Agent 采集数据

复杂结构

多级 Agent 之间串联

2. Flume 实战案例

案例：使用网络telent命令向一台机器发送一些网络数据，然后通过flume采集网络端口数据

2.1. Flume 的安装部署

Step 1: 下载解压修改配置文件

下载地址：

http://archive.apache.org/dist/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz

Flume的安装非常简单，只需要解压即可，当然，前提是已有hadoop环境

上传安装包到数据源所在节点上

这里我们采用在第三台机器来进行安装

  1. cd /export/softwares/
    2. 

  2. tar -zxvf apache-flume-1.8.-bin.tar.gz -C ../servers/
    3. 

  3. cd /export/servers/apache-flume-1.8.-bin/conf
    4. 

  4. cp  flume-env.sh.template flume-env.sh
    5. 

  5. vim flume-env.sh
    6. 

  6. export JAVA_HOME=/export/servers/jdk1.8.0_141
Step 2: 开发配置文件

根据数据采集的需求配置采集方案，描述在配置文件中(文件名可任意自定义)

配置我们的网络收集的配置文件 
在flume的conf目录下新建一个配置文件（采集方案）

  1. vim   /export/servers/apache-flume-1.8.-bin/conf/netcat-logger.conf


  1. # 定义这个agent中各组件的名字
    2. 

  2. a1.sources = r1
    3. 

  3. a1.sinks = k1
    4. 

  4. a1.channels = c1
    5. 

    6. 

  6. # 描述和配置source组件：r1
    7. 

  7. a1.sources.r1.type = netcat
    8. 

  8. a1.sources.r1.bind = 192.168.174.
    9. 

  9. a1.sources.r1.port = 
    10. 

    11. 

  11. # 描述和配置sink组件：k1
    12. 

  12. a1.sinks.k1.type = logger
    13. 

    14. 

  14. # 描述和配置channel组件，此处使用是内存缓存的方式
    15. 

  15. a1.channels.c1.type = memory
    16. 

  16. a1.channels.c1.capacity = 
    17. 

  17. a1.channels.c1.transactionCapacity = 
    18. 

    19. 

  19. # 描述和配置source  channel   sink之间的连接关系
    20. 

  20. a1.sources.r1.channels = c1
    21. 

  21. a1.sinks.k1.channel = c1

Step 3: 启动配置文件

指定采集方案配置文件，在相应的节点上启动flume agent

先用一个最简单的例子来测试一下程序环境是否正常 
启动agent去采集数据

  1. bin/flume-ng agent -c conf -f conf/netcat-logger.conf -n a1  -Dflume.root.logger=INFO,console
  • -c conf 指定flume自身的配置文件所在目录
  • -f conf/netcat-logger.con 指定我们所描述的采集方案
  • -n a1 指定我们这个agent的名字
Step 4: 安装 Telnet 准备测试

在node02机器上面安装telnet客户端，用于模拟数据的发送

  1. yum -y install telnet
    2. 

  2. telnet  node03   # 使用telnet模拟数据发送

2.2. 采集案例

2.2.3. 采集目录到 HDFS

需求

某服务器的某特定目录下，会不断产生新的文件，每当有新文件出现，就需要把文件采集到HDFS中去

思路

根据需求，首先定义以下3大要素

  1. 数据源组件，即source ——监控文件目录 : spooldir
    1. 监视一个目录，只要目录中出现新文件，就会采集文件中的内容
    2. 采集完成的文件，会被agent自动添加一个后缀：COMPLETED
    3. 所监视的目录中不允许重复出现相同文件名的文件,否则报错\罢工
  2. 下沉组件，即sink——HDFS文件系统 : hdfs sink
  3. 通道组件，即channel——可用file channel 也可以用内存channel
Step 1: Flume 配置文件
  1. cd  /export/servers/apache-flume-1.8.-bin/conf
    2. 

  2. mkdir -p /export/servers/dirfile
    3. 

  3. vim spooldir.conf


  1. # Name the components on this agent
    2. 

  2. a1.sources = r1
    3. 

  3. a1.sinks = k1
    4. 

  4. a1.channels = c1
    5. 

    6. 

    7. 

  7. # Describe/configure the source
    8. 

  8. ##注意：不能往监控目中重复丢同名文件
    9. 

  9. a1.sources.r1.type = spooldir
    10. 

  10. a1.sources.r1.spoolDir = /export/servers/dirfile
    11. 

  11. a1.sources.r1.fileHeader = true
    12. 

    13. 

    14. 

  14. # Describe the sink
    15. 

  15. a1.sinks.k1.type = hdfs
    16. 

  16. a1.sinks.k1.channel = c1
    17. 

  17. a1.sinks.k1.hdfs.path = hdfs://node01:8020/spooldir/files/%y-%m-%d/%H%M/
    18. 

  18. a1.sinks.k1.hdfs.filePrefix = events-
    19. 

    20. 

  20. #控制文件夹以多少时间滚动     10分钟
    21. 

  21. a1.sinks.k1.hdfs.round = true
    22. 

  22. a1.sinks.k1.hdfs.roundValue = 
    23. 

  23. a1.sinks.k1.hdfs.roundUnit = minute
    24. 

  24. #roll控制写入hdfs文件,以何种方式滚动
    25. 

  25. #时间间隔
    26. 

  26. a1.sinks.k1.hdfs.rollInterval = 
    27. 

  27. #文件大小
    28. 

  28. a1.sinks.k1.hdfs.rollSize = 
    29. 

  29. #even数量
    30. 

  30. a1.sinks.k1.hdfs.rollCount = 
    31. 

  31. a1.sinks.k1.hdfs.batchSize = 
    32. 

  32. #不想滚动,设置0
    33. 

    34. 

  34. a1.sinks.k1.hdfs.useLocalTimeStamp = true
    35. 

    36. 

  36. #生成的文件类型，默认是Sequencefile，可用DataStream，则为普通文本
    37. 

    38. 

  38. a1.sinks.k1.hdfs.fileType = DataStream
    39. 

    40. 

  40. # Use a channel which buffers events in memory
    41. 

  41. a1.channels.c1.type = memory
    42. 

  42. #胶囊容量
    43. 

  43. a1.channels.c1.capacity = 
    44. 

  44. #一次向sink运输多少个event
    45. 

  45. a1.channels.c1.transactionCapacity = 
    46. 

    47. 

  47. # Bind the source and sink to the channel
    48. 

  48. a1.sources.r1.channels = c1
    49. 

  49. a1.sinks.k1.channel = c1

Channel参数解释

capacity：默认该通道中最大的可以存储的event数量 
trasactionCapacity：每次最大可以从source中拿到或者送到sink中的event数量 
keep-alive：event添加到通道中或者移出的允许时间

Step 2: 启动 Flume
  1. bin/flume-ng agent -c ./conf -f ./conf/spooldir.conf -n a1 -Dflume.root.logger=INFO,console
    2. 

  2. #命令的精简版
Step 3: 上传文件到指定目录

将不同的文件放到下面目录里面去，注意文件不能重名

  1. cd /export/servers/dirfile

2.2.4. 采集文件到 HDFS

需求

比如业务系统使用log4j生成的日志，日志内容不断增加，需要把追加到日志文件中的数据实时采集到hdfs

分析

根据需求，首先定义以下3大要素

  • 采集源，即source——监控文件内容更新 : exec ‘tail -F file’
  • 下沉目标，即sink——HDFS文件系统 : hdfs sink
  • Source和sink之间的传递通道——channel，可用file channel 也可以用 内存channel
Step 1: 定义 Flume 配置文件
  1. cd /export/servers/apache-flume-1.8.-bin/conf
    2. 

  2. vim tail-file.conf


  1. # Name the components on this agent
    2. 

  2. a1.sources = r1
    3. 

  3. a1.sinks = k1
    4. 

  4. a1.channels = c1
    5. 

    6. 

  6. # Describe/configure the source
    7. 

  7. a1.sources.r1.type = exec
    8. 

  8. a1.sources.r1.command = tail -F /root/logs/test.log
    9. 

  9. a1.sources.r1.channels = c1
    10. 

    11. 

  11. # Describe the sink
    12. 

  12. a1.sinks.k1.type = hdfs
    13. 

  13. a1.sinks.k1.channel = c1
    14. 

  14. a1.sinks.k1.hdfs.path = /flume/tailout/%y-%m-%d/%H-%M/
    15. 

  15. a1.sinks.k1.hdfs.filePrefix = events-
    16. 

  16. a1.sinks.k1.hdfs.round = true
    17. 

  17. a1.sinks.k1.hdfs.roundValue = 
    18. 

  18. a1.sinks.k1.hdfs.roundUnit = minute
    19. 

  19. a1.sinks.k1.hdfs.rollInterval = 
    20. 

  20. a1.sinks.k1.hdfs.rollSize = 
    21. 

  21. a1.sinks.k1.hdfs.rollCount = 
    22. 

  22. a1.sinks.k1.hdfs.batchSize = 
    23. 

  23. a1.sinks.k1.hdfs.useLocalTimeStamp = true
    24. 

  24. #生成的文件类型，默认是Sequencefile，可用DataStream，则为普通文本
    25. 

  25. a1.sinks.k1.hdfs.fileType = DataStream
    26. 

    27. 

  27. # Use a channel which buffers events in memory
    28. 

  28. a1.channels.c1.type = memory
    29. 

  29. a1.channels.c1.capacity = 
    30. 

  30. a1.channels.c1.transactionCapacity = 
    31. 

    32. 

  32. # Bind the source and sink to the channel
    33. 

  33. a1.sources.r1.channels = c1
    34. 

  34. a1.sinks.k1.channel = c1
Step 2: 启动 Flume
  1. cd  /export/servers/apache-flume-1.8.-bin
    2. 

  2. bin/flume-ng agent -c conf -f conf/tail-file.conf -n agent1  -Dflume.root.logger=INFO,console
Step 3: 开发 Shell 脚本定时追加文件内容
  1. mkdir -p /export/servers/shells/
    2. 

  2. cd  /export/servers/shells/
    3. 

  3. vim tail-file.sh


  1. #!/bin/bash
    2. 

  2. while true
    3. 

  3. do
    4. 

  4. date >> /export/servers/taillogs/access_log;
    5. 

  5.   sleep 0.5;
    6. 

  6. done
Step 4: 启动脚本
  1. # 创建文件夹
    2. 

  2. mkdir -p /export/servers/taillogs
    3. 

  3. # 启动脚本
    4. 

  4. sh /export/servers/shells/tail-file.sh

2.2.5. Agent 级联

分析

第一个agent负责收集文件当中的数据，通过网络发送到第二个agent当中去 
第二个agent负责接收第一个agent发送的数据，并将数据保存到hdfs上面去

Step 1: Node02 安装 Flume

将node03机器上面解压后的flume文件夹拷贝到node02机器上面去

  1. cd  /export/servers
    2. 

  2. scp -r apache-flume-1.8.-bin/ node02:$PWD
Step 2: Node02 配置 Flume

在node02机器配置我们的flume

  1. cd /export/servers/ apache-flume-1.8.-bin/conf
    2. 

  2. vim tail-avro-avro-logger.conf


  1. ##################
    2. 

  2. # Name the components on this agent
    3. 

  3. a1.sources = r1
    4. 

  4. a1.sinks = k1
    5. 

  5. a1.channels = c1
    6. 

  6. # Describe/configure the source
    7. 

  7. a1.sources.r1.type = exec
    8. 

  8. a1.sources.r1.command = tail -F /export/servers/taillogs/access_log
    9. 

  9. a1.sources.r1.channels = c1
    10. 

  10. # Describe the sink
    11. 

  11. ##sink端的avro是一个数据发送者
    12. 

  12. a1.sinks = k1
    13. 

  13. a1.sinks.k1.type = avro
    14. 

  14. a1.sinks.k1.channel = c1
    15. 

  15. a1.sinks.k1.hostname = node03
    16. 

  16. a1.sinks.k1.port = 
    17. 

  17. a1.sinks.k1.batch-size = 
    18. 

  18. # Use a channel which buffers events in memory
    19. 

  19. a1.channels.c1.type = memory
    20. 

  20. a1.channels.c1.capacity = 
    21. 

  21. a1.channels.c1.transactionCapacity = 
    22. 

  22. # Bind the source and sink to the channel
    23. 

  23. a1.sources.r1.channels = c1
    24. 

  24. a1.sinks.k1.channel = c1
Step 3: 开发脚本向文件中写入数据

直接将node03下面的脚本和数据拷贝到node02即可，node03机器上执行以下命令

  1. cd  /export/servers
    2. 

  2. scp -r shells/ taillogs/ node02:$PWD
Step 4: Node03 Flume 配置文件

在node03机器上开发flume的配置文件

  1. cd /export/servers/apache-flume-1.8.-bin/conf
    2. 

  2. vim avro-hdfs.conf


  1. # Name the components on this agent
    2. 

  2. a1.sources = r1
    3. 

  3. a1.sinks = k1
    4. 

  4. a1.channels = c1
    5. 

  5. # Describe/configure the source
    6. 

  6. ##source中的avro组件是一个接收者服务
    7. 

  7. a1.sources.r1.type = avro
    8. 

  8. a1.sources.r1.channels = c1
    9. 

  9. a1.sources.r1.bind = node03
    10. 

  10. a1.sources.r1.port = 
    11. 

  11. # Describe the sink
    12. 

  12. a1.sinks.k1.type = hdfs
    13. 

  13. a1.sinks.k1.hdfs.path = hdfs://node01:8020/av/%y-%m-%d/%H%M/
    14. 

  14. a1.sinks.k1.hdfs.filePrefix = events-
    15. 

  15. a1.sinks.k1.hdfs.round = true
    16. 

  16. a1.sinks.k1.hdfs.roundValue = 
    17. 

  17. a1.sinks.k1.hdfs.roundUnit = minute
    18. 

  18. a1.sinks.k1.hdfs.rollInterval = 
    19. 

  19. a1.sinks.k1.hdfs.rollSize = 
    20. 

  20. a1.sinks.k1.hdfs.rollCount = 
    21. 

  21. a1.sinks.k1.hdfs.batchSize = 
    22. 

  22. a1.sinks.k1.hdfs.useLocalTimeStamp = true
    23. 

  23. #生成的文件类型，默认是Sequencefile，可用DataStream，则为普通文本
    24. 

  24. a1.sinks.k1.hdfs.fileType = DataStream
    25. 

  25. # Use a channel which buffers events in memory
    26. 

  26. a1.channels.c1.type = memory
    27. 

  27. a1.channels.c1.capacity = 
    28. 

  28. a1.channels.c1.transactionCapacity = 
    29. 

    30. 

  30. # Bind the source and sink to the channel
    31. 

  31. a1.sources.r1.channels = c1
    32. 

  32. a1.sinks.k1.channel = c1
Step 5: 顺序启动

node03机器启动flume进程

  1. cd /export/servers/apache-flume-1.8.-bin
    2. 

  2. bin/flume-ng agent -c conf -f conf/avro-hdfs.conf -n a1  -Dflume.root.logger=INFO,console

node02机器启动flume进程

  1. cd /export/servers/apache-flume-1.8.-bin/
    2. 

  2. bin/flume-ng agent -c conf -f conf/tail-avro-avro-logger.conf -n a1  -Dflume.root.logger=INFO,console

node02机器启shell脚本生成文件

  1. cd  /export/servers/shells
    2. 

  2. sh tail-file.sh

3. flume的高可用方案-failover

在完成单点的Flume NG搭建后，下面我们搭建一个高可用的Flume NG集群，架构图如下所示：

3.1. 角色分配

Flume的Agent和Collector分布如下表所示：

名称 HOST 角色
Agent1 node01 Web Server
Collector1 node02 AgentMstr1
Collector2 node03 AgentMstr2

图中所示，Agent1数据分别流入到Collector1和Collector2，Flume NG本身提供了Failover机制，可以自动切换和恢复。在上图中，有3个产生日志服务器分布在不同的机房，要把所有的日志都收集到一个集群中存储。下 面我们开发配置Flume NG集群

3.2. Node01 安装和配置

将node03机器上面的flume安装包以及文件生产的两个目录拷贝到node01机器上面去

node03机器执行以下命令

  1. cd /export/servers
    2. 

  2. scp -r apache-flume-1.8.-bin/ node01:$PWD
    3. 

  3. scp -r shells/ taillogs/ node01:$PWD

node01机器配置agent的配置文件

  1. cd /export/servers/apache-flume-1.8.-bin/conf
    2. 

  2. vim agent.conf


  1. #agent1 name
    2. 

  2. agent1.channels = c1
    3. 

  3. agent1.sources = r1
    4. 

  4. agent1.sinks = k1 k2
    5. 

  5. #
    6. 

  6. ##set gruop
    7. 

  7. agent1.sinkgroups = g1
    8. 

  8. #
    9. 

    10. 

  10. agent1.sources.r1.channels = c1
    11. 

  11. agent1.sources.r1.type = exec
    12. 

  12. agent1.sources.r1.command = tail -F /export/servers/taillogs/access_log
    13. 

  13. #
    14. 

  14. ##set channel
    15. 

  15. agent1.channels.c1.type = memory
    16. 

  16. agent1.channels.c1.capacity = 
    17. 

  17. agent1.channels.c1.transactionCapacity = 
    18. 

  18. #
    19. 

  19. ## set sink1
    20. 

  20. agent1.sinks.k1.channel = c1
    21. 

  21. agent1.sinks.k1.type = avro
    22. 

  22. agent1.sinks.k1.hostname = node02
    23. 

  23. agent1.sinks.k1.port = 
    24. 

  24. #
    25. 

  25. ## set sink2
    26. 

  26. agent1.sinks.k2.channel = c1
    27. 

  27. agent1.sinks.k2.type = avro
    28. 

  28. agent1.sinks.k2.hostname = node03
    29. 

  29. agent1.sinks.k2.port = 
    30. 

  30. #
    31. 

  31. ##set sink group
    32. 

  32. agent1.sinkgroups.g1.sinks = k1 k2
    33. 

  33. #
    34. 

  34. ##set failover
    35. 

  35. agent1.sinkgroups.g1.processor.type = failover
    36. 

  36. agent1.sinkgroups.g1.processor.priority.k1 = 
    37. 

  37. agent1.sinkgroups.g1.processor.priority.k2 = 
    38. 

  38. agent1.sinkgroups.g1.processor.maxpenalty = 


  1. #agent1 name
    2. 

  2. agent1.channels = c1
    3. 

  3. agent1.sources = r1
    4. 

  4. agent1.sinks = k1 k2
    5. 

    6. 

  6. #set gruop
    7. 

  7. agent1.sinkgroups = g1
    8. 

    9. 

  9. #set channel
    10. 

  10. agent1.channels.c1.type = memory
    11. 

  11. agent1.channels.c1.capacity = 
    12. 

  12. agent1.channels.c1.transactionCapacity = 
    13. 

    14. 

  14. agent1.sources.r1.channels = c1
    15. 

  15. agent1.sources.r1.type = exec
    16. 

  16. agent1.sources.r1.command = tail -F /root/logs/456.log
    17. 

    18. 

  18. # set sink1
    19. 

  19. agent1.sinks.k1.channel = c1
    20. 

  20. agent1.sinks.k1.type = avro
    21. 

  21. agent1.sinks.k1.hostname = node02
    22. 

  22. agent1.sinks.k1.port = 
    23. 

    24. 

  24. # set sink2
    25. 

  25. agent1.sinks.k2.channel = c1
    26. 

  26. agent1.sinks.k2.type = avro
    27. 

  27. agent1.sinks.k2.hostname = node03
    28. 

  28. agent1.sinks.k2.port = 
    29. 

    30. 

  30. #set sink group
    31. 

  31. agent1.sinkgroups.g1.sinks = k1 k2
    32. 

    33. 

  33. #set failover
    34. 

  34. agent1.sinkgroups.g1.processor.type = failover
    35. 

  35. agent1.sinkgroups.g1.processor.priority.k1 = 
    36. 

  36. agent1.sinkgroups.g1.processor.priority.k2 = 
    37. 

  37. agent1.sinkgroups.g1.processor.maxpenalty =

3.3. Node02 与 Node03 配置 FlumeCollection

node02机器修改配置文件

  1. cd /export/servers/apache-flume-1.8.-bin/conf
    2. 

  2. vim collector.conf


  1. #set Agent name
    2. 

  2. a1.sources = r1
    3. 

  3. a1.channels = c1
    4. 

  4. a1.sinks = k1
    5. 

  5. #
    6. 

  6. ##set channel
    7. 

  7. a1.channels.c1.type = memory
    8. 

  8. a1.channels.c1.capacity = 
    9. 

  9. a1.channels.c1.transactionCapacity = 
    10. 

  10. #
    11. 

  11. ## other node,nna to nns
    12. 

  12. a1.sources.r1.type = avro
    13. 

  13. a1.sources.r1.bind = node02
    14. 

  14. a1.sources.r1.port = 
    15. 

  15. a1.sources.r1.channels = c1
    16. 

  16. #
    17. 

  17. ##set sink to hdfs
    18. 

  18. a1.sinks.k1.type=hdfs
    19. 

  19. a1.sinks.k1.hdfs.path= hdfs://node01:8020/flume/failover/
    20. 

  20. a1.sinks.k1.hdfs.fileType=DataStream
    21. 

  21. a1.sinks.k1.hdfs.writeFormat=TEXT
    22. 

  22. a1.sinks.k1.hdfs.rollInterval=
    23. 

  23. a1.sinks.k1.channel=c1
    24. 

  24. a1.sinks.k1.hdfs.filePrefix=%Y-%m-%d
    25. 

  25. #


  1. # Name the components on this agent
    2. 

  2. a1.sources = r1
    3. 

  3. a1.sinks = k1
    4. 

  4. a1.channels = c1
    5. 

    6. 

  6. # Describe/configure the source
    7. 

  7. a1.sources.r1.type = avro
    8. 

  8. a1.sources.r1.channels = c1
    9. 

  9. a1.sources.r1.bind = node02
    10. 

  10. a1.sources.r1.port = 
    11. 

    12. 

  12. # Describe the sink
    13. 

  13. a1.sinks.k1.type = logger
    14. 

    15. 

  15. # Use a channel which buffers events in memory
    16. 

  16. a1.channels.c1.type = memory
    17. 

  17. a1.channels.c1.capacity = 
    18. 

  18. a1.channels.c1.transactionCapacity = 
    19. 

    20. 

  20. # Bind the source and sink to the channel
    21. 

  21. a1.sources.r1.channels = c1
    22. 

  22. a1.sinks.k1.channel = c1

node03机器修改配置文件

  1. cd  /export/servers/apache-flume-1.8.-bin/conf
    2. 

  2. vim collector.conf


  1. #set Agent name
    2. 

  2. a1.sources = r1
    3. 

  3. a1.channels = c1
    4. 

  4. a1.sinks = k1
    5. 

  5. #
    6. 

  6. ##set channel
    7. 

  7. a1.channels.c1.type = memory
    8. 

  8. a1.channels.c1.capacity = 
    9. 

  9. a1.channels.c1.transactionCapacity = 
    10. 

  10. #
    11. 

  11. ## other node,nna to nns
    12. 

  12. a1.sources.r1.type = avro
    13. 

  13. a1.sources.r1.bind = node03
    14. 

  14. a1.sources.r1.port = 
    15. 

  15. a1.sources.r1.channels = c1
    16. 

  16. #
    17. 

  17. ##set sink to hdfs
    18. 

  18. a1.sinks.k1.type=hdfs
    19. 

  19. a1.sinks.k1.hdfs.path= hdfs://node01:8020/flume/failover/
    20. 

  20. a1.sinks.k1.hdfs.fileType=DataStream
    21. 

  21. a1.sinks.k1.hdfs.writeFormat=TEXT
    22. 

  22. a1.sinks.k1.hdfs.rollInterval=
    23. 

  23. a1.sinks.k1.channel=c1
    24. 

  24. a1.sinks.k1.hdfs.filePrefix=%Y-%m-%d


  1. # Name the components on this agent
    2. 

  2. a1.sources = r1
    3. 

  3. a1.sinks = k1
    4. 

  4. a1.channels = c1
    5. 

    6. 

  6. # Describe/configure the source
    7. 

  7. a1.sources.r1.type = avro
    8. 

  8. a1.sources.r1.channels = c1
    9. 

  9. a1.sources.r1.bind = node03
    10. 

  10. a1.sources.r1.port = 
    11. 

    12. 

  12. # Describe the sink
    13. 

  13. a1.sinks.k1.type = logger
    14. 

    15. 

  15. # Use a channel which buffers events in memory
    16. 

  16. a1.channels.c1.type = memory
    17. 

  17. a1.channels.c1.capacity = 
    18. 

  18. a1.channels.c1.transactionCapacity = 
    19. 

    20. 

  20. # Bind the source and sink to the channel
    21. 

  21. a1.sources.r1.channels = c1
    22. 

  22. a1.sinks.k1.channel = c1

3.4. 顺序启动

node03机器上面启动flume

  1. cd /export/servers/apache-flume-1.8.-bin
    2. 

  2. bin/flume-ng agent -n a1 -c conf -f conf/collector.conf -Dflume.root.logger=DEBUG,console

node02机器上面启动flume

  1. cd /export/servers/apache-flume-1.8.-bin
    2. 

  2. bin/flume-ng agent -n a1 -c conf -f conf/collector.conf -Dflume.root.logger=DEBUG,console

node01机器上面启动flume

  1. cd /export/servers/apache-flume-1.8.-bin
    2. 

  2. bin/flume-ng agent -n agent1 -c conf -f conf/agent.conf -Dflume.root.logger=DEBUG,console

node01机器启动文件产生脚本

  1. cd  /export/servers/shells
    2. 

  2. sh tail-file.sh

3.5. Failover 测试

下面我们来测试下Flume NG集群的高可用（故障转移）。场景如下：我们在Agent1节点上传文件，由于我们配置Collector1的权重比Collector2大，所以 Collector1优先采集并上传到存储系统。然后我们kill掉Collector1，此时有Collector2负责日志的采集上传工作，之后，我 们手动恢复Collector1节点的Flume服务，再次在Agent1上次文件，发现Collector1恢复优先级别的采集工作。具体截图如下所 示：

Collector1优先上传

HDFS集群中上传的log内容预览

Collector1宕机，Collector2获取优先上传权限

重启Collector1服务，Collector1重新获得优先上传的权限

4. flume 的负载均衡

负载均衡是用于解决一台机器(一个进程)无法解决所有请求而产生的一种算法。Load balancing Sink Processor 能够实现 load balance 功能，如下图Agent1 是一个路由节点，负责将 Channel 暂存的 Event 均衡到对应的多个 Sink组件上，而每个 Sink 组件分别连接到一个独立的 Agent 上，示例配置，如下所示：

在此处我们通过三台机器来进行模拟flume的负载均衡

三台机器规划如下：

node01：采集数据，发送到node02和node03机器上去

node02：接收node01的部分数据

node03：接收node01的部分数据

第一步：开发node01服务器的flume配置

node01服务器配置：

  1. cd /export/servers/apache-flume-1.8.-bin/conf
    2. 

  2. vim load_banlancer_client.conf


  1. #agent name
    2. 

    3. 

  3. a1.channels = c1
    4. 

  4. a1.sources = r1
    5. 

  5. a1.sinks = k1 k2
    6. 

    7. 

    8. 

    9. 

  9. #set gruop
    10. 

    11. 

  11. a1.sinkgroups = g1
    12. 

    13. 

  13. #set channel
    14. 

    15. 

  15. a1.channels.c1.type = memory
    16. 

  16. a1.channels.c1.capacity = 
    17. 

  17. a1.channels.c1.transactionCapacity = 
    18. 

  18. a1.sources.r1.channels = c1
    19. 

  19. a1.sources.r1.type = exec
    20. 

  20. a1.sources.r1.command = tail -F /export/servers/taillogs/access_log
    21. 

    22. 

  22. # set sink1
    23. 

    24. 

  24. a1.sinks.k1.channel = c1
    25. 

  25. a1.sinks.k1.type = avro
    26. 

  26. #对接端口
    27. 

  27. a1.sinks.k1.hostname = node02
    28. 

  28. a1.sinks.k1.port = 
    29. 

    30. 

  30. # set sink2
    31. 

  31. a1.sinks.k2.channel = c1
    32. 

  32. a1.sinks.k2.type = avro
    33. 

  33. #对接端口
    34. 

  34. a1.sinks.k2.hostname = node03
    35. 

  35. a1.sinks.k2.port = 
    36. 

  36. #set sink group
    37. 

    38. 

  38. a1.sinkgroups.g1.sinks = k1 k2
    39. 

    40. 

  40. #set failover
    41. 

  41. #负载均衡
    42. 

  42. a1.sinkgroups.g1.processor.type = load_balance
    43. 

  43. a1.sinkgroups.g1.processor.backoff = true
    44. 

  44. #轮训
    45. 

  45. a1.sinkgroups.g1.processor.selector = round_robin
    46. 

  46. a1.sinkgroups.g1.processor.selector.maxTimeOut=
    47.

第二步：开发node02服务器的flume配置

  1. cd /export/servers/apache-flume-1.8.-bin/conf
    2. 

    3. 

  3. vim load_banlancer_server.conf


  1. # Name the components on this agent
    2. 

    3. 

  3. a1.sources = r1
    4. 

  4. a1.sinks = k1
    5. 

  5. a1.channels = c1
    6. 

    7. 

  7. # Describe/configure the source
    8. 

    9. 

  9. a1.sources.r1.type = avro
    10. 

  10. a1.sources.r1.channels = c1
    11. 

  11. a1.sources.r1.bind = node02
    12. 

  12. a1.sources.r1.port = 
    13. 

    14. 

  14. # Describe the sink
    15. 

  15. a1.sinks.k1.type = logger
    16. 

    17. 

  17. # Use a channel which buffers events in memory
    18. 

  18. a1.channels.c1.type = memory
    19. 

  19. a1.channels.c1.capacity = 
    20. 

  20. a1.channels.c1.transactionCapacity = 
    21. 

    22. 

  22. # Bind the source and sink to the channel
    23. 

  23. a1.sources.r1.channels = c1
    24. 

  24. a1.sinks.k1.channel = c1
    25.

第三步：开发node03服务器flume配置

node03服务器配置

  1. cd /export/servers/apache-flume-1.8.-bin/conf
    2. 

  2. vim load_banlancer_server.conf


  1. # Name the components on this agent
    2. 

  2. a1.sources = r1
    3. 

  3. a1.sinks = k1
    4. 

  4. a1.channels = c1
    5. 

    6. 

  6. # Describe/configure the source
    7. 

  7. a1.sources.r1.type = avro
    8. 

  8. a1.sources.r1.channels = c1
    9. 

  9. a1.sources.r1.bind = node03
    10. 

  10. a1.sources.r1.port = 
    11. 

    12. 

  12. # Describe the sink
    13. 

  13. a1.sinks.k1.type = logger
    14. 

    15. 

  15. # Use a channel which buffers events in memory
    16. 

  16. a1.channels.c1.type = memory
    17. 

  17. a1.channels.c1.capacity = 
    18. 

  18. a1.channels.c1.transactionCapacity = 
    19. 

    20. 

    21. 

  21. # Bind the source and sink to the channel
    22. 

  22. a1.sources.r1.channels = c1
    23. 

  23. a1.sinks.k1.channel = c1

第四步：准备启动flume服务

启动node03的flume服务

  1. cd /export/servers/apache-flume-1.8.-bin
    2. 

  2. bin/flume-ng agent -n a1 -c conf -f conf/load_banlancer_server.conf -Dflume.root.logger=DEBUG,console

启动node02的flume服务

  1. cd /export/servers/apache-flume-1.8.-bin
    2. 

  2. bin/flume-ng agent -n a1 -c conf -f conf/load_banlancer_server.conf -Dflume.root.logger=DEBUG,console

启动node01的flume服务

  1. cd /export/servers/apache-flume-1.8.-bin
    2. 

    3. 

  3. bin/flume-ng agent -n a1 -c conf -f conf/load_banlancer_client.conf -Dflume.root.logger=DEBUG,console

第五步：node01服务器运行脚本产生数据

cd /export/servers/shells

sh tail-file.sh

5. Flume 案例-静态拦截器

1. 案例场景

A、B两台日志服务机器实时生产日志主要类型为access.log、nginx.log、web.log

现在要求：

把A、B 机器中的access.log、nginx.log、web.log 采集汇总到C机器上然后统一收集到hdfs中。

但是在hdfs中要求的目录为：

  1. /source/logs/access//**
    2. 

  2. /source/logs/nginx/20180101/**
    3. 

  3. /source/logs/web/20180101/**

2. 场景分析

​ 图一

3. 数据流程处理分析

4、实现

服务器A对应的IP为 192.168.174.100

服务器B对应的IP为 192.168.174.110

服务器C对应的IP为 node03

采集端配置文件开发

node01与node02服务器开发flume的配置文件

  1. cd /export/servers/apache-flume-1.8.-bin/conf
    2. 

  2. vim exec_source_avro_sink.conf


  1. # Name the components on this agent
    2. 

  2. a1.sources = r1 r2 r3
    3. 

  3. a1.sinks = k1
    4. 

  4. a1.channels = c1
    5. 

  5. # Describe/configure the source
    6. 

  6. a1.sources.r1.type = exec
    7. 

  7. a1.sources.r1.command = tail -F /export/servers/taillogs/access.log
    8. 

  8. a1.sources.r1.interceptors = i1
    9. 

  9. a1.sources.r1.interceptors.i1.type = static
    10. 

  10. ##  static拦截器的功能就是往采集到的数据的header中插入自己定## 义的key-value对
    11. 

  11. a1.sources.r1.interceptors.i1.key = type
    12. 

  12. a1.sources.r1.interceptors.i1.value = access
    13. 

    14. 

  14. a1.sources.r2.type = exec
    15. 

  15. a1.sources.r2.command = tail -F /export/servers/taillogs/nginx.log
    16. 

  16. a1.sources.r2.interceptors = i2
    17. 

  17. a1.sources.r2.interceptors.i2.type = static
    18. 

  18. a1.sources.r2.interceptors.i2.key = type
    19. 

  19. a1.sources.r2.interceptors.i2.value = nginx
    20. 

    21. 

  21. a1.sources.r3.type = exec
    22. 

  22. a1.sources.r3.command = tail -F /export/servers/taillogs/web.log
    23. 

  23. a1.sources.r3.interceptors = i3
    24. 

  24. a1.sources.r3.interceptors.i3.type = static
    25. 

  25. a1.sources.r3.interceptors.i3.key = type
    26. 

  26. a1.sources.r3.interceptors.i3.value = web
    27. 

    28. 

  28. # Describe the sink
    29. 

  29. a1.sinks.k1.type = avro
    30. 

  30. a1.sinks.k1.hostname = node03
    31. 

  31. a1.sinks.k1.port = 
    32. 

  32. # Use a channel which buffers events in memory
    33. 

  33. a1.channels.c1.type = memory
    34. 

  34. a1.channels.c1.capacity = 
    35. 

  35. a1.channels.c1.transactionCapacity = 
    36. 

    37. 

  37. # Bind the source and sink to the channel
    38. 

  38. a1.sources.r1.channels = c1
    39. 

  39. a1.sources.r2.channels = c1
    40. 

  40. a1.sources.r3.channels = c1
    41. 

  41. a1.sinks.k1.channel = c1
    42.

服务端配置文件开发

在node03上面开发flume配置文件

  1. cd /export/servers/apache-flume-1.8.-bin/conf
    2. 

  2. vim avro_source_hdfs_sink.conf


  1. a1.sources = r1
    2. 

  2. a1.sinks = k1
    3. 

  3. a1.channels = c1
    4. 

  4. #定义source
    5. 

  5. a1.sources.r1.type = avro
    6. 

  6. a1.sources.r1.bind = node03
    7. 

  7. a1.sources.r1.port =
    8. 

    9. 

  9. #添加时间拦截器
    10. 

  10. a1.sources.r1.interceptors = i1
    11. 

  11. a1.sources.r1.interceptors.i1.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
    12. 

  12. #定义channels
    13. 

  13. a1.channels.c1.type = memory
    14. 

  14. a1.channels.c1.capacity = 
    15. 

  15. a1.channels.c1.transactionCapacity = 
    16. 

  16. #定义sink
    17. 

  17. a1.sinks.k1.type = hdfs
    18. 

  18. a1.sinks.k1.hdfs.path=hdfs://node01:8020/source/logs/%{type}/%Y%m%d
    19. 

  19. a1.sinks.k1.hdfs.filePrefix =events
    20. 

  20. a1.sinks.k1.hdfs.fileType = DataStream
    21. 

  21. a1.sinks.k1.hdfs.writeFormat = Text
    22. 

  22. #时间类型
    23. 

  23. a1.sinks.k1.hdfs.useLocalTimeStamp = true
    24. 

  24. #生成的文件不按条数生成
    25. 

  25. a1.sinks.k1.hdfs.rollCount = 
    26. 

  26. #生成的文件按时间生成
    27. 

  27. a1.sinks.k1.hdfs.rollInterval = 
    28. 

  28. #生成的文件按大小生成
    29. 

  29. a1.sinks.k1.hdfs.rollSize  = 
    30. 

  30. #批量写入hdfs的个数
    31. 

  31. a1.sinks.k1.hdfs.batchSize = 
    32. 

  32. #flume操作hdfs的线程数（包括新建，写入等）
    33. 

  33. a1.sinks.k1.hdfs.threadsPoolSize=
    34. 

  34. #操作hdfs超时时间
    35. 

  35. a1.sinks.k1.hdfs.callTimeout=
    36. 

  36. #组装source、channel、sink
    37. 

  37. a1.sources.r1.channels = c1
    38. 

  38. a1.sinks.k1.channel = c1

采集端文件生成脚本

在node01与node02上面开发shell脚本，模拟数据生成

  1. cd /export/servers/shells
    2. 

  2. vim server.sh


  1. #!/bin/bash
    2. 

    3. 

  3. while true
    4. 

    5. 

  5. do 
    6. 

    7. 

  7. date >> /export/servers/taillogs/access.log;
    8. 

    9. 

  9. date >> /export/servers/taillogs/web.log;
    10. 

    11. 

  11. date >> /export/servers/taillogs/nginx.log;
    12. 

    13. 

  13.   sleep 0.5;
    14. 

    15. 

  15. done
    16.

顺序启动服务

node03启动flume实现数据收集

  1. cd /export/servers/apache-flume-1.8.-bin
    2. 

    3. 

  3. bin/flume-ng agent -c conf -f conf/avro_source_hdfs_sink.conf -name a1 -Dflume.root.logger=DEBUG,console

node01与node02启动flume实现数据监控

  1. cd /export/servers/apache-flume-1.8.-bin
    2. 

    3. 

  3. bin/flume-ng agent -c conf -f conf/exec_source_avro_sink.conf -name a1 -Dflume.root.logger=DEBUG,console

node01与node02启动生成文件脚本

  1. cd /export/servers/shells
    2. 

  2. sh server.sh

5、项目实现截图

6. Flume 案例二

案例需求：

在数据采集之后，通过flume的拦截器，实现不需要的数据过滤掉，并将指定的第一个字段进行加密，加密之后再往hdfs上面保存

原始数据与处理之后的数据对比

图一 原始文件内容

图二 HDFS上产生收集到的处理数

实现步骤

第一步：创建maven java工程，导入jar包

  1. <?xml version="1.0" encoding="UTF-8"?>
    2. 

  2. <project xmlns="http://maven.apache.org/POM/4.0.0"
    3. 

  3.  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    4. 

  4.  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    5. 

  5.  <modelVersion>4.0.0</modelVersion>
    6. 

    7. 

  7.  <groupId>cn.le.cloud</groupId>
    8. 

  8.  <artifactId>example-flume-intercepter</artifactId>
    9. 

  9.  <version>1.0-SNAPSHOT</version>
    10. 

    11. 

  11.  <dependencies>
    12. 

  12.  <dependency>
    13. 

  13.  <groupId>org.apache.flume</groupId>
    14. 

  14.  <artifactId>flume-ng-sdk</artifactId>
    15. 

  15.  <version>1.8.0</version>
    16. 

  16.  </dependency>
    17. 

  17.  <dependency>
    18. 

  18.  <groupId>org.apache.flume</groupId>
    19. 

  19.  <artifactId>flume-ng-core</artifactId>
    20. 

  20.  <version>1.8.0</version>
    21. 

  21.  </dependency>
    22. 

  22.  </dependencies>
    23. 

    24. 

  24.  <build>
    25. 

  25.  <plugins>
    26. 

  26.  <plugin>
    27. 

  27.  <groupId>org.apache.maven.plugins</groupId>
    28. 

  28.  <artifactId>maven-compiler-plugin</artifactId>
    29. 

  29.  <version>3.0</version>
    30. 

  30.  <configuration>
    31. 

  31.  <source>1.8</source>
    32. 

  32.  <target>1.8</target>
    33. 

  33.  <encoding>UTF-8</encoding>
    34. 

  34.  <!--    <verbal>true</verbal>-->
    35. 

  35.  </configuration>
    36. 

  36.  </plugin>
    37. 

  37.  <plugin>
    38. 

  38.  <groupId>org.apache.maven.plugins</groupId>
    39. 

  39.  <artifactId>maven-shade-plugin</artifactId>
    40. 

  40.  <version>3.1.1</version>
    41. 

  41.  <executions>
    42. 

  42.  <execution>
    43. 

  43.  <phase>package</phase>
    44. 

  44.  <goals>
    45. 

  45.  <goal>shade</goal>
    46. 

  46.  </goals>
    47. 

  47.  <configuration>
    48. 

  48.  <filters>
    49. 

  49.  <filter>
    50. 

  50.  <artifact>*:*</artifact>
    51. 

  51.  <excludes>
    52. 

  52.  <exclude>META-INF/*.SF</exclude>
    53. 

  53.  <exclude>META-INF/*.DSA</exclude>
    54. 

  54.  <exclude>META-INF/*.RSA</exclude>
    55. 

  55.  </excludes>
    56. 

  56.  </filter>
    57. 

  57.  </filters>
    58. 

  58.  <transformers>
    59. 

  59.  <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
    60. 

  60.  <mainClass></mainClass>
    61. 

  61.  </transformer>
    62. 

  62.  </transformers>
    63. 

  63.  </configuration>
    64. 

  64.  </execution>
    65. 

  65.  </executions>
    66. 

  66.  </plugin>
    67. 

  67.  </plugins>
    68. 

  68.  </build>
    69. 

  69. </project>

第二步：自定义flume的拦截器

  1. package cn.le.iterceptor;
    2. 

  2. import com.google.common.base.Charsets;
    3. 

  3. import org.apache.flume.Context;
    4. 

  4. import org.apache.flume.Event;
    5. 

  5. import org.apache.flume.interceptor.Interceptor;
    6. 

  6. import java.security.MessageDigest;
    7. 

  7. import java.security.NoSuchAlgorithmException;
    8. 

  8. import java.util.ArrayList;
    9. 

  9. import java.util.List;
    10. 

  10. import java.util.regex.Matcher;
    11. 

  11. import java.util.regex.Pattern;
    12. 

    13. 

  13. import static cn.le.iterceptor.CustomParameterInterceptor.Constants.*;
    14. 

    15. 

  15. public class CustomParameterInterceptor implements Interceptor {
    16. 

  16.  /** The field_separator.指明每一行字段的分隔符 */
    17. 

  17.  private final String fields_separator;
    18. 

    19. 

  19.  /** The indexs.通过分隔符分割后，指明需要那列的字段 下标*/
    20. 

  20.  private final String indexs;
    21. 

    22. 

  22.  /** The indexs_separator. 多个下标的分隔符*/
    23. 

  23.  private final String indexs_separator;
    24. 

    25. 

  25.  /**
    26. 

  26.      *
    27. 

  27.      * @param indexs
    28. 

  28.      * @param indexs_separator
    29. 

  29.      */
    30. 

  30.  public CustomParameterInterceptor( String fields_separator,
    31. 

  31.  String indexs, String indexs_separator,String encrypted_field_index) {
    32. 

  32.  String f = fields_separator.trim();
    33. 

  33.  String i = indexs_separator.trim();
    34. 

  34.  this.indexs = indexs;
    35. 

  35.  this.encrypted_field_index=encrypted_field_index.trim();
    36. 

  36.  if (!f.equals("")) {
    37. 

  37.             f = UnicodeToString(f);
    38. 

  38.  }
    39. 

  39.  this.fields_separator =f;
    40. 

  40.  if (!i.equals("")) {
    41. 

  41.             i = UnicodeToString(i);
    42. 

  42.  }
    43. 

  43.  this.indexs_separator = i;
    44. 

  44.  }
    45. 

    46. 

  46.  /*
    47. 

  47.      *
    48. 

  48.      * \t 制表符 ('\u0009') \n 新行（换行）符 (' ') \r 回车符 (' ') \f 换页符 ('\u000C') \a 报警
    49. 

  49.      * (bell) 符 ('\u0007') \e 转义符 ('\u001B') \cx  空格(\u0020)对应于 x 的控制符
    50. 

  50.      *
    51. 

  51.      * @param str
    52. 

  52.      * @return
    53. 

  53.      * @data:2015-6-30
    54. 

  54.      */
    55. 

    56. 

  56.  /** The encrypted_field_index. 需要加密的字段下标*/
    57. 

  57.  private final String encrypted_field_index;
    58. 

  58.  public static String UnicodeToString(String str) {
    59. 

  59.  Pattern pattern = Pattern.compile("(\\\\u(\\p{XDigit}{4}))");
    60. 

  60.  Matcher matcher = pattern.matcher(str);
    61. 

  61.  char ch;
    62. 

  62.  while (matcher.find()) {
    63. 

  63.             ch = (char) Integer.parseInt(matcher.group(), );
    64. 

  64.             str = str.replace(matcher.group(), ch + "");
    65. 

  65.  }
    66. 

  66.  return str;
    67. 

  67.  }
    68. 

    69. 

  69.  /*
    70. 

  70.      * @see org.apache.flume.interceptor.Interceptor#intercept(org.apache.flume.Event)
    71. 

  71.      * 单个event拦截逻辑
    72. 

  72.      */
    73. 

  73.  public Event intercept(Event event) {
    74. 

  74.  if (event == null) {
    75. 

  75.  return null;
    76. 

  76.  }
    77. 

  77.  try {
    78. 

  78.  String line = new String(event.getBody(), Charsets.UTF_8);
    79. 

  79.  String[] fields_spilts = line.split(fields_separator);
    80. 

  80.  String[] indexs_split = indexs.split(indexs_separator);
    81. 

  81.  String newLine="";
    82. 

  82.  for (int i = ; i < indexs_split.length; i++) {
    83. 

  83.  int parseInt = Integer.parseInt(indexs_split[i]);
    84. 

  84.  //对加密字段进行加密
    85. 

  85.  if(!"".equals(encrypted_field_index)&&encrypted_field_index.equals(indexs_split[i])){
    86. 

  86.                     newLine+=StringUtils.GetMD5Code(fields_spilts[parseInt]);
    87. 

  87.  }else{
    88. 

  88.                     newLine+=fields_spilts[parseInt];
    89. 

  89.  }
    90. 

    91. 

  91.  if(i!=indexs_split.length-){
    92. 

  92.                     newLine+=fields_separator;
    93. 

  93.  }
    94. 

  94.  }
    95. 

  95.             event.setBody(newLine.getBytes(Charsets.UTF_8));
    96. 

  96.  return event;
    97. 

  97.  } catch (Exception e) {
    98. 

  98.  return event;
    99. 

  99.  }
    100. 

  100.  }
    101. 

    102. 

  102.  /*
    103. 

  103.      * @see org.apache.flume.interceptor.Interceptor#intercept(java.util.List)
    104. 

  104.      * 批量event拦截逻辑
    105. 

  105.      */
    106. 

  106.  public List<Event> intercept(List<Event> events) {
    107. 

  107.  List<Event> out = new ArrayList<Event>();
    108. 

  108.  for (Event event : events) {
    109. 

  109.  Event outEvent = intercept(event);
    110. 

  110.  if (outEvent != null) {
    111. 

  111.                 out.add(outEvent);
    112. 

  112.  }
    113. 

  113.  }
    114. 

  114.  return out;
    115. 

  115.  }
    116. 

    117. 

  117.  /*
    118. 

  118.      * @see org.apache.flume.interceptor.Interceptor#initialize()
    119. 

  119.      */
    120. 

  120.  public void initialize() {
    121. 

  121.  // TODO Auto-generated method stub
    122. 

    123. 

  123.  }
    124. 

    125. 

  125.  /*
    126. 

  126.      * @see org.apache.flume.interceptor.Interceptor#close()
    127. 

  127.      */
    128. 

  128.  public void close() {
    129. 

  129.  // TODO Auto-generated method stub
    130. 

    131. 

  131.  }
    132. 

    133. 

    134. 

  134.  /**
    135. 

  135.      * 相当于自定义Interceptor的工厂类
    136. 

  136.      * 在flume采集配置文件中通过制定该Builder来创建Interceptor对象
    137. 

  137.      * 可以在Builder中获取、解析flume采集配置文件中的拦截器Interceptor的自定义参数：
    138. 

  138.      * 字段分隔符，字段下标，下标分隔符、加密字段下标 ...等
    139. 

  139.      * @author
    140. 

  140.      *
    141. 

  141.      */
    142. 

  142.  public static class Builder implements Interceptor.Builder {
    143. 

    144. 

  144.  /** The fields_separator.指明每一行字段的分隔符 */
    145. 

  145.  private String fields_separator;
    146. 

    147. 

  147.  /** The indexs.通过分隔符分割后，指明需要那列的字段 下标*/
    148. 

  148.  private String indexs;
    149. 

    150. 

  150.  /** The indexs_separator. 多个下标下标的分隔符*/
    151. 

  151.  private String indexs_separator;
    152. 

    153. 

  153.  /** The encrypted_field. 需要加密的字段下标*/
    154. 

  154.  private String encrypted_field_index;
    155. 

  155.  /*
    156. 

  156.          * @see org.apache.flume.conf.Configurable#configure(org.apache.flume.Context)
    157. 

  157.          */
    158. 

  158.  public void configure(Context context) {
    159. 

  159.             fields_separator = context.getString(FIELD_SEPARATOR, DEFAULT_FIELD_SEPARATOR);
    160. 

  160.             indexs = context.getString(INDEXS, DEFAULT_INDEXS);
    161. 

  161.             indexs_separator = context.getString(INDEXS_SEPARATOR, DEFAULT_INDEXS_SEPARATOR);
    162. 

  162.             encrypted_field_index= context.getString(ENCRYPTED_FIELD_INDEX, DEFAULT_ENCRYPTED_FIELD_INDEX);
    163. 

  163.  }
    164. 

  164.  /*
    165. 

  165.          * @see org.apache.flume.interceptor.Interceptor.Builder#build()
    166. 

  166.          */
    167. 

  167.  public Interceptor build() {
    168. 

  168.  return new CustomParameterInterceptor(fields_separator, indexs, indexs_separator,encrypted_field_index);
    169. 

  169.  }
    170. 

  170.  }
    171. 

  171.  /**
    172. 

  172.      * 常量
    173. 

  173.      *
    174. 

  174.      */
    175. 

  175.  public static class Constants {
    176. 

  176.  /** The Constant FIELD_SEPARATOR. */
    177. 

  177.  public static final String FIELD_SEPARATOR = "fields_separator";
    178. 

    179. 

  179.  /** The Constant DEFAULT_FIELD_SEPARATOR. */
    180. 

  180.  public static final String DEFAULT_FIELD_SEPARATOR =" ";
    181. 

    182. 

  182.  /** The Constant INDEXS. */
    183. 

  183.  public static final String INDEXS = "indexs";
    184. 

    185. 

  185.  /** The Constant DEFAULT_INDEXS. */
    186. 

  186.  public static final String DEFAULT_INDEXS = "0";
    187. 

    188. 

  188.  /** The Constant INDEXS_SEPARATOR. */
    189. 

  189.  public static final String INDEXS_SEPARATOR = "indexs_separator";
    190. 

    191. 

  191.  /** The Constant DEFAULT_INDEXS_SEPARATOR. */
    192. 

  192.  public static final String DEFAULT_INDEXS_SEPARATOR = ",";
    193. 

    194. 

  194.  /** The Constant ENCRYPTED_FIELD_INDEX. */
    195. 

  195.  public static final String ENCRYPTED_FIELD_INDEX = "encrypted_field_index";
    196. 

    197. 

  197.  /** The Constant DEFAUL_TENCRYPTED_FIELD_INDEX. */
    198. 

  198.  public static final String DEFAULT_ENCRYPTED_FIELD_INDEX = "";
    199. 

    200. 

  200.  /** The Constant PROCESSTIME. */
    201. 

  201.  public static final String PROCESSTIME = "processTime";
    202. 

  202.  /** The Constant PROCESSTIME. */
    203. 

  203.  public static final String DEFAULT_PROCESSTIME = "a";
    204. 

    205. 

  205.  }
    206. 

  206.  /**
    207. 

  207.      * 工具类：字符串md5加密
    208. 

  208.      */
    209. 

  209.  public static class StringUtils {
    210. 

  210.  // 全局数组
    211. 

  211.  private final static String[] strDigits = { "0", "1", "2", "3", "4", "5",
    212. 

  212.  "6", "7", "8", "9", "a", "b", "c", "d", "e", "f" };
    213. 

  213.  // 返回形式为数字跟字符串
    214. 

  214.  private static String byteToArrayString(byte bByte) {
    215. 

  215.  int iRet = bByte;
    216. 

  216.  // System.out.println("iRet="+iRet);
    217. 

  217.  if (iRet < ) {
    218. 

  218.                 iRet += ;
    219. 

  219.  }
    220. 

  220.  int iD1 = iRet / ;
    221. 

  221.  int iD2 = iRet % ;
    222. 

  222.  return strDigits[iD1] + strDigits[iD2];
    223. 

  223.  }
    224. 

    225. 

  225.  // 返回形式只为数字
    226. 

  226.  private static String byteToNum(byte bByte) {
    227. 

  227.  int iRet = bByte;
    228. 

  228.  System.out.println("iRet1=" + iRet);
    229. 

  229.  if (iRet < ) {
    230. 

  230.                 iRet += ;
    231. 

  231.  }
    232. 

  232.  return String.valueOf(iRet);
    233. 

  233.  }
    234. 

    235. 

  235.  // 转换字节数组为16进制字串
    236. 

  236.  private static String byteToString(byte[] bByte) {
    237. 

  237.  StringBuffer sBuffer = new StringBuffer();
    238. 

  238.  for (int i = ; i < bByte.length; i++) {
    239. 

  239.                 sBuffer.append(byteToArrayString(bByte[i]));
    240. 

  240.  }
    241. 

  241.  return sBuffer.toString();
    242. 

  242.  }
    243. 

    244. 

  244.  public static String GetMD5Code(String strObj) {
    245. 

  245.  String resultString = null;
    246. 

  246.  try {
    247. 

  247.                 resultString = new String(strObj);
    248. 

  248.  MessageDigest md = MessageDigest.getInstance("MD5");
    249. 

  249.  // md.digest() 该函数返回值为存放哈希值结果的byte数组
    250. 

  250.                 resultString = byteToString(md.digest(strObj.getBytes()));
    251. 

  251.  } catch (NoSuchAlgorithmException ex) {
    252. 

  252.                 ex.printStackTrace();
    253. 

  253.  }
    254. 

  254.  return resultString;
    255. 

  255.  }
    256. 

  256.  }
    257. 

    258. 

  258. }

第三步：打包上传服务器

将我们的拦截器打成jar包放到flume的lib目录下

第四步：开发flume的配置文件

第三台机器开发flume的配置文件

  1. cd  /export/servers/apache-flume-1.8.-bin/conf
    2. 

  2. vim spool-interceptor-hdfs.conf


  1. a1.channels = c1
    2. 

  2. a1.sources = r1
    3. 

  3. a1.sinks = s1
    4. 

  4. #channel
    5. 

    6. 

  6. a1.channels.c1.type = memory
    7. 

  7. a1.channels.c1.capacity=
    8. 

  8. a1.channels.c1.transactionCapacity=
    9. 

    10. 

  10. #source
    11. 

    12. 

  12. a1.sources.r1.channels = c1
    13. 

  13. a1.sources.r1.type = spooldir
    14. 

  14. a1.sources.r1.spoolDir = /export/servers/intercept
    15. 

  15. a1.sources.r1.batchSize= 
    16. 

  16. a1.sources.r1.inputCharset = UTF-
    17. 

    18. 

    19. 

    20. 

  20. a1.sources.r1.interceptors =i1 i2
    21. 

  21. a1.sources.r1.interceptors.i1.type =cn.le.iterceptor.CustomParameterInterceptor$Builder
    22. 

  22. a1.sources.r1.interceptors.i1.fields_separator=\\u0009
    23. 

    24. 

  24. a1.sources.r1.interceptors.i1.indexs =,,,,
    25. 

  25. a1.sources.r1.interceptors.i1.indexs_separator =\\u002c
    26. 

  26. a1.sources.r1.interceptors.i1.encrypted_field_index =
    27. 

  27. a1.sources.r1.interceptors.i2.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
    28. 

    29. 

  29. #sink
    30. 

    31. 

  31. a1.sinks.s1.channel = c1
    32. 

    33. 

  33. a1.sinks.s1.type = hdfs
    34. 

    35. 

  35. a1.sinks.s1.hdfs.path =hdfs://node01:8020/flume/intercept/%Y%m%d
    36. 

    37. 

  37. a1.sinks.s1.hdfs.filePrefix = event
    38. 

  38. a1.sinks.s1.hdfs.fileSuffix = .log
    39. 

  39. a1.sinks.s1.hdfs.rollSize = 
    40. 

  40. a1.sinks.s1.hdfs.rollInterval =
    41. 

  41. a1.sinks.s1.hdfs.rollCount = 
    42. 

  42. a1.sinks.s1.hdfs.batchSize = 
    43. 

    44. 

  44. a1.sinks.s1.hdfs.round = true
    45. 

  45. a1.sinks.s1.hdfs.roundUnit = minute
    46. 

  46. a1.sinks.s1.hdfs.threadsPoolSize = 
    47. 

  47. a1.sinks.s1.hdfs.useLocalTimeStamp = true
    48. 

  48. a1.sinks.s1.hdfs.minBlockReplicas = 
    49. 

  49. a1.sinks.s1.hdfs.fileType =DataStream
    50. 

  50. a1.sinks.s1.hdfs.writeFormat = Text
    51. 

  51. a1.sinks.s1.hdfs.callTimeout = 
    52. 

  52. a1.sinks.s1.hdfs.idleTimeout =

第五步：上传测试数据

上传我们的测试数据到/export/servers/intercept 这个目录下面去，如果目录不存在则创建

  1. mkdir  -p /export/servers/intercept

测试数据如下

  1. 13601249301       
    2. 

  2. 13601249302       
    3. 

  3. 13601249303       
    4. 

  4. 13601249304       
    5. 

  5. 13601249305       
    6. 

  6. 13601249306       
    7. 

  7. 13601249307       
    8. 

  8. 13601249308       
    9. 

  9. 13601249309       
    10. 

  10. 13601249310       
    11. 

  11. 13601249311       
    12. 

  12. 13601249312

第六步：启动flume

  1. cd /export/servers/apache-flume-1.8.-bin
    2. 

  2. bin/flume-ng agent -c conf -f conf/spool-interceptor-hdfs.conf -name a1 -Dflume.root.logger=DEBUG,console
    3. 




  1. package cn.le.flumesource;
    2. 

    3. 

  3. import org.apache.flume.Context;
    4. 

  4. import org.apache.flume.Event;
    5. 

  5. import org.apache.flume.EventDeliveryException;
    6. 

  6. import org.apache.flume.PollableSource;
    7. 

  7. import org.apache.flume.conf.Configurable;
    8. 

  8. import org.apache.flume.event.SimpleEvent;
    9. 

  9. import org.apache.flume.source.AbstractSource;
    10. 

  10. import org.slf4j.Logger;
    11. 

    12. 

  12. import java.util.ArrayList;
    13. 

  13. import java.util.HashMap;
    14. 

  14. import java.util.List;
    15. 

    16. 

  16. import static org.slf4j.LoggerFactory.*;
    17. 

    18. 

  18. public class MySqlSource extends AbstractSource implements Configurable, PollableSource {
    19. 

    20. 

  20.  //打印日志
    21. 

  21.  private static final Logger LOG = getLogger(MySqlSource.class);
    22. 

  22.  //定义sqlHelper
    23. 

  23.  private QueryMySql sqlSourceHelper;
    24. 

    25. 

    26. 

  26.  @Override
    27. 

  27.  public long getBackOffSleepIncrement() {
    28. 

  28.  return ;
    29. 

  29.  }
    30. 

    31. 

  31.  @Override
    32. 

  32.  public long getMaxBackOffSleepInterval() {
    33. 

  33.  return ;
    34. 

  34.  }
    35. 

    36. 

  36.  @Override
    37. 

  37.  public void configure(Context context) {
    38. 

  38.  //初始化
    39. 

  39.         sqlSourceHelper = new QueryMySql(context);
    40. 

  40.  }
    41. 

    42. 

  42.  @Override
    43. 

  43.  public PollableSource.Status process() throws EventDeliveryException {
    44. 

  44.  try {
    45. 

  45.  //查询数据表
    46. 

  46.  List<List<Object>> result = sqlSourceHelper.executeQuery();
    47. 

  47.  //存放event的集合
    48. 

  48.  List<Event> events = new ArrayList<>();
    49. 

  49.  //存放event头集合
    50. 

  50.  HashMap<String, String> header = new HashMap<>();
    51. 

  51.  //如果有返回数据，则将数据封装为event
    52. 

  52.  if (!result.isEmpty()) {
    53. 

  53.  List<String> allRows = sqlSourceHelper.getAllRows(result);
    54. 

  54.  Event event = null;
    55. 

  55.  for (String row : allRows) {
    56. 

  56.                     event = new SimpleEvent();
    57. 

  57.                     event.setBody(row.getBytes());
    58. 

  58.                     event.setHeaders(header);
    59. 

  59.                     events.add(event);
    60. 

  60.  }
    61. 

  61.  //将event写入channel
    62. 

  62.  this.getChannelProcessor().processEventBatch(events);
    63. 

  63.  //更新数据表中的offset信息
    64. 

  64.                 sqlSourceHelper.updateOffset2DB(result.size());
    65. 

  65.  }
    66. 

  66.  //等待时长
    67. 

  67.  Thread.sleep(sqlSourceHelper.getRunQueryDelay());
    68. 

  68.  return Status.READY;
    69. 

  69.  } catch (InterruptedException e) {
    70. 

  70.             LOG.error("Error procesing row", e);
    71. 

  71.  return Status.BACKOFF;
    72. 

  72.  }
    73. 

  73.  }
    74. 

    75. 

  75.  @Override
    76. 

  76.  public synchronized void stop() {
    77. 

  77.         LOG.info("Stopping sql source {} ...", getName());
    78. 

  78.  try {
    79. 

  79.  //关闭资源
    80. 

  80.             sqlSourceHelper.close();
    81. 

  81.  } finally {
    82. 

  82.  super.stop();
    83. 

  83.  }
    84. 

  84.  }
    85. 

  85. }
    86. 



  1. package cn.le.flumesource;
    2. 

    3. 

  3. import org.apache.flume.Context;
    4. 

  4. import org.apache.flume.conf.ConfigurationException;
    5. 

  5. import org.apache.http.ParseException;
    6. 

  6. import org.slf4j.Logger;
    7. 

  7. import org.slf4j.LoggerFactory;
    8. 

    9. 

  9. import java.sql.*;
    10. 

  10. import java.util.ArrayList;
    11. 

  11. import java.util.List;
    12. 

  12. import java.util.Properties;
    13. 

    14. 

  14. public class QueryMySql {
    15. 

  15.  private static final Logger LOG = LoggerFactory.getLogger(QueryMySql.class);
    16. 

    17. 

  17.  private int runQueryDelay, //两次查询的时间间隔
    18. 

  18.             startFrom, //开始id
    19. 

  19.             currentIndex, //当前id
    20. 

  20.             recordSixe = , //每次查询返回结果的条数
    21. 

  21.             maxRow; //每次查询的最大条数
    22. 

    23. 

    24. 

  24.  private String table, //要操作的表
    25. 

  25.             columnsToSelect, //用户传入的查询的列
    26. 

  26.             customQuery, //用户传入的查询语句
    27. 

  27.             query, //构建的查询语句
    28. 

  28.             defaultCharsetResultSet;//编码集
    29. 

    30. 

  30.  //上下文，用来获取配置文件
    31. 

  31.  private Context context;
    32. 

    33. 

  33.  //为定义的变量赋值（默认值），可在flume任务的配置文件中修改
    34. 

  34.  private static final int DEFAULT_QUERY_DELAY = ;
    35. 

  35.  private static final int DEFAULT_START_VALUE = ;
    36. 

  36.  private static final int DEFAULT_MAX_ROWS = ;
    37. 

  37.  private static final String DEFAULT_COLUMNS_SELECT = "*";
    38. 

  38.  private static final String DEFAULT_CHARSET_RESULTSET = "UTF-8";
    39. 

    40. 

  40.  private static Connection conn = null;
    41. 

  41.  private static PreparedStatement ps = null;
    42. 

  42.  private static String connectionURL, connectionUserName, connectionPassword;
    43. 

    44. 

  44.  //加载静态资源
    45. 

  45.  static {
    46. 

  46.  Properties p = new Properties();
    47. 

  47.  try {
    48. 

  48.             p.load(QueryMySql.class.getClassLoader().getResourceAsStream("jdbc.properties"));
    49. 

  49.             connectionURL = p.getProperty("dbUrl");
    50. 

  50.             connectionUserName = p.getProperty("dbUser");
    51. 

  51.             connectionPassword = p.getProperty("dbPassword");
    52. 

  52.  Class.forName(p.getProperty("dbDriver"));
    53. 

  53.  } catch (Exception e) {
    54. 

  54.             LOG.error(e.toString());
    55. 

  55.  }
    56. 

  56.  }
    57. 

    58. 

  58.  //获取JDBC连接
    59. 

  59.  private static Connection InitConnection(String url, String user, String pw) {
    60. 

  60.  try {
    61. 

  61.  Connection conn = DriverManager.getConnection(url, user, pw);
    62. 

  62.  if (conn == null)
    63. 

  63.  throw new SQLException();
    64. 

  64.  return conn;
    65. 

  65.  } catch (SQLException e) {
    66. 

  66.             e.printStackTrace();
    67. 

  67.  }
    68. 

  68.  return null;
    69. 

  69.  }
    70. 

    71. 

  71.  //构造方法
    72. 

  72.  QueryMySql(Context context) throws ParseException {
    73. 

  73.  //初始化上下文
    74. 

  74.  this.context = context;
    75. 

    76. 

  76.  //有默认值参数：获取flume任务配置文件中的参数，读不到的采用默认值
    77. 

  77.  this.columnsToSelect = context.getString("columns.to.select", DEFAULT_COLUMNS_SELECT);
    78. 

  78.  this.runQueryDelay = context.getInteger("run.query.delay", DEFAULT_QUERY_DELAY);
    79. 

  79.  this.startFrom = context.getInteger("start.from", DEFAULT_START_VALUE);
    80. 

  80.  this.defaultCharsetResultSet = context.getString("default.charset.resultset", DEFAULT_CHARSET_RESULTSET);
    81. 

    82. 

  82.  //无默认值参数：获取flume任务配置文件中的参数
    83. 

  83.  this.table = context.getString("table");
    84. 

  84.  this.customQuery = context.getString("custom.query");
    85. 

  85.         connectionURL = context.getString("connection.url");
    86. 

  86.         connectionUserName = context.getString("connection.user");
    87. 

  87.         connectionPassword = context.getString("connection.password");
    88. 

  88.         conn = InitConnection(connectionURL, connectionUserName, connectionPassword);
    89. 

    90. 

  90.  //校验相应的配置信息，如果没有默认值的参数也没赋值，抛出异常
    91. 

  91.         checkMandatoryProperties();
    92. 

  92.  //获取当前的id
    93. 

  93.         currentIndex = getStatusDBIndex(startFrom);
    94. 

  94.  //构建查询语句
    95. 

  95.         query = buildQuery();
    96. 

  96.  }
    97. 

    98. 

  98.  //校验相应的配置信息（表，查询语句以及数据库连接的参数）
    99. 

  99.  private void checkMandatoryProperties() {
    100. 

  100.  if (table == null) {
    101. 

  101.  throw new ConfigurationException("property table not set");
    102. 

  102.  }
    103. 

  103.  if (connectionURL == null) {
    104. 

  104.  throw new ConfigurationException("connection.url property not set");
    105. 

  105.  }
    106. 

  106.  if (connectionUserName == null) {
    107. 

  107.  throw new ConfigurationException("connection.user property not set");
    108. 

  108.  }
    109. 

  109.  if (connectionPassword == null) {
    110. 

  110.  throw new ConfigurationException("connection.password property not set");
    111. 

  111.  }
    112. 

  112.  }
    113. 

    114. 

  114.  //构建sql语句
    115. 

  115.  private String buildQuery() {
    116. 

  116.  String sql = "";
    117. 

  117.  //获取当前id
    118. 

  118.         currentIndex = getStatusDBIndex(startFrom);
    119. 

  119.         LOG.info(currentIndex + "");
    120. 

  120.  if (customQuery == null) {
    121. 

  121.             sql = "SELECT " + columnsToSelect + " FROM " + table;
    122. 

  122.  } else {
    123. 

  123.             sql = customQuery;
    124. 

  124.  }
    125. 

  125.  StringBuilder execSql = new StringBuilder(sql);
    126. 

  126.  //以id作为offset
    127. 

  127.  if (!sql.contains("where")) {
    128. 

  128.             execSql.append(" where ");
    129. 

  129.             execSql.append("id").append(">").append(currentIndex);
    130. 

  130.  return execSql.toString();
    131. 

  131.  } else {
    132. 

  132.  int length = execSql.toString().length();
    133. 

  133.  return execSql.toString().substring(, length - String.valueOf(currentIndex).length()) + currentIndex;
    134. 

  134.  }
    135. 

  135.  }
    136. 

    137. 

  137.  //执行查询
    138. 

  138.  List<List<Object>> executeQuery() {
    139. 

  139.  try {
    140. 

  140.  //每次执行查询时都要重新生成sql，因为id不同
    141. 

  141.             customQuery = buildQuery();
    142. 

  142.  //存放结果的集合
    143. 

  143.  List<List<Object>> results = new ArrayList<>();
    144. 

  144.  if (ps == null) {
    145. 

  145.  //
    146. 

  146.                 ps = conn.prepareStatement(customQuery);
    147. 

  147.  }
    148. 

  148.  ResultSet result = ps.executeQuery(customQuery);
    149. 

  149.  while (result.next()) {
    150. 

  150.  //存放一条数据的集合（多个列）
    151. 

  151.  List<Object> row = new ArrayList<>();
    152. 

  152.  //将返回结果放入集合
    153. 

  153.  for (int i = ; i <= result.getMetaData().getColumnCount(); i++) {
    154. 

  154.                     row.add(result.getObject(i));
    155. 

  155.  }
    156. 

  156.                 results.add(row);
    157. 

  157.  }
    158. 

  158.             LOG.info("execSql:" + customQuery + "\nresultSize:" + results.size());
    159. 

  159.  return results;
    160. 

  160.  } catch (SQLException e) {
    161. 

  161.             LOG.error(e.toString());
    162. 

  162.  // 重新连接
    163. 

  163.             conn = InitConnection(connectionURL, connectionUserName, connectionPassword);
    164. 

  164.  }
    165. 

  165.  return null;
    166. 

  166.  }
    167. 

    168. 

  168.  //将结果集转化为字符串，每一条数据是一个list集合，将每一个小的list集合转化为字符串
    169. 

  169.  List<String> getAllRows(List<List<Object>> queryResult) {
    170. 

  170.  List<String> allRows = new ArrayList<>();
    171. 

  171.  if (queryResult == null || queryResult.isEmpty())
    172. 

  172.  return allRows;
    173. 

  173.  StringBuilder row = new StringBuilder();
    174. 

  174.  for (List<Object> rawRow : queryResult) {
    175. 

  175.  Object value = null;
    176. 

  176.  for (Object aRawRow : rawRow) {
    177. 

  177.                 value = aRawRow;
    178. 

  178.  if (value == null) {
    179. 

  179.                     row.append(",");
    180. 

  180.  } else {
    181. 

  181.                     row.append(aRawRow.toString()).append(",");
    182. 

  182.  }
    183. 

  183.  }
    184. 

  184.             allRows.add(row.toString());
    185. 

  185.             row = new StringBuilder();
    186. 

  186.  }
    187. 

  187.  return allRows;
    188. 

  188.  }
    189. 

    190. 

  190.  //更新offset元数据状态，每次返回结果集后调用。必须记录每次查询的offset值，为程序中断续跑数据时使用，以id为offset
    191. 

  191.  void updateOffset2DB(int size) {
    192. 

  192.  //以source_tab做为KEY，如果不存在则插入，存在则更新（每个源表对应一条记录）
    193. 

  193.  String sql = "insert into flume_meta(source_tab,currentIndex) VALUES('"
    194. 

  194.  + this.table
    195. 

  195.  + "','" + (recordSixe += size)
    196. 

  196.  + "') on DUPLICATE key update source_tab=values(source_tab),currentIndex=values(currentIndex)";
    197. 

  197.         LOG.info("updateStatus Sql:" + sql);
    198. 

  198.         execSql(sql);
    199. 

  199.  }
    200. 

    201. 

  201.  //执行sql语句
    202. 

  202.  private void execSql(String sql) {
    203. 

  203.  try {
    204. 

  204.             ps = conn.prepareStatement(sql);
    205. 

  205.             LOG.info("exec::" + sql);
    206. 

  206.             ps.execute();
    207. 

  207.  } catch (SQLException e) {
    208. 

  208.             e.printStackTrace();
    209. 

  209.  }
    210. 

  210.  }
    211. 

    212. 

  212.  //获取当前id的offset
    213. 

  213.  private Integer getStatusDBIndex(int startFrom) {
    214. 

  214.  //从flume_meta表中查询出当前的id是多少
    215. 

  215.  String dbIndex = queryOne("select currentIndex from flume_meta where source_tab='" + table + "'");
    216. 

  216.  if (dbIndex != null) {
    217. 

  217.  return Integer.parseInt(dbIndex);
    218. 

  218.  }
    219. 

  219.  //如果没有数据，则说明是第一次查询或者数据表中还没有存入数据，返回最初传入的值
    220. 

  220.  return startFrom;
    221. 

  221.  }
    222. 

    223. 

  223.  //查询一条数据的执行语句(当前id)
    224. 

  224.  private String queryOne(String sql) {
    225. 

  225.  ResultSet result = null;
    226. 

  226.  try {
    227. 

  227.             ps = conn.prepareStatement(sql);
    228. 

  228.             result = ps.executeQuery();
    229. 

  229.  while (result.next()) {
    230. 

  230.  return result.getString();
    231. 

  231.  }
    232. 

  232.  } catch (SQLException e) {
    233. 

  233.             e.printStackTrace();
    234. 

  234.  }
    235. 

  235.  return null;
    236. 

  236.  }
    237. 

    238. 

  238.  //关闭相关资源
    239. 

  239.  void close() {
    240. 

  240.  try {
    241. 

  241.             ps.close();
    242. 

  242.             conn.close();
    243. 

  243.  } catch (SQLException e) {
    244. 

  244.             e.printStackTrace();
    245. 

  245.  }
    246. 

  246.  }
    247. 

    248. 

  248.  int getCurrentIndex() {
    249. 

  249.  return currentIndex;
    250. 

  250.  }
    251. 

    252. 

  252.  void setCurrentIndex(int newValue) {
    253. 

  253.         currentIndex = newValue;
    254. 

  254.  }
    255. 

    256. 

  256.  int getRunQueryDelay() {
    257. 

  257.  return runQueryDelay;
    258. 

  258.  }
    259. 

    260. 

  260.  String getQuery() {
    261. 

  261.  return query;
    262. 

  262.  }
    263. 

    264. 

  264.  String getConnectionURL() {
    265. 

  265.  return connectionURL;
    266. 

  266.  }
    267. 

    268. 

  268.  private boolean isCustomQuerySet() {
    269. 

  269.  return (customQuery != null);
    270. 

  270.  }
    271. 

    272. 

  272.  Context getContext() {
    273. 

  273.  return context;
    274. 

  274.  }
    275. 

    276. 

  276.  public String getConnectionUserName() {
    277. 

  277.  return connectionUserName;
    278. 

  278.  }
    279. 

    280. 

  280.  public String getConnectionPassword() {
    281. 

  281.  return connectionPassword;
    282. 

  282.  }
    283. 

    284. 

  284.  String getDefaultCharsetResultSet() {
    285. 

  285.  return defaultCharsetResultSet;
    286. 

  286.  }
    287. 

  287. }

Hadoop-(Flume)的更多相关文章

  1. hadoop flume 架构及监控的部署

    1 Flume架构解释  Flume概念 Flume是一个分布式 ,可靠的,和高可用的,海量的日志聚合系统 支持在系统中定制各类的数据发送方 用于收集数据 提供简单的数据提取能力 并写入到各种接受方 ...

  2. flume学习安装

    近期项目组有需求点击流日志须要自己收集,学习了一下flume而且成功安装了.相关信息记录一下. 1)下载flume1.5版本号  wget http://www.apache.org/dyn/clos ...

  3. flume从kafka读取数据到hdfs中的配置

    #source的名字 agent.sources = kafkaSource # channels的名字,建议按照type来命名 agent.channels = memoryChannel # si ...

  4. 最新Hadoop Shell完全讲解

    本文为原创博客,转载请注明出处:http://www.cnblogs.com/MrFee/p/4683953.html    1.appendToFile   功能:将一个或多个源文件系统的内容追加至 ...

  5. Flume集群搭建

    0. 软件版本下载 http://mirror.bit.edu.cn/apache/flume/   1. 集群环境 Master 172.16.11.97 Slave1 172.16.11.98 S ...

  6. flume学习笔记

    #################################################################################################### ...

  7. Flume定时启动任务 防止挂掉

    一,查看Flume条数:ps -ef|grep java|grep flume|wc -l       ==>15 检查进程:给sh脚本添加权限,chmod 777 xx.sh #!/bin/s ...

  8. flume 日志导入elasticsearch

    Flume配置 . flume生成的数据结构 <span style="font-size:18px;">"_index" : "logs ...

  9. Flume+Morphlines实现数据的实时ETL

    转载:http://mp.weixin.qq.com/s/xCSdkQo1XMQwU91lch29Uw Apache Flume介绍: Apache Flume是一个Apache的开源项目,是一个分布 ...

  10. Flume Channel Selectors + kafka

    http://flume.apache.org/FlumeUserGuide.html#custom-channel-selector 官方文档上channel selectors 有两种类型: Re ...

随机推荐

  1. ARTS打卡计划第十五周

    Algorithms: https://leetcode-cn.com/problems/single-number/submissions/ Review: “What Makes a Good D ...

  2. 预处理、const、static与sizeof-static全局变量与普通的全局变量有什么区别

    1:全局变量的说明之前再加上static就构成了静态的全局变量.全局变量本身就是静态存储方式,静态全局变量当然也是静态存储方式.这两者在存储方式上并无不同.这两者的区别在于,非静态全局变量的作用域是整 ...

  3. 【sed】进阶

      sed的基本用法已能满足大多数需求,但当需要时,知道这些高级特效的存在及如何使用将提供莫大的帮助!   1. 多行命令         sed编辑器提供三个用于处理多行文本的特殊命令: N:将数据 ...

  4. centos6.9实现双网卡绑定

    1.创建bond0文件 # vi /etc/sysconfig/network-scripts/ifcfg-bond0 DEVICE=bond0 NM_CONTROLLED=no #是否由networ ...

  5. redis常见7种使用场景

    一,简单字符串缓存实例 $redis->connect('127.0.0.1', 6379); $strCacheKey = 'Test_bihu'; //SET 应用 $arrCacheDat ...

  6. C# 7 .NET / CLR / Visual Studio version requirements

    C# 7 .NET / CLR / Visual Studio version requirements   You do NOT need to target .NET 4.6 and above, ...

  7. activemq备忘

    ActiveMQ队列消息积压问题调研 http://blog.51cto.com/winters1224/2049432ActiveMQ的插件开发介绍 https://blog.csdn.net/zh ...

  8. [Java]简单计算下一段Java代码段运行了多少秒

    long startTime = System.currentTimeMillis(); ...... long endTime = System.currentTimeMillis(); logge ...

  9. mybatis配置文件祥解（mybatis.xml）

    以下是mybatis.xml文件,提倡放在src目录下,文件名任意 <?xml version="1.0" encoding="UTF-8"?> & ...

  10. easyUI之函数

    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <hea ...