测试flume,将数据送到hive表中,首先建表。

create table order_flume(
order_id string,
user_id string,
eval_set string,
order_number string,
order_dow string,
order_hour_of_day string,
days_since_prior_order string)
clustered by (order_id) into 5 buckets
stored as orc; # 记得加 orc,不然后面会出错

  flume conf 配置如下:

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1 # Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -f /root/code/flume_exec_test.txt # Describe the sink
a1.sinks.k1.type = hive
a1.sinks.k1.hive.metastore = thrift://master:9083
a1.sinks.k1.hive.database = hw
a1.sinks.k1.hive.table = order_flume
a1.sinks.k1.serializer = DELIMITED
a1.sinks.k1.serializer.delimiter = ","
a1.sinks.k1.serializer.serdeSeparator = ','
a1.sinks.k1.serializer.fieldnames = order_id,user_id,eval_set,order_number,order_dow,order_hour_of_day,days_since_prior_order # Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000 # 设置大一点,默认是1000
a1.channels.c1.transactionCapacity = 1000 # 设置大一点,默认是100 # Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

  这个时候如果启动flume的话会报错,需要将hive中的jar包移动到flume 中。

/usr/local/src/apache-hive-1.2.2-bin/hcatalog/share/hcatalog/* $FLUME_HOME/lib

/usr/local/src/apache-hive-1.2.2-bin/lib/* $FLUME_HOME/lib

  此时,在修改修改 hive-site.xml,将下面的值进行修改。

      <property>
<name>hive.support.concurrency</name>
<value>true</value>
</property>
<property>
<name>hive.exec.dynamic.partition.mode</name>
<value>nonstrict</value>
</property>
<property>
<name>hive.txn.manager</name>
<value>org.apache.hadoop.hive.ql.lockmgr.DbTxnManager</value>
</property>
<property>
<name>hive.compactor.initiator.on</name>
<value>true</value>
</property>
<property>
<name>hive.compactor.worker.threads</name>
<value>1</value>
</property>

  上面的配置完成之后,先启动 hive metastore,在启动 flume。

# 一定要重启
# 先重启 mysql
# 再重启 hadoop
# 再启动 hive metstore
# 再启动 flume service mysql restart start-dfs.sh
start-yarn.sh hive --service metastore flume-ng agent -c conf -f conf/hive.conf -n a1 -Dflume.root.logger=INFO,console

  一开始调试的时候,配置的都是对的,无论怎么跑都是连不上,都提示失败。

org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://master:9083', database='hw', table='order_flume', partitionVals=[] }
at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:343)
at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:295)
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flume.sink.hive.HiveWriter$TxnBatchException: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://master:9083', database='hw', table='oreder_flume', partitionVals=[] }
at org.apache.flume.sink.hive.HiveWriter.nextTxnBatch(HiveWriter.java:400)
at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:90)
... 6 more
Caused by: org.apache.hive.hcatalog.streaming.TransactionBatchUnAvailable: Unable to acquire transaction batch on end point: {metaStoreUri='thrift://master:9083', database='hw', table='order_flume', partitionVals=[] }
at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:514)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:464)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.fetchTransactionBatchImpl(HiveEndPoint.java:351)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.fetchTransactionBatch(HiveEndPoint.java:331)
at org.apache.flume.sink.hive.HiveWriter$9.call(HiveWriter.java:395)
at org.apache.flume.sink.hive.HiveWriter$9.call(HiveWriter.java:392)
at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.thrift.TApplicationException: Internal error processing open_txns
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_open_txns(ThriftHiveMetastore.java:4195)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.open_txns(ThriftHiveMetastore.java:4182)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxns(HiveMetaStoreClient.java:1988)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.openTxnImpl(HiveEndPoint.java:523)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:507)
... 10 more

  重启了几次还是同样的错误,后面尝试先启动 hive,去看一下其中表的情况,发现启动hive失败。

FAILED: LockException [Error 10280]: Error communicating with the metastore

  再次尝试重启 metastore,在尝试启动 hive,一开始尝试2次,发现还是同样的错误,结果过了几分钟,再次重新启动后,发现它自己好了,一直没有弄明白为什么,可能是之前没重启成功,重新编辑一下 hive-site.xml文件,在次保存退出,在重启可能会好。

最终flume 链接 hive 成功,信息如下。

2019-07-20 12:26:28,968 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:342)] k1: Creating Writer to Hive end point : {metaStoreUri='thrift://master:9083', database='hw', table='order_flume', partitionVals=[] }
2019-07-20 12:26:30,747 (hive-k1-call-runner-0) [INFO - org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:377)] Trying to connect to metastore with URI thrift://master:9083
2019-07-20 12:26:30,865 (hive-k1-call-runner-0) [INFO - org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:473)] Connected to metastore.
2019-07-20 12:26:31,494 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:377)] Trying to connect to metastore with URI thrift://master:9083
2019-07-20 12:26:31,516 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:473)] Connected to metastore.
2019-07-20 12:26:35,677 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hive.HiveWriter.nextTxnBatch(HiveWriter.java:335)] Acquired Txn Batch TxnIds=[1...100] on endPoint = {metaStoreUri='thrift://master:9083', database='hw', table='order_flume', partitionVals=[] }. Switching to first txn
2019-07-20 12:26:38,808 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hive.HiveWriter.commitTxn(HiveWriter.java:275)] Committing Txn 1 on EndPoint: {metaStoreUri='thrift://master:9083', database='hw', table='order_flume', partitionVals=[] }

  

flume 测试 hive sink的更多相关文章

  1. Flume + HDFS + Hive日志收集系统

    最近一段时间,负责公司的产品日志埋点与收集工作,搭建了基于Flume+HDFS+Hive日志搜集系统. 一.日志搜集系统架构: 简单画了一下日志搜集系统的架构图,可以看出,flume承担了agent与 ...

  2. Flume 测试 Kafka 案例

    Flume Kafka 测试案例,Flume 的配置. a1.sources = s1 a1.channels = c1 a1.sinks = k1 a1.sources.s1.type = netc ...

  3. 简单的Flume和hive的结合

    1. 日志格式 #Software: Microsoft Internet Information Services 6.0 #Version: 1.0 #Date: -- :: #Fields: d ...

  4. Flume配置Failover Sink Processor

    1 官网内容 2 看一张图一目了然 3 详细配置 source配置文件 #配置文件: a1.sources= r1 a1.sinks= k1 k2 a1.channels= c1 #负载平衡 a1.s ...

  5. 自定义flume的hbase sink 的序列化程序

    package com.hello.hbase; import java.nio.charset.Charset; import java.text.SimpleDateFormat; import ...

  6. 从0到1搭建基于Kafka、Flume和Hive的海量数据分析系统(一)数据收集应用

    大数据时代,一大技术特征是对海量数据采集.存储和分析的多组件解决方案.而其中对来自于传感器.APP的SDK和各类互联网应用的原生日志数据的采集存储则是基本中的基本.本系列文章将从0到1,概述一下搭建基 ...

  7. 流量分析系统---flume(测试flume+kafka)

    1.在flume官方网站下载最新的flume     wget http://124.205.69.169/files/A1540000011ED5DB/mirror.bit.edu.cn/apach ...

  8. Flume的Avro Sink和Avro Source研究之二 : Avro Sink

    啊,AvroSink要复杂好多:< 好吧,先确定主要问题: AvroSink为啥这么多代码?有必要吗?它都有哪些逻辑需要实现? 你看,avro-rpc-quickstart里是这么建client ...

  9. Hadoop实战-Flume之自定义Sink(十九)

    import java.io.File; import java.io.FileNotFoundException; import java.io.FileOutputStream; import j ...

随机推荐

  1. P1018 乘积最大(DP)

    题目 P1018 乘积最大 解析 区间DP 设\(f[i][j]\)表示选\(i\)个数,插入\(j\)个乘号时的最大值 设\(num[i][j]\)是\(s[i,j]\)里的数字 转移方程就是\(f ...

  2. 易百教程人工智能python修正-人工智能NLTK性别发现器

    在这个问题陈述中,将通过提供名字来训练分类器以找到性别(男性或女性). 我们需要使用启发式构造特征向量并训练分类器.这里使用scikit-learn软件包中的标签数据. 以下是构建性别查找器的Pyth ...

  3. pandas-04 多级index操作

    pandas-04 多级index操作 在pandas中可以为series和dataframe设置多个index,也就是说可以有多级index和column.这样可以对pandas的操作更加灵活. i ...

  4. Code Clean读书笔记

    代码整洁之道读书笔记 by fangpc 序言部分 "神在细节之中" - 建筑师路德维希 5S哲学(精益) 整理(Seiri):搞清楚事物之所在--通过恰当地命名之类的手段--至关 ...

  5. let 和 var 定义变量的区别

    一.变量提升 var 存在变量提升,而 let 不存在变量提升,所以用 let 定义的变量一定要在声明后再使用,否则会报错. var //var定义的变量存在变量提升,变量会把声明提升到整个作用域的最 ...

  6. Mac 环境变量的配置

    1.打开终端. 2.输入命令:sudo vi /etc/paths,然后enter,输入电脑开机密码,继续enter(这个地方的密码不会显示,只要你输入完了就按enter). 3.此时vi编辑器打开了 ...

  7. Upgrade Windows Server 2016 to Windows Server 2019

    Pre-Upgrade Upgrade path: Windows Server 2016 can be upgraded to Windows 2019 in a single upgrade pr ...

  8. Android笔记(七十五) Android中的图片压缩

    这几天在做图记的时候遇第一次遇到了OOM,好激动~~ 追究原因,是因为在ListView中加载的图片太大造成的,因为我使用的都是手机相机直接拍摄的照片,图片都比较大,所以在加载的时候会出现内存溢出,那 ...

  9. Vi 和 Vim 编辑器详细使用方法

    学习linux的一项必会技能,熟练使用vi/vim编辑器那便最重要的了.不过一堆操作看的也是太头疼了,以下整理了些常用到的命令. 工作模式 vi编辑界面有三种不同的工作模式,分别为命令模式.输入模式. ...

  10. 如何将scratch3.0的作业自动提交到后台数据库

    大家都知道Scratch3.0开发后,默认是可以下载文件到电脑,但是如果是作为商业系统来说,我们需要将作业自动的提交到后台,因此有了这篇文章. 首先,我们来分解下开发步骤: 1.在菜单栏新增一个上传到 ...