cdh环境下,spark streaming与flume的集成问题总结
文章发自:http://www.cnblogs.com/hark0623/p/4170156.html 转发请注明
如何做集成,其实特别简单,网上其实就是教程。


package com.hark import java.io.File import org.apache.spark.SparkConf
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.flume.FlumeUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.StreamingContext._ /**
* Created by Administrator on 2014-12-16.
*/
object SparkStreamingFlumeTest {
def main(args: Array[String]) {
//println("harkhark") val path = new File(".").getCanonicalPath()
//File workaround = new File(".");
System.getProperties().put("hadoop.home.dir", path);
new File("./bin").mkdirs();
new File("./bin/winutils.exe").createNewFile(); //val sparkConf = new SparkConf().setAppName("HdfsWordCount").setMaster("local[2]")
val sparkConf = new SparkConf().setAppName("HdfsWordCount") // Create the context
val ssc = new StreamingContext(sparkConf, Seconds(20)) //val hostname = "127.0.0.1"
val hostname = "localhost"
val port = 2345
val storageLevel = StorageLevel.MEMORY_ONLY
val flumeStream = FlumeUtils.createStream(ssc, hostname, port, storageLevel) flumeStream.count().map(cnt => "Received " + cnt + " flume events." ).print() ssc.start()
ssc.awaitTermination() }
}
# Please paste flume.conf here. Example: # Sources, channels, and sinks are defined per
# agent name, in this case 'tier1'.
tier1.sources = source1
tier1.channels = channel1
tier1.sinks = sink1 # For each source, channel, and sink, set
# standard properties.
tier1.sources.source1.type = exec
tier1.sources.source1.command = tail -F /opt/data/test3/123
tier1.sources.source1.channels = channel1
tier1.channels.channel1.type = memory
#tier1.sinks.sink1.type = logger
tier1.sinks.sink1.type = avro
tier1.sinks.sink1.hostname = localhost
tier1.sinks.sink1.port = 2345
tier1.sinks.sink1.channel = channel1 # Other properties are specific to each type of yhx.hadoop.dn01
# source, channel, or sink. In this case, we
# specify the capacity of the memory channel.
tier1.channels.channel1.capacity = 100
spark-submit --driver-memory 512m --executor-memory 512m --executor-cores 1 --num-executors 3 --class com.hark.SparkStreamingFlumeTest --deploy-mode cluster --master yarn /opt/spark/SparkTest.jar
flume-ng agent --conf /opt/cloudera-manager/run/cloudera-scm-agent/process/585-flume-AGENT --conf-file /opt/cloudera-manager/run/cloudera-scm-agent/process/585-flume-AGENT/flume.conf --name tier1 -Dflume.root.logger=INFO,console
cdh环境下,spark streaming与flume的集成问题总结的更多相关文章
- Spark Streaming揭秘 Day28 在集成开发环境中详解Spark Streaming的运行日志内幕
Spark Streaming揭秘 Day28 在集成开发环境中详解Spark Streaming的运行日志内幕 今天会逐行解析一下SparkStreaming运行的日志,运行的是WordCountO ...
- Spark学习之路(十五)—— Spark Streaming 整合 Flume
一.简介 Apache Flume是一个分布式,高可用的数据收集系统,可以从不同的数据源收集数据,经过聚合后发送到分布式计算框架或者存储系统中.Spark Straming提供了以下两种方式用于Flu ...
- Spark 系列(十五)—— Spark Streaming 整合 Flume
一.简介 Apache Flume 是一个分布式,高可用的数据收集系统,可以从不同的数据源收集数据,经过聚合后发送到分布式计算框架或者存储系统中.Spark Straming 提供了以下两种方式用于 ...
- Spark Streaming 整合 Flume
Spark Streaming 整合 Flume 一.简介二.推送式方法 2.1 配置日志收集Flume 2.2 项目依赖 2.3 Spark Strea ...
- Spark Streaming从Flume Poll数据案例实战和内幕源码解密
本节课分成二部分讲解: 一.Spark Streaming on Polling from Flume实战 二.Spark Streaming on Polling from Flume源码 第一部分 ...
- Spark Streaming处理Flume数据练习
把Flume Source(netcat类型),从终端上不断给Flume Source发送消息,Flume把消息汇集到Sink(avro类型),由Sink把消息推送给Spark Streaming并处 ...
- kerberos环境下spark消费kafka写入到Hbase
一.准备环境: 创建Kafka Topic和HBase表 1. 在kerberos环境下创建Kafka Topic 1.1 因为kafka默认使用的协议为PLAINTEXT,在kerberos环境下需 ...
- Centos 6.5 x64环境下 spark 1.6 maven 编译-- 已验证
Centos 6.5 x64 jdk 1.7 scala 2.10 maven 3.3.3 cd spark-1.6 export MAVEN_OPTS="-Xmx2g -XX:MaxPer ...
- Spark Streaming整合Flume + Kafka wordCount
flume配置文件 flume_to_kafka.conf a1.sources = r1 a1.sinks = k1 a1.channels = c1 a1.sources.r1.type = sp ...
随机推荐
- ACM/ICPC 之 两道dijkstra练习题(ZOJ1053(POJ1122)-ZOJ1053)
两道较为典型的单源最短路径问题,采用dijkstra解法 本来是四道练习题,后来发现后面两道用dijkstra来解的话总觉得有点冗余了,因此暂且分成三篇博客(本篇以及后两篇). ZOJ1053(POJ ...
- webpack学习笔记一
主要参考: https://blog.madewithlove.be/post/webpack-your-bags/ 起因: 作为运维狗, 对前端一窍不通但心向往之, 最近做一个Dashboard, ...
- js闭包问题
function picLinkInit(parentClassName, imgW, imgH, childClassObjs) { var $match = $(parentClassName); ...
- NEFU 506&&ZOJ 3353 Chess Board (四种构造的高斯消元)
题目链接 题意:有四种翻转方式,问是否能使得所有棋子都变为0,求最小步数. 题解:依次构造枚举求出最小值即可. #include <iostream> #include <cstdi ...
- IOS-frame和bounds有什么不同
frame指的是:该view在父view坐标系统中的位置和大小.(参照点是父亲的坐标系统) 它的坐标原点是随着父View位置的改变而改变的 bounds指的是:该view在本身坐标系统中 的位置和大小 ...
- iOS 本人必装插件
本人觉得比较好用也实用的Xcode插件记录: 1. Alcatraz 插件通过它来管理 : https://github.com/alcatraz/Alcatraz.git 2. Cocoa ...
- August 11th 2016, Week 33rd Thursday
A particular fine spring came around. 转眼又是一番分外明媚的春光. Hey, it is hot outside, sometimes even unbearab ...
- XStream xml to bean
<!-- pom.xml --> <dependency> <groupId>com.thoughtworks.xstream</groupId> &l ...
- Linux下C语言多线程,网络通信简单聊天程序
http://www.cnblogs.com/zhuxianji/archive/2011/01/06/1928970.html
- auto(c++11)
C++primer 第五版,第三章出现了此段程序,求解读附源码:代码1:#include<iostream>#include<string>using namespace st ...