7.spark Streaming 技术内幕 : 从DSteam到RDD全过程解析
上篇博客讨论了Spark Streaming 程序动态生成Job的过程,并留下一个疑问: JobScheduler将动态生成的Job提交,然后调用了Job对象的run方法,最后run方法的调用是如何触发RDD的Action操作,从而真正触发Job的执行的呢?本文就具体讲解这个问题。

object WordCount{
def main(args:Array[String]):Unit={
val sparkConf =newSparkConf().setMaster("local[4]").setAppName("WordCount")
val ssc =newStreamingContext(sparkConf,Seconds(1))
val lines = ssc.socketTextStream("localhost",9999)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x =>(x,1)).reduceByKey(_+_)
wordCounts.print()
ssc.start()
ssc.awaitTermination()
}
}







Output Operation | Meaning |
---|---|
print() | Prints the first ten elements of every batch of data in a DStream on the driver node running the streaming application. This is useful for development and debugging. Python API This is called pprint() in the Python API. |
saveAsTextFiles(prefix, [suffix]) | Save this DStream's contents as text files. The file name at each batch interval is generated based onprefix and suffix: "prefix-TIME_IN_MS[.suffix]". |
saveAsObjectFiles(prefix, [suffix]) | Save this DStream's contents as SequenceFiles of serialized Java objects. The file name at each batch interval is generated based on prefix and suffix: "prefix-TIME_IN_MS[.suffix]". Python API This is not available in the Python API. |
saveAsHadoopFiles(prefix, [suffix]) | Save this DStream's contents as Hadoop files. The file name at each batch interval is generated based on prefix and suffix: "prefix-TIME_IN_MS[.suffix]". Python API This is not available in the Python API. |
foreachRDD(func) | The most generic output operator that applies a function, func, to each RDD generated from the stream. This function should push the data in each RDD to an external system, such as saving the RDD to files, or writing it over the network to a database. Note that the function func is executed in the driver process running the streaming application, and will usually have RDD actions in it that will force the computation of the streaming RDDs. |









7.spark Streaming 技术内幕 : 从DSteam到RDD全过程解析的更多相关文章
- 9. Spark Streaming技术内幕 : Receiver在Driver的精妙实现全生命周期彻底研究和思考
原创文章,转载请注明:转载自 听风居士博客(http://www.cnblogs.com/zhouyf/) Spark streaming 程序需要不断接收新数据,然后进行业务逻辑 ...
- Spark streaming技术内幕6 : Job动态生成原理与源码解析
原创文章,转载请注明:转载自 周岳飞博客(http://www.cnblogs.com/zhouyf/) Spark streaming 程序的运行过程是将DStream的操作转化成RDD的操作,S ...
- 6.Spark streaming技术内幕 : Job动态生成原理与源码解析
原创文章,转载请注明:转载自 周岳飞博客(http://www.cnblogs.com/zhouyf/) Spark streaming 程序的运行过程是将DStream的操作转化成RDD的操作, ...
- 贯通Spark Streaming JobScheduler内幕实现和深入思考
本节主要内容: 一.SparkStreaming Job生成深度思考 二.SparkStreaming Job生成源码解析 JobScheduler的地位非常的重要,所有的关键都在JobSchedul ...
- 16.Spark Streaming源码解读之数据清理机制解析
原创文章,转载请注明:转载自 听风居士博客(http://www.cnblogs.com/zhouyf/) 本期内容: 一.Spark Streaming 数据清理总览 二.Spark Streami ...
- 10.Spark Streaming源码分析:Receiver数据接收全过程详解
原创文章,转载请注明:转载自 听风居士博客(http://www.cnblogs.com/zhouyf/) 在上一篇中介绍了Receiver的整体架构和设计原理,本篇内容主要介绍Receiver在 ...
- Spark技术内幕:Sort Based Shuffle实现解析
在Spark 1.2.0中,Spark Core的一个重要的升级就是将默认的Hash Based Shuffle换成了Sort Based Shuffle,即spark.shuffle.manager ...
- 9.Spark Streaming
Spark Streaming 1 Why Apache Spark 2 关于Apache Spark 3 如何安装Apache Spark 4 Apache Spark的工作原理 5 spark弹性 ...
- Spark Streaming的优化之路—从Receiver到Direct模式
作者:个推数据研发工程师 学长 1 业务背景 随着大数据的快速发展,业务场景越来越复杂,离线式的批处理框架MapReduce已经不能满足业务,大量的场景需要实时的数据处理结果来进行分析.决 ...
随机推荐
- lnmp 环境搭建
3.安装 lnmp : wget http://soft.vpser.net/lnmp/lnmp1.3-full.tar.gz tar -zxvf lnmp1.3-full.tar.gz cd lnm ...
- MySQL5.5安装图解
MySQL5.5安装图解... ====================== 第一部分:去官网下载MySQL安装包... MySQL下载官网:https://dev.mysql.com/downloa ...
- Linux下iptables安全配置
Linux下配置IPTables,只开放特定端口,禁用其他网络. *filter :INPUT DROP [0:0] :FORWARD DROP [0:0] :OUTPUT ACCEPT [0:0] ...
- poi-对于word的操作(二)
poi对于word文本的底纹和下划线的样式的展现 package poi.test; import java.io.FileOutputStream; import java.math.BigInte ...
- 1-shell学习(bash)
1.为什么需要学习shell: (1)通用性,基本上所有的linux机器都会支持 (2)文字传输操作更快 (3)以后的系统管理需要使用 2.知识点: (1)变量相关:
- HDU 5868 Different Circle Permutation Burnside引理+矩阵快速幂+逆元
题意:有N个座位,人可以选座位,但选的座位不能相邻,且旋转不同构的坐法有几种.如4个座位有3种做法.\( 1≤N≤1000000000 (10^9) \). 题解:首先考虑座位不相邻的选法问题,如果不 ...
- C语言数据结构-栈
一.栈的定义 栈(statck)这种数据结构在计算机中是相当出名的.栈中的数据是先进后出的(First In Last Out, FILO).栈只有一个出口,允许新增元素(只能在栈顶上增加). 移出元 ...
- 启动hbase输出ignoring option PermSize=128m; support was removed in 8.0告警信息
./start-hbase.sh starting master, logging to /home/hadoop/hbase-1.2.4/bin/../logs/hbase-hadoop-maste ...
- free命令buff和cache的区别
[root@iz8vb8kca7twx6177bqg0fz ~]# free -h total used free shared b ...
- resultAPI示例
什么是Restfull API Restfull API 从字面就可以知道,他是rest式的接口,所以就要先了解什么是rest rest 不是一个技术,也不是一个协议 rest 指的是一组架构约束条件 ...