Spark Streaming - DStream
map
, reduce
, join
and window
. Finally, processed data can be pushed out to filesystems, databases, and live dashboards. step2 Define the streaming computations by applying transformation and output operations to DStreams.
step3 Start receiving data and processing it using streamingContext.start().
step4 Wait for the processing to be stopped (manually or due to any error) using streamingContext.awaitTermination().
step5 The processing can be manually stopped using streamingContext.stop().
Points to remember:
- Once a context has been started, no new streaming computations can be set up or added to it.
- Once a context has been stopped, it cannot be restarted.
- Only one StreamingContext can be active in a JVM at the same time.
- stop() on StreamingContext also stops the SparkContext. To stop only the StreamingContext, set the optional parameter of
stop()
calledstopSparkContext
to false. - A SparkContext can be re-used to create multiple StreamingContexts, as long as the previous StreamingContext is stopped (without stopping the SparkContext) before the next StreamingContext is created.
Points to remember
When running a Spark Streaming program locally, do not use “local” or “local[1]” as the master URL. Either of these means that only one thread will be used for running tasks locally. If you are using a input DStream based on a receiver (e.g. sockets, Kafka, Flume, etc.), then the single thread will be used to run the receiver, leaving no thread for processing the received data. Hence, when running locally, always use “local[n]” as the master URL, where n > number of receivers to run (see Spark Properties for information on how to set the master).
Extending the logic to running on a cluster, the number of cores allocated to the Spark Streaming application must be more than the number of receivers. Otherwise the system will receive data, but not be able to process it.
window length - The duration of the window (3 in the figure).
sliding interval - The interval at which the window operation is performed (2 in the figure).
// Reduce last 30 seconds of data, every 10 seconds
val windowedWordCounts = pairs.reduceByKeyAndWindow((a:Int,b:Int) => (a + b), Seconds(30), Seconds(10))
4.1 Reducing the Batch Processing Times
Spark Streaming - DStream的更多相关文章
- 58、Spark Streaming: DStream的output操作以及foreachRDD详解
一.output操作 1.output操作 DStream中的所有计算,都是由output操作触发的,比如print().如果没有任何output操作,那么,压根儿就不会执行定义的计算逻辑. 此外,即 ...
- 54、Spark Streaming:DStream的transformation操作概览
一. transformation操作概览 Transformation Meaning map 对传入的每个元素,返回一个新的元素 flatMap 对传入的每个元素,返回一个或多个元素 filter ...
- spark streaming(2) DAG静态定义及DStream,DStreamGraph
DAG 中文名有向无环图.它不是spark独有技术.它是一种编程思想 ,甚至于hadoop阵营里也有运用DAG的技术,比如Tez,Oozie.有意思的是,Tez是从MapReduce的基础上深化而来的 ...
- 大数据技术之_19_Spark学习_04_Spark Streaming 应用解析 + Spark Streaming 概述、运行、解析 + DStream 的输入、转换、输出 + 优化
第1章 Spark Streaming 概述1.1 什么是 Spark Streaming1.2 为什么要学习 Spark Streaming1.3 Spark 与 Storm 的对比第2章 运行 S ...
- Spark Streaming源码分析 – DStream
A Discretized Stream (DStream), the basic abstraction in Spark Streaming, is a continuous sequence o ...
- spark streaming 2: DStream
DStream是类似于RDD概念,是对数据的抽象封装.它是一序列的RDD,事实上,它大部分的操作都是对RDD支持的操作的封装,不同的是,每次DStream都要遍历它内部所有的RDD执行这些操作.它可以 ...
- Spark Streaming消费Kafka Direct方式数据零丢失实现
使用场景 Spark Streaming实时消费kafka数据的时候,程序停止或者Kafka节点挂掉会导致数据丢失,Spark Streaming也没有设置CheckPoint(据说比较鸡肋,虽然可以 ...
- Spark Streaming
Spark Streaming Spark Streaming 是Spark为了用户实现流式计算的模型. 数据源包括Kafka,Flume,HDFS等. DStream 离散化流(discretize ...
- spark streaming kafka1.4.1中的低阶api createDirectStream使用总结
转载:http://blog.csdn.net/ligt0610/article/details/47311771 由于目前每天需要从kafka中消费20亿条左右的消息,集群压力有点大,会导致job不 ...
随机推荐
- 笔记: js构造函数与原型
目录 构造函数与原型介绍 涉及三种引用的操作 有关原型及原型链的一些相关方法总结 @ 构造函数与原型介绍 1.函数与函数的原型对象(prototype object): 在JavaScript中,创建 ...
- JQ中的选择器children()和find()区别
1:children及find方法都用是用来获得element的子elements的,两者都不会返回 text node,就像大多数的jQuery方法一样. 2:children方法获得的仅仅是元素一 ...
- linux操作之软件安装(一)
rpm 包安装 RedHat Package Manager的缩写 , linux 的软件包可能存在依赖关系,比如某某依赖某某才能使用. 挂载一个光盘 mount -t auto /dev/cdrom ...
- 为什么我要放弃javaScript数据结构与算法(第十章)—— 排序和搜索算法
本章将会学习最常见的排序和搜索算法,如冒泡排序.选择排序.插入排序.归并排序.快速排序和堆排序,以及顺序排序和二叉搜索算法. 第十章 排序和搜索算法 排序算法 我们会从一个最慢的开始,接着是一些性能好 ...
- Lingo解决最优化问题
目录 Lingo解决优化问题 前言 一.优化模型介绍 二.运输问题 2.1 问题描述 2.2 问题分析 2.2 优化模型构建 2.3 模型求解 2.4 求解结果 三.待更新 Lingo解决优化问题 @ ...
- 成都Uber优步司机奖励政策(2月23日)
滴快车单单2.5倍,注册地址:http://www.udache.com/ 如何注册Uber司机(全国版最新最详细注册流程)/月入2万/不用抢单:http://www.cnblogs.com/mfry ...
- CF 643 E. Bear and Destroying Subtrees
E. Bear and Destroying Subtrees http://codeforces.com/problemset/problem/643/E 题意: Q个操作. 加点,在原来的树上加一 ...
- LSD_SLAM(一)运行数据集
虽然按照作者的推荐使用的是OpenCV 2.4.8,但编译依然通不过,一直有错误: 因为lsd_slam是2014年出的,用的ros编译系统不是catkin而是rosbuild,有一些不理解的地方. ...
- spring源码-国际化-3.5
一.国际化在实际代码中是非常常见的一中方式.为了结合web做一下语言上面的切换,而达到展示的目的. 二.这里呢,主要是介绍spring中对于国际化做了哪些处理. 三.实现方式 1)xml配置 < ...
- ubuntu14.04上安装Java
apt-get安装 sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install orac ...