spark streaming 2: DStream
/**
* A Discretized Stream (DStream), the basic abstraction in Spark Streaming, is a continuous
* sequence of RDDs (of the same type) representing a continuous stream of data (see
* org.apache.spark.rdd.RDD in the Spark core documentation for more details on RDDs).
* DStreams can either be created from live data (such as, data from TCP sockets, Kafka, Flume,
* etc.) using a [[org.apache.spark.streaming.StreamingContext]] or it can be generated by
* transforming existing DStreams using operations such as `map`,
* `window` and `reduceByKeyAndWindow`. While a Spark Streaming program is running, each DStream
* periodically generates a RDD, either from live data or by transforming the RDD generated by a
* parent DStream.
*
* This class contains the basic operations available on all DStreams, such as `map`, `filter` and
* `window`. In addition, [[org.apache.spark.streaming.dstream.PairDStreamFunctions]] contains
* operations available only on DStreams of key-value pairs, such as `groupByKeyAndWindow` and
* `join`. These operations are automatically available on any DStream of pairs
* (e.g., DStream[(Int, Int)] through implicit conversions when
* `org.apache.spark.streaming.StreamingContext._` is imported.
*
* DStreams internally is characterized by a few basic properties:
* - A list of other DStreams that the DStream depends on
* - A time interval at which the DStream generates an RDD
* - A function that is used to generate an RDD after each time interval
*/
abstract class DStream[T: ClassTag] (
@transient private[streaming] var ssc: StreamingContext
) extends Serializable with Logging {
// =======================================================================
// Methods that should be implemented by subclasses of DStream
// =======================================================================
/** Time interval after which the DStream generates a RDD */
def slideDuration: Duration
/** List of parent DStreams on which this DStream depends on */
def dependencies: List[DStream[_]]
/** Method that generates a RDD for the given time */
def compute (validTime: Time): Option[RDD[T]]
// =======================================================================
// Methods and fields available on all DStreams
// =======================================================================
// RDDs generated, marked as private[streaming] so that testsuites can access it
@transient
private[streaming] var generatedRDDs = new HashMap[Time, RDD[T]] ()
/**
* Get the RDD corresponding to the given time; either retrieve it from cache
* or compute-and-cache it.
*/
private[streaming] def getOrCompute(time: Time): Option[RDD[T]] = {
spark streaming 2: DStream的更多相关文章
- 53、Spark Streaming:输入DStream之Kafka数据源实战
一.基于Receiver的方式 1.概述 基于Receiver的方式: Receiver是使用Kafka的高层次Consumer API来实现的.receiver从Kafka中获取的数据都是存储在Sp ...
- Spark入门实战系列--7.Spark Streaming(上)--实时流计算Spark Streaming原理介绍
[注]该系列文章以及使用到安装包/测试数据 可以在<倾情大奉送--Spark入门实战系列>获取 .Spark Streaming简介 1.1 概述 Spark Streaming 是Spa ...
- Spark Streaming
Spark Streaming Spark Streaming 是Spark为了用户实现流式计算的模型. 数据源包括Kafka,Flume,HDFS等. DStream 离散化流(discretize ...
- Spark学习之Spark Streaming
一.简介 许多应用需要即时处理收到的数据,例如用来实时追踪页面访问统计的应用.训练机器学习模型的应用,还有自动检测异常的应用.Spark Streaming 是 Spark 为这些应用而设计的模型.它 ...
- Spark Streaming 实现思路与模块概述
一.基于 Spark 做 Spark Streaming 的思路 Spark Streaming 与 Spark Core 的关系可以用下面的经典部件图来表述: 在本节,我们先探讨一下基于 Spark ...
- .Spark Streaming(上)--实时流计算Spark Streaming原理介
Spark入门实战系列--7.Spark Streaming(上)--实时流计算Spark Streaming原理介绍 http://www.cnblogs.com/shishanyuan/p/474 ...
- spark streaming的理解和应用
1.Spark Streaming简介 官方网站解释:http://spark.apache.org/docs/latest/streaming-programming-guide.html 该博客转 ...
- 实时流计算Spark Streaming原理介绍
1.Spark Streaming简介 1.1 概述 Spark Streaming 是Spark核心API的一个扩展,可以实现高吞吐量的.具备容错机制的实时流数据的处理.支持从多种数据源获取数据,包 ...
- Spark Streaming之一:整体介绍
提到Spark Streaming,我们不得不说一下BDAS(Berkeley Data Analytics Stack),这个伯克利大学提出的关于数据分析的软件栈.从它的视角来看,目前的大数据处理可 ...
随机推荐
- L2Dwidget.js L2D网页动画人物添加
hexo 添加live2d看板动画 https://www.jianshu.com/p/3a6342e16e57 首先贴出官网代码 官网地址配置:https://www.npmjs.com/packa ...
- iServer-Linux环境下开机自启动实现
备注:该方案的前提是linux环境下已经安装部署好了iServer 1.在/etc/init.d/目录下创建iserver服务脚本文件. [root@localhost /]# vim /etc/in ...
- c# log4net安装时在AssemblyInfo中提示找不到log4net解决办法
在安装log4net时,按照安装手册需要在AssemblyInfo.cs里添加log4net的配置信息 [assembly: log4net.Config.XmlConfigurator(Config ...
- RouterOS Firewall v6 流程图
1. Firewall v5和Firewall v6对比图 2.Firewall v6的流程图
- Delphi Edit组件
- 韦东山嵌入式Linux学习笔记07--Nandflash
常用的flash有两种, Norflash和Nandflash, 前几年市场上的产品比较常见的方案时Norflash和Nandflash搭配使用, 因为norflash比较昂贵,相同的容量norfla ...
- Linux进程管理工具之ps
1.PS进程管理指令 ps -aux USER:用户名称 PID:进程号 %CPU:进程占用CPU的百分比 %MEM:进程占用物理内存的百分比 VSZ:进程占用的虚拟内存大小(单位:KB) RS ...
- Linux添加虚拟网卡的多种方法
Linux添加虚拟网卡的多种方法有时候,一台服务器需要设置多个ip,但又不想添加多块网卡,那就需要设置虚拟网卡.这里介绍几种方式在linux服务器上添加虚拟网卡. 我们向eth0中添加一块虚拟网卡: ...
- 浙大数据结构课后习题 练习二 7-3 Pop Sequence (25 分)
Given a stack which can keep M numbers at most. Push N numbers in the order of 1, 2, 3, ..., N and p ...
- python file对象测试数据的读写操作及OS模块介绍(四)
import from....import 引入模块 引入类 ①import 如果文件在lib下而且是python模块 :import 模块名. ②from....import from 包名.包 ...