DStream是类似于RDD概念,是对数据的抽象封装。它是一序列的RDD,事实上,它大部分的操作都是对RDD支持的操作的封装,不同的是,每次DStream都要遍历它内部所有的RDD执行这些操作。它可以由StreamingContext通过流数据产生或者其他DStream使用map方法产生(与RDD一样)
time属性对DStream而言非常重要,DStream里面的RDD就是通过某个时间间隔产生的,而且以产生的时间为索引。所以在访问DStream的某个RDD时,实际上是访问它在某个时间点的RDD。




/**
* A Discretized Stream (DStream), the basic abstraction in Spark Streaming, is a continuous
* sequence of RDDs (of the same type) representing a continuous stream of data (see
* org.apache.spark.rdd.RDD in the Spark core documentation for more details on RDDs).
* DStreams can either be created from live data (such as, data from TCP sockets, Kafka, Flume,
* etc.) using a [[org.apache.spark.streaming.StreamingContext]] or it can be generated by
* transforming existing DStreams using operations such as `map`,
* `window` and `reduceByKeyAndWindow`. While a Spark Streaming program is running, each DStream
* periodically generates a RDD, either from live data or by transforming the RDD generated by a
* parent DStream.
*
* This class contains the basic operations available on all DStreams, such as `map`, `filter` and
* `window`. In addition, [[org.apache.spark.streaming.dstream.PairDStreamFunctions]] contains
* operations available only on DStreams of key-value pairs, such as `groupByKeyAndWindow` and
* `join`. These operations are automatically available on any DStream of pairs
* (e.g., DStream[(Int, Int)] through implicit conversions when
* `org.apache.spark.streaming.StreamingContext._` is imported.
*
* DStreams internally is characterized by a few basic properties:
* - A list of other DStreams that the DStream depends on
* - A time interval at which the DStream generates an RDD
* - A function that is used to generate an RDD after each time interval
*/

abstract class DStream[T: ClassTag] (
@transient private[streaming] var ssc: StreamingContext
) extends Serializable with Logging {
重要属性:
// =======================================================================
// Methods that should be implemented by subclasses of DStream
// =======================================================================
/** Time interval after which the DStream generates a RDD */
def slideDuration: Duration
/** List of parent DStreams on which this DStream depends on */
def dependencies: List[DStream[_]]
/** Method that generates a RDD for the given time */
def compute (validTime: Time): Option[RDD[T]]
当前已经产生了的RDD,以产生的时间为索引
// =======================================================================
// Methods and fields available on all DStreams
// =======================================================================

// RDDs generated, marked as private[streaming] so that testsuites can access it
@transient
private[streaming] var generatedRDDs = new HashMap[Time, RDD[T]] ()
为某个时间点产生一个RDD
/**
* Get the RDD corresponding to the given time; either retrieve it from cache
* or compute-and-cache it.
*/
private[streaming] def getOrCompute(time: Time): Option[RDD[T]] = {














spark streaming 2: DStream的更多相关文章

  1. 53、Spark Streaming:输入DStream之Kafka数据源实战

    一.基于Receiver的方式 1.概述 基于Receiver的方式: Receiver是使用Kafka的高层次Consumer API来实现的.receiver从Kafka中获取的数据都是存储在Sp ...

  2. Spark入门实战系列--7.Spark Streaming(上)--实时流计算Spark Streaming原理介绍

    [注]该系列文章以及使用到安装包/测试数据 可以在<倾情大奉送--Spark入门实战系列>获取 .Spark Streaming简介 1.1 概述 Spark Streaming 是Spa ...

  3. Spark Streaming

    Spark Streaming Spark Streaming 是Spark为了用户实现流式计算的模型. 数据源包括Kafka,Flume,HDFS等. DStream 离散化流(discretize ...

  4. Spark学习之Spark Streaming

    一.简介 许多应用需要即时处理收到的数据,例如用来实时追踪页面访问统计的应用.训练机器学习模型的应用,还有自动检测异常的应用.Spark Streaming 是 Spark 为这些应用而设计的模型.它 ...

  5. Spark Streaming 实现思路与模块概述

    一.基于 Spark 做 Spark Streaming 的思路 Spark Streaming 与 Spark Core 的关系可以用下面的经典部件图来表述: 在本节,我们先探讨一下基于 Spark ...

  6. .Spark Streaming(上)--实时流计算Spark Streaming原理介

    Spark入门实战系列--7.Spark Streaming(上)--实时流计算Spark Streaming原理介绍 http://www.cnblogs.com/shishanyuan/p/474 ...

  7. spark streaming的理解和应用

    1.Spark Streaming简介 官方网站解释:http://spark.apache.org/docs/latest/streaming-programming-guide.html 该博客转 ...

  8. 实时流计算Spark Streaming原理介绍

    1.Spark Streaming简介 1.1 概述 Spark Streaming 是Spark核心API的一个扩展,可以实现高吞吐量的.具备容错机制的实时流数据的处理.支持从多种数据源获取数据,包 ...

  9. Spark Streaming之一:整体介绍

    提到Spark Streaming,我们不得不说一下BDAS(Berkeley Data Analytics Stack),这个伯克利大学提出的关于数据分析的软件栈.从它的视角来看,目前的大数据处理可 ...

随机推荐

  1. L2Dwidget.js L2D网页动画人物添加

    hexo 添加live2d看板动画 https://www.jianshu.com/p/3a6342e16e57 首先贴出官网代码 官网地址配置:https://www.npmjs.com/packa ...

  2. iServer-Linux环境下开机自启动实现

    备注:该方案的前提是linux环境下已经安装部署好了iServer 1.在/etc/init.d/目录下创建iserver服务脚本文件. [root@localhost /]# vim /etc/in ...

  3. c# log4net安装时在AssemblyInfo中提示找不到log4net解决办法

    在安装log4net时,按照安装手册需要在AssemblyInfo.cs里添加log4net的配置信息 [assembly: log4net.Config.XmlConfigurator(Config ...

  4. RouterOS Firewall v6 流程图

    1. Firewall v5和Firewall v6对比图 2.Firewall v6的流程图

  5. Delphi Edit组件

  6. 韦东山嵌入式Linux学习笔记07--Nandflash

    常用的flash有两种, Norflash和Nandflash, 前几年市场上的产品比较常见的方案时Norflash和Nandflash搭配使用, 因为norflash比较昂贵,相同的容量norfla ...

  7. Linux进程管理工具之ps

    1.PS进程管理指令 ps    -aux USER:用户名称 PID:进程号 %CPU:进程占用CPU的百分比 %MEM:进程占用物理内存的百分比 VSZ:进程占用的虚拟内存大小(单位:KB) RS ...

  8. Linux添加虚拟网卡的多种方法

    Linux添加虚拟网卡的多种方法有时候,一台服务器需要设置多个ip,但又不想添加多块网卡,那就需要设置虚拟网卡.这里介绍几种方式在linux服务器上添加虚拟网卡. 我们向eth0中添加一块虚拟网卡: ...

  9. 浙大数据结构课后习题 练习二 7-3 Pop Sequence (25 分)

    Given a stack which can keep M numbers at most. Push N numbers in the order of 1, 2, 3, ..., N and p ...

  10. python file对象测试数据的读写操作及OS模块介绍(四)

    import   from....import 引入模块 引入类 ①import 如果文件在lib下而且是python模块 :import 模块名. ②from....import from 包名.包 ...