feature   

strom (trident) spark streaming 说明
并行框架
基于DAG的任务并行计算引擎(task parallel continuous computational engine Using DAG)
基于spark的数据并行计算引擎(data parallel general purpose batch processing engine)

数据处理模式
(one at a time)一次处理一个事件(消息)
trident: (Micro-batch)一次   处理多个事件
(Micro-batch)一次   处理多个事件

延时
小于一秒
trident(数秒)
数秒)

Thanks for the article!
Could you please explain this point in a bit more detail? "But, it relies on transactions to update state, which is slower and often has to be implemented by the user."
If I want to write my output to a persistent store e.g. redis, then why would it be slower in Storm than in Spark Streaming?

Reply

Replies
  1. Hi Josh, please check out the slide about Storm/Trident here: http://spark-summit.org/wp-content/uploads/2013/10/Spark-Summit-2013-Spark-Streaming.pdf
    If you want exactly-once semantics with Trident, you have to store a per-state transaction ID for each state. I.e., in word-count, for each word, you would store both the count as well as a transaction ID; each key-value pair would look like: (Key:word, Value: count, txid). Before updating the count, you would read in the old transaction ID to make sure it's up to date, and this read causes extra latency. If you are using redis in memory, that might be okay, but if it has to go to disk then that would add noticeable latency to the update. Whereas in Spark, you don't have to store a per-state transaction ID.
    For the details of Trident transactional processing, see http://storm.apache.org/documentation/Trident-state

  2. Hi Xinh, thanks for the explanation. I see, isn't that similar to Spark checkpointing - where it saves states to HDFS every ~10 seconds? or is your point that with Storm it would (by default) persist the state much more frequently than Spark?

  3. Hi Josh, yes, the fault tolerance in Spark involves periodic (~10 second) checkpointing of RDDs. Yes, my point is that with Storm Trident the persistence occurs when each batch is processed, and by default that occurs a lot more than once every 10 seconds. And, in tuning any of these parameters, there's a tradeoff in the frequency of persistence vs. recovery time in the case of failure.

容错
至少一次
trident:精确一次
精确一次
源出处
BackType and Twitter
UCB
实现语言
Clojure scala
API支持
java、python、ruby等
jscala、java、python

平台集成
NA(基于zookeeper)
spark(所以可以统一(或共用)时事处理与历史数据的处理)

产品、支持
Storm has been around for several years and has run in production at Twitter since 2011, as well as at many other companies
Meanwhile, Spark Streaming is a newer project; its only production deployment (that I am aware of) has been at Sharethrough since 2013.

计算理论框架   
Storm is the streaming solution in the Hortonworks Hadoop data platform
Spark Streaming is in both MapR's distribution and Cloudera's Enterprise data platformDatabricks

集群集成,部署方式
依赖zookeeper,standalone,messo
standalone,yarn,messo   

google trend   



bug燃烧图   

https://issues.apache.org/jira/browse/STORM/

https://issues.apache.org/jira/browse/SPARK/
可见spark问题解决比storm要及时得多









spark streaming 与 storm的对比的更多相关文章

  1. Spark Straming,Spark Streaming与Storm的对比分析

    Spark Straming,Spark Streaming与Storm的对比分析 一.大数据实时计算介绍 二.大数据实时计算原理 三.Spark Streaming简介 3.1 SparkStrea ...

  2. Spark Streaming与Storm的对比及使用场景

    Spark Streaming与Storm都可以做实时计算,那么在做技术选型的时候到底应该选择哪个呢?通过下图可以从计算模型.计算延迟.吞吐量.事物.容错性.动态并行度等方方面进行对比. 对比点    ...

  3. Spark Streaming与Storm的对比

  4. Apache 流框架 Flink,Spark Streaming,Storm对比分析(一)

    本文由  网易云发布. 1.Flink架构及特性分析 Flink是个相当早的项目,开始于2008年,但只在最近才得到注意.Flink是原生的流处理系统,提供high level的API.Flink也提 ...

  5. Apache 流框架 Flink,Spark Streaming,Storm对比分析(二)

    本文由  网易云发布. 本文内容接上一篇Apache 流框架 Flink,Spark Streaming,Storm对比分析(一) 2.Spark Streaming架构及特性分析 2.1 基本架构 ...

  6. Apache 流框架 Flink,Spark Streaming,Storm对比分析(2)

    此文已由作者岳猛授权网易云社区发布. 欢迎访问网易云社区,了解更多网易技术产品运营经验. 2.Spark Streaming架构及特性分析 2.1 基本架构 基于是spark core的spark s ...

  7. spark streaming与storm比较

  8. Apache 流框架 Flink,Spark Streaming,Storm对比分析(1)

    此文已由作者岳猛授权网易云社区发布. 欢迎访问网易云社区,了解更多网易技术产品运营经验. 1.Flink架构及特性分析 Flink是个相当早的项目,开始于2008年,但只在最近才得到注意.Flink是 ...

  9. spark streaming (一)

    实时计算介绍 Spark Streaming, 其实就是一种Spark提供的, 对于大数据, 进行实时计算的一种框架. 它的底层, 其实, 也是基于我们之前讲解的Spark Core的. 基本的计算模 ...

随机推荐

  1. 使用脚本启动fabric时出错

    Error: got unexpected status: BAD_REQUEST -- error authorizing update: error validating ReadSet: rea ...

  2. 错误处理:java.lang.NoClassDefFoundError: javax/jms/JMSContext

    原因是少包,需要在pom文件增加依赖 <dependency> <groupId>javax.jms</groupId> <artifactId>jav ...

  3. JavaJDBC【五、事务】

    概念: 事务(Transaction)作为单个逻辑工作单元执行的一系列操作. 这些操作都是作为一个整体一起向系统提交,要么都执行,要么都不执行. 特点: 原子性:一个完整操作. 一致性:当事务完成时, ...

  4. DHCP显示

    两种PXE启动芯片 开机显示:Inter® Boot Agent GE V1.2.45或者Intel UNDI PXE2.0 (Build 082):其中UNDI是Universal Network ...

  5. 第四章· MySQL客户端工具及SQL讲解

    一.客户端命令介绍 1.mysql 1.用于数据库的连接管理 1) 连接(略) 2) 管理: #MySQL接口自带的命令 \h 或 help 或? 查看帮助 \G 格式化查看数据(key:value) ...

  6. 十二,k8s集群访问控制之RBAC授权

    目录 角色访问控制RBAC (Role-Based Access Control) 常用的授权插件: RBAC控制: role 和 clusterrole rolebinding 和 clusterr ...

  7. C - Covered Points Count CodeForces - 1000C (差分,离散化,统计)

    C - Covered Points Count CodeForces - 1000C You are given nn segments on a coordinate line; each end ...

  8. Almost Regular Bracket Sequence CodeForces - 1095E (线段树,单点更新,区间查询维护括号序列)

    Almost Regular Bracket Sequence CodeForces - 1095E You are given a bracket sequence ss consisting of ...

  9. Linux命令行——scp命令

     原创声明:本文系博主原创文章,转载或引用请注明出处. scp 一般格式: scp [option] src dst 1. src和dst格式为: [user@]host:/path/to/file ...

  10. Systemd-journalctl日志管理

    Systemd 统一管理所有 Unit 的启动日志.带来的好处就是,可以只用journalctl一个命令,查看所有日志(内核日志和应用日志).日志的配置文件/etc/systemd/journald. ...