在worker中通过executor/mk-executor worker e, 创建每个executor

(defn mk-executor [worker executor-id]
(let [executor-data (mk-executor-data worker executor-id) ;;1.mk-executor-data
_ (log-message "Loading executor " (:component-id executor-data) ":" (pr-str executor-id))
task-datas (->> executor-data
:task-ids
(map (fn [t] [t (task/mk-task executor-data t)])) ;;2.mk-task
(into {})
(HashMap.))
_ (log-message "Loaded executor tasks " (:component-id executor-data) ":" (pr-str executor-id))
report-error-and-die (:report-error-and-die executor-data)
component-id (:component-id executor-data) ;;3.创建threads
        ;; starting the batch-transfer->worker ensures that anything publishing to that queue
;; doesn't block (because it's a single threaded queue and the caching/consumer started
;; trick isn't thread-safe)
system-threads [(start-batch-transfer->worker-handler! worker executor-data)]
handlers (with-error-reaction report-error-and-die
(mk-threads executor-data task-datas))
threads (concat handlers system-threads)]
    ;;使用schedule-recurring定期产生SYSTEM_TICK(触发spout pending rotate) 
    (setup-ticks! worker executor-data) 

1. mk-executor-data

(defn mk-executor-data [worker executor-id]
(let [worker-context (worker-context worker)
task-ids (executor-id->tasks executor-id) ;;包含的tasks
component-id (.getComponentId worker-context (first task-ids)) ;;所属于的component
storm-conf (normalized-component-conf (:storm-conf worker) worker-context component-id)
executor-type (executor-type worker-context component-id) ;;executor类型, blot或者spout
batch-transfer->worker (disruptor/disruptor-queue ;;executor的发送缓存queue
(storm-conf TOPOLOGY-EXECUTOR-SEND-BUFFER-SIZE)
:claim-strategy :single-threaded
:wait-strategy (storm-conf TOPOLOGY-DISRUPTOR-WAIT-STRATEGY))
]
(recursive-map
:worker worker
:worker-context worker-context
:executor-id executor-id
:task-ids task-ids
:component-id component-id
:open-or-prepare-was-called? (atom false)
:storm-conf storm-conf
:receive-queue ((:executor-receive-queue-map worker) executor-id) ;;取出executor所对应的disruptor queue
:storm-id (:storm-id worker)
:conf (:conf worker)
:shared-executor-data (HashMap.)
:storm-active-atom (:storm-active-atom worker)
:batch-transfer-queue batch-transfer->worker
:transfer-fn (mk-executor-transfer-fn batch-transfer->worker) ;;(1.1)
:suicide-fn (:suicide-fn worker)
:storm-cluster-state (cluster/mk-storm-cluster-state (:cluster-state worker))
:type executor-type
;; TODO: should refactor this to be part of the executor specific map (spout or bolt with :common field)
:stats (mk-executor-stats <> (sampling-rate storm-conf)) ;;(1.2)
:interval->task->metric-registry (HashMap.)
:task->component (:task->component worker)
:stream->component->grouper (outbound-components worker-context component-id)
:report-error (throttled-report-error-fn <>)
:report-error-and-die (fn [error] ;;将error写到zk的error目录下,其他daemon进程可以知道
((:report-error <>) error)
((:suicide-fn <>)))
:deserializer (KryoTupleDeserializer. storm-conf worker-context)
:sampler (mk-stats-sampler storm-conf) ;;1.3 mk-stats-sampler
;; TODO: add in the executor-specific stuff in a :specific... or make a spout-data, bolt-data function?
)))
 

1.1 mk-executor-transfer-fn

executor会把需要发送的tuple缓存到batch-transfer->worker queue中

参考下面的comments, 为了避免component block (大量的tuple没有被及时处理), 额外创建了overflow buffer, 只有当这个buffer也满了, 才停止nextTuple(对于spout executor比较需要overflow buffer)

        ;; the overflow buffer is used to ensure that spouts never block when emitting
;; this ensures that the spout can always clear the incoming buffer (acks and fails), which
;; prevents deadlock from occuring across the topology (e.g. Spout -> Bolt -> Acker -> Spout, and all
;; buffers filled up)
;; when the overflow buffer is full, spouts stop calling nextTuple until it's able to clear the overflow buffer
;; this limits the size of the overflow buffer to however many tuples a spout emits in one call of nextTuple,
;; preventing memory issues
overflow-buffer (LinkedList.)]

返回fn, fn用于将[task, tuple]放到overflow-buffer或者batch-transfer->worker queue中

注意, 这是executor->transfer-fn, 不同于worker->transfer-fn, 名字起的不好, 会混淆

executor的transfer-fn将tuple缓存到executor的batch-transfer->worker, 而worker->transfer-fn将tuple发送到worker的transfer queue

;; in its own function so that it can be mocked out by tracked topologies
(defn mk-executor-transfer-fn [batch-transfer->worker]
(fn this
([task tuple block? ^List overflow-buffer]
(if (and overflow-buffer (not (.isEmpty overflow-buffer))) ;;overflow存在并且不为空,说明queue已经满了,所以直接放overflow-buffer中
(.add overflow-buffer [task tuple])
(try-cause
(disruptor/publish batch-transfer->worker [task tuple] block?)
(catch InsufficientCapacityException e
(if overflow-buffer
(.add overflow-buffer [task tuple])
(throw e))
))))
([task tuple overflow-buffer]
(this task tuple (nil? overflow-buffer) overflow-buffer))
([task tuple]
(this task tuple nil)
)))

1.2 mk-executor-stats <> (sampling-rate storm-conf)

Storm-源码分析-Stats (backtype.storm.stats)

 

1.3 mk-stats-sampler

根据conf里面的sampling-rate创建一个sampler

(defn mk-stats-sampler [conf]
(even-sampler (sampling-rate conf)))

这里创建的是even-sampler,

(defn even-sampler [freq]
(let [freq (int freq)
start (int 0)
r (java.util.Random.)
curr (MutableInt. -1)
target (MutableInt. (.nextInt r freq))] ;;[0,freq]中的随机值
(with-meta
(fn []
(let [i (.increment curr)]
(when (>= i freq)
(.set curr start)
(.set target (.nextInt r freq))))
(= (.get curr) (.get target))) ;;FP没有直接赋值, 所以==简化为=
{:rate freq})))
(defn sampler-rate [sampler]
(:rate (meta sampler)))

even-sampler, 返回的是个fn ,并且通过with-meta添加metadata({:rate freq})

所以, 通过(:rate (meta sampler)), 可以从sampler的meta里面取出rate值

sampler就是fn, 每次调用都会返回(= curr target)

curr从start开始递增, 在达到target之前, 调用fn都是返回false

当curr等于target时, 调用fn返回true

当curr大于target时, 从新随机生成target, 将curr清零

所以sampler实际产生的效果, 就是不停的调用sampler, 会随机出现若干次false和一次true (在freq的范围内)

从而达到sampler的效果, 只有是true的时候才取样

其实对于简单的sampler, 比如rate是20%, 可以简单的每跳过4个取一个, 但是这样可能的问题是, 取样的规律性太强, 如果数据恰好符合你的规律, 比如5倍数的数据相同, 就会有问题

所以这里为了增加随机性, 采用这样的实现

并且这里对闭包和metadata的应用, 值得借鉴

 

2.mk-task, 创建task

(task/mk-task executor-data t)

Storm-源码分析-Topology Submit-Task

 

3.创建threads

3.1 batch-transfer-queue handle thread, spout发送线程

从batch-transfer-queue取出messages, 没有到达batchend时, 放到cached-emit中的arraylist中

当达到batchend时, 使用transfer-fn将messages发送到transfer-queue (spout应该没有发送给自己的tuple吧)

(defn start-batch-transfer->worker-handler! [worker executor-data]
(let [worker-transfer-fn (:transfer-fn worker)
cached-emit (MutableObject. (ArrayList.)) ;;用于cache所有messages,直到batchend
storm-conf (:storm-conf executor-data)
serializer (KryoTupleSerializer. storm-conf (:worker-context executor-data))
]
(disruptor/consume-loop*
(:batch-transfer-queue executor-data)
(disruptor/handler [o seq-id batch-end?]
(let [^ArrayList alist (.getObject cached-emit)]
(.add alist o)
(when batch-end?
(worker-transfer-fn serializer alist)
(.setObject cached-emit (ArrayList.))
)))
:kill-fn (:report-error-and-die executor-data))))

 

Worker, transfer-fn

将task分为local和remote

对于local的, 使用local-transfer将messages发送到对应的recieve-queue里面

而对于remote的, 使用disruptor/publish发送到transfer-queue里面

storm使用kryo作为其java的序列化F/W (http://code.google.com/p/kryo/)

(defn mk-transfer-fn [worker]
(let [local-tasks (-> worker :task-ids set)
local-transfer (:transfer-local-fn worker)
^DisruptorQueue transfer-queue (:transfer-queue worker)]
(fn [^KryoTupleSerializer serializer tuple-batch]
(let [local (ArrayList.)
remote (ArrayList.)]
(fast-list-iter [[task tuple :as pair] tuple-batch]
(if (local-tasks task)
(.add local pair)
(.add remote pair)
))
(local-transfer local)
;; not using map because the lazy seq shows up in perf profiles
(let [serialized-pairs (fast-list-for [[task ^TupleImpl tuple] remote] [task (.serialize serializer tuple)])]
(disruptor/publish transfer-queue serialized-pairs)

 

3.2 executor的执行thread

try…catch mk-threads函数, 如果发生异常将error写到zk, 以便其他的daemon能及时知道

handlers (with-error-reaction report-error-and-die

                   (mk-threads executor-data task-datas))

Storm-源码分析-Topology Submit-Executor-mk-threads

Storm-源码分析-Topology Submit-Executor的更多相关文章

  1. Storm源码分析--Nimbus-data

    nimbus-datastorm-core/backtype/storm/nimbus.clj (defn nimbus-data [conf inimbus] (let [forced-schedu ...

  2. JStorm与Storm源码分析(四)--均衡调度器,EvenScheduler

    EvenScheduler同DefaultScheduler一样,同样实现了IScheduler接口, 由下面代码可以看出: (ns backtype.storm.scheduler.EvenSche ...

  3. JStorm与Storm源码分析(三)--Scheduler,调度器

    Scheduler作为Storm的调度器,负责为Topology分配可用资源. Storm提供了IScheduler接口,用户可以通过实现该接口来自定义Scheduler. 其定义如下: public ...

  4. JStorm与Storm源码分析(二)--任务分配,assignment

    mk-assignments主要功能就是产生Executor与节点+端口的对应关系,将Executor分配到某个节点的某个端口上,以及进行相应的调度处理.代码注释如下: ;;参数nimbus为nimb ...

  5. storm源码分析之任务分配--task assignment

    在"storm源码分析之topology提交过程"一文最后,submitTopologyWithOpts函数调用了mk-assignments函数.该函数的主要功能就是进行topo ...

  6. JStorm与Storm源码分析(一)--nimbus-data

    Nimbus里定义了一些共享数据结构,比如nimbus-data. nimbus-data结构里定义了很多公用的数据,请看下面代码: (defn nimbus-data [conf inimbus] ...

  7. storm源码分析之topology提交过程

    storm集群上运行的是一个个topology,一个topology是spouts和bolts组成的图.当我们开发完topology程序后将其打成jar包,然后在shell中执行storm jar x ...

  8. Nimbus<三>Storm源码分析--Nimbus启动过程

    Nimbus server, 首先从启动命令开始, 同样是使用storm命令"storm nimbus”来启动看下源码, 此处和上面client不同, jvmtype="-serv ...

  9. JStorm与Storm源码分析(五)--SpoutOutputCollector与代理模式

    本文主要是解析SpoutOutputCollector源码,顺便分析该类中所涉及的设计模式–代理模式. 首先介绍一下Spout输出收集器接口–ISpoutOutputCollector,该接口主要声明 ...

  10. Mesos源码分析(15): Test Executor的运行

    Test Executor的代码在src/examples/test_executor.cpp中   int main(int argc, char** argv) {   TestExecutor ...

随机推荐

  1. Apollo配置中心转

    尊重原创,本文转自:https://www.cnblogs.com/FlyAway2013/p/8811385.html 前我们项目,所有的配置基本都是通过本地properties 文件进行配置的,比 ...

  2. Jquery js框架使用

    jquery  众所周知 ,强大的 js框架 自己使用的一些笔记 //1.json格式定义方法 var product_obj={    check_init:function(){          ...

  3. PHP学习笔记(13)班级和学生管理---班级

    两个文件夹,一个班级cls,一个学生stu. 两个表,一个班级cls,一个学生stu. 每个文件夹里有7个php文件:主界面cls.php-------增add.php,insert.php----- ...

  4. 【Spring】java.lang.IndexOutOfBoundsException: Index: 256, Size: 256

    Spring接受前台的数据超过256出现例如以下异常: org.springframework.beans.InvalidPropertyException: Invalid property 'sp ...

  5. git patch生成方法

    先把改动commit掉,然后生产改动patch给提交代码的同事,详细操作过程例如以下: 改动代码的同事: git format-patch al821_xxx origin/al821_xxx 会生成 ...

  6. I帧、B帧、P帧、NALU类型

    i帧 i frame,即内部画面 intra picture,通常是GOP的第一个帧(即IDR)I帧是最大去除图像空间冗余信息而压缩得到的帧,自带全部信息,不参考其他帧可独立解码,称为帧内编码帧所有视 ...

  7. ArcGIS教程:“流向”的工作原理

    获取表面的水文特征的关键之中的一个是可以确定从栅格中的每一个像元流出的方向.这可通过流向工具来完毕. 该工具把表面作为输入,然后输出一个显示从每一个像元流出方向的栅格. 假设选择了输出下降率栅格数据选 ...

  8. 微信小程序动态显示项目倒计时效果

    效果: wxml代码: <view class='spellNum'> <view> <text style='color: #fff;'>团长</text& ...

  9. JDK7 Garbage Frist

    JDK7 G1新型垃圾回收器. http://www.infoq.com/cn/articles/jdk7-garbage-first-collector

  10. 'Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set.

    UserWarning: Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. Defaulting SQLALCHEMY_DATA ...