2.1 Storm基本概念

在运行一个Storm任务之前，需要了解一些概念：

  1. Topologies ：[tə'pɑ:lədʒɪ]拓扑结构
  2. Streams
  3. Spouts：[spaʊt]喷出; 喷射; 滔滔不绝地讲; 把…典当掉;
  4. Bolts:[bəʊlt]筛选; 囫囵吞下; （把门、窗等） 闩上; 突然说出，脱口说出;
  5. Stream groupings
  6. Reliability
  7. Tasks
  8. Workers
  9. Configuration

  10. supervisor:英[ˈsu:pəvaɪzə(r)]监督者，管理者; 镇长;

  11. nimbus：英[ˈnɪmbəs]美[ˈnɪmbəs]（人和物产生的） 灵气，（神像头上的）光轮;

Storm集群和Hadoop集群表面上看很类似。但是Hadoop上运行的是MapReduce jobs，而在Storm上运行的是拓扑（topology），这两者之间是非常不一样的。一个关键的区别是： 一个MapReduce job最终会结束， 而一个topology永远会运行（除非你手动kill掉）。

在Storm的集群里面有两种节点： 控制节点（master node）和工作节点（worker node）。控制节点上面运行一个叫Nimbus(nimbus[英][ˈnɪmbəs][美][ˈnɪmbəs]

n.（人和物产生的）灵气，（神像头上的）光轮; )后台程序，它的作用类似Hadoop里面的JobTracker。Nimbus负责在集群里面分发代码，分配计算任务给机器， 并且监控状态。

每一个工作节点上面运行一个叫做Supervisor的节点。Supervisor会监听分配给它那台机器的工作，根据需要启动/关闭工作进程。每一个工作进程执行一个topology的一个子集；一个运行的topology由运行在很多机器上的很多工作进程组成。

Nimbus和Supervisor之间的所有协调工作都是通过Zookeeper集群完成。另外，Nimbus进程和Supervisor进程都是快速失败（fail-fast)和无状态的。所有的状态要么在zookeeper里面， 要么在本地磁盘上。这也就意味着你可以用kill -9来杀死Nimbus和Supervisor进程， 然后再重启它们，就好像什么都没有发生过。这个设计使得Storm异常的稳定。

2.1.1 Topologies

一个topology是spouts和bolts组成的图， 通过stream groupings将图中的spouts和bolts连接起来，如下图：

一个topology会一直运行直到你手动kill掉，Storm自动重新分配执行失败的任务， 并且Storm可以保证你不会有数据丢失（如果开启了高可靠性的话）。如果一些机器意外停机它上面的所有任务会被转移到其他机器上。

运行一个topology很简单。首先，把你所有的代码以及所依赖的jar打进一个jar包。然后运行类似下面的这个命令：

storm jar all-my-code.jar backtype.storm.MyTopology arg1 arg2

这个命令会运行主类: backtype.strom.MyTopology, 参数是arg1, arg2。这个类的main函数定义这个topology并且把它提交给Nimbus。storm jar负责连接到Nimbus并且上传jar包。

Topology的定义是一个Thrift结构，并且Nimbus就是一个Thrift服务， 你可以提交由任何语言创建的topology。上面的方面是用JVM-based语言提交的最简单的方法。

2.1.2 Streams

消息流stream是storm里的关键抽象。一个消息流是一个没有边界的tuple序列， 而这些tuple序列会以一种分布式的方式并行地创建和处理。通过对stream中tuple序列中每个字段命名来定义stream。在默认的情况下，tuple的字段类型可以是：integer，long，short， byte，string，double，float，boolean和byte array。你也可以自定义类型（只要实现相应的序列化器）。

每个消息流在定义的时候会被分配给一个id，因为单向消息流使用的相当普遍， OutputFieldsDeclarer定义了一些方法让你可以定义一个stream而不用指定这个id。在这种情况下这个stream会分配个值为‘default’默认的id 。

Storm提供的最基本的处理stream的原语是spout和bolt。你可以实现spout和bolt提供的接口来处理你的业务逻辑。

2.1.3  Spouts

消息源spout是Storm里面一个topology里面的消息生产者。一般来说消息源会从一个外部源读取数据并且向topology里面发出消息：tuple。Spout可以是可靠的也可以是不可靠的。如果这个tuple没有被storm成功处理，可靠的消息源spouts可以重新发射一个tuple， 但是不可靠的消息源spouts一旦发出一个tuple就不能重发了。

消息源可以发射多条消息流stream。使用OutputFieldsDeclarer.declareStream来定义多个stream，然后使用SpoutOutputCollector来发射指定的stream。

Spout类里面最重要的方法是nextTuple。要么发射一个新的tuple到topology里面或者简单的返回如果已经没有新的tuple。要注意的是nextTuple方法不能阻塞，因为storm在同一个线程上面调用所有消息源spout的方法。

另外两个比较重要的spout方法是ack和fail。storm在检测到一个tuple被整个topology成功处理的时候调用ack，否则调用fail。storm只对可靠的spout调用ack和fail。

2.1.4  Bolts

所有的消息处理逻辑被封装在bolts里面。Bolts可以做很多事情：过滤，聚合，查询数据库等等。

Bolts可以简单的做消息流的传递。复杂的消息流处理往往需要很多步骤，从而也就需要经过很多bolts。比如算出一堆图片里面被转发最多的图片就至少需要两步：第一步算出每个图片的转发数量。第二步找出转发最多的前10个图片。（如果要把这个过程做得更具有扩展性那么可能需要更多的步骤）。

Bolts可以发射多条消息流， 使用OutputFieldsDeclarer.declareStream定义stream，使用OutputCollector.emit来选择要发射的stream。

Bolts的主要方法是execute, 它以一个tuple作为输入，bolts使用OutputCollector来发射tuple，bolts必须要为它处理的每一个tuple调用OutputCollector的ack方法，以通知Storm这个tuple被处理完成了，从而通知这个tuple的发射者spouts。 一般的流程是： bolts处理一个输入tuple,  发射0个或者多个tuple, 然后调用ack通知storm自己已经处理过这个tuple了。storm提供了一个IBasicBolt会自动调用ack。

2.1.5  Stream groupings

定义一个topology的其中一步是定义每个bolt接收什么样的流作为输入。stream grouping就是用来定义一个stream应该如果分配数据给bolts上面的多个tasks。

Storm里面有7种类型的stream grouping（7种stream grouping策略）

  1. Shuffle Grouping: 随机分组， 随机派发stream里面的tuple，保证每个bolt接收到的tuple数目大致相同。
  2. Fields Grouping：按字段分组， 比如按userid来分组， 具有同样userid的tuple会被分到相同的Bolts里的一个task， 而不同的userid则会被分配到不同的bolts里的task。
  3. All Grouping：广播发送，对于每一个tuple，所有的bolts都会收到。
  4. Global Grouping：全局分组， 这个tuple被分配到storm中的一个bolt的其中一个task。再具体一点就是分配给id值最低的那个task。
  5. Non Grouping：不分组，这个分组的意思是说stream不关心到底谁会收到它的tuple。目前这种分组和Shuffle grouping是一样的效果， 有一点不同的是storm会把这个bolt放到这个bolt的订阅者同一个线程里面去执行。
  6. Direct Grouping： 直接分组， 这是一种比较特别的分组方法，用这种分组意味着消息的发送者指定由消息接收者的哪个task处理这个消息。 只有被声明为Direct Stream的消息流可以声明这种分组方法。而且这种消息tuple必须使用emitDirect方法来发射。消息处理者可以通过TopologyContext来获取处理它的消息的task的id （OutputCollector.emit方法也会返回task的id）。
  7. Local or shuffle grouping：如果目标bolt有一个或者多个task在同一个工作进程中，tuple将会被随机发生给这些tasks。否则，和普通的Shuffle Grouping行为一致。

2.1.6  Reliability

Storm保证每个tuple会被topology完整的执行。Storm会追踪由每个spout tuple所产生的tuple树（一个bolt处理一个tuple之后可能会发射别的tuple从而形成树状结构），并且跟踪这棵tuple树什么时候成功处理完。每个topology都有一个消息超时的设置，如果storm在这个超时的时间内检测不到某个tuple树到底有没有执行成功， 那么topology会把这个tuple标记为执行失败，并且过一会儿重新发射这个tuple。

为了利用Storm的可靠性特性，在你发出一个新的tuple以及你完成处理一个tuple的时候你必须要通知storm。这一切是由OutputCollector来完成的。通过emit方法来通知一个新的tuple产生了，通过ack方法通知一个tuple处理完成了。

Storm的可靠性我们在第四章会深入介绍。

2.1.7  Tasks

每一个spout和bolt会被当作很多task在整个集群里执行。每一个executor对应到一个线程，在这个线程上运行多个task，而stream grouping则是定义怎么从一堆task发射tuple到另外一堆task。你可以调用TopologyBuilder类的setSpout和setBolt来设置并行度（也就是有多少个task）。

2.1.8  Workers

一个topology可能会在一个或者多个worker（工作进程）里面执行，每个worker是一个物理JVM并且执行整个topology的一部分。比如，对于并行度是300的topology来说，如果我们使用50个工作进程来执行，那么每个工作进程会处理其中的6个tasks。Storm会尽量均匀的工作分配给所有的worker。

2.1.9 Configuration

Storm里面有一堆参数可以配置来调整Nimbus, Supervisor(supervisor[英][ˈsu:pəvaɪzə(r)][美][ˈsu:pərvaɪzə(r)]n.监督者，管理者; 镇长; )以及正在运行的topology的行为，一些配置是系统级别的，一些配置是topology级别的。default.yaml里面有所有的默认配置。你可以通过定义个storm.yaml在你的classpath里来覆盖这些默认配置。并且你也可以在代码里面设置一些topology相关的配置信息（使用StormSubmitter）。

2.2  构建Topology

1. 实现的目标：

我们将设计一个topology，来实现对一个句子里面的单词出现的频率进行统计。这是一个简单的例子，目的是让大家对于topology快速上手，有一个初步的理解。

2. 设计Topology结构：

在开始开发Storm项目的第一步，就是要设计topology。确定好你的数据处理逻辑，我们今天将的这个简单的例子，topology也非常简单。整个topology如下：

整个topology分为三个部分：

Input(Spout):数据源，负责发送sentence

Splitsentence(Bolt):负责将sentence切分

Wordcount(Bolt):负责对单词的频率进行累加

3. 设计数据流

这个topology从Input读取句子,并把句子划分成单词,然后汇总每个单词出现的次数,一个tuple负责读取句子,每一个tuple分别对应计算每一个单词出现的次数,大概样子如下所示：

4. 代码实现：

maven的pom.xml

  1. <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  2.   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  3.   <modelVersion>4.0.0</modelVersion>
  4.  
  5.   <groupId>study</groupId>
  6.   <artifactId>dxz</artifactId>
  7.   <version>0.0.1-SNAPSHOT</version>
  8.   <packaging>jar</packaging>
  9.  
  10.   <name>dxz</name>
  11.   <url>http://maven.apache.org</url>
  12.  
  13.   <properties>
  14.     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  15.   </properties>
  16.  
  17.   <dependencies>
  18.     <dependency>
  19.       <groupId>junit</groupId>
  20.       <artifactId>junit</artifactId>
  21.       <version>3.8.1</version>
  22.       <scope>test</scope>
  23.     </dependency>
  24.     <dependency>
  25.       <groupId>org.apache.storm</groupId>
  26.       <artifactId>storm-core</artifactId>
  27.       <version>0.9.3</version>
  28.       <scope>provided</scope>
  29.     </dependency>
  30.   </dependencies>
  31. </project>

java代码：

  1. package com.stormdemo.bolt;
  2.  
  3. import java.util.HashMap;
  4. import java.util.Map;
  5. import java.util.Random;
  6.  
  7. import backtype.storm.Config;
  8. import backtype.storm.LocalCluster;
  9. import backtype.storm.StormSubmitter;
  10. import backtype.storm.generated.AlreadyAliveException;
  11. import backtype.storm.generated.InvalidTopologyException;
  12. import backtype.storm.spout.SpoutOutputCollector;
  13. import backtype.storm.task.OutputCollector;
  14. import backtype.storm.task.TopologyContext;
  15. import backtype.storm.topology.IRichBolt;
  16. import backtype.storm.topology.IRichSpout;
  17. import backtype.storm.topology.OutputFieldsDeclarer;
  18. import backtype.storm.topology.TopologyBuilder;
  19. import backtype.storm.tuple.Fields;
  20. import backtype.storm.tuple.Tuple;
  21. import backtype.storm.tuple.Values;
  22. import backtype.storm.utils.Utils;
  23.  
  24. public class WordCount {
  25.  
  26.     public static class WordProduct implements IRichSpout {
  27.         private static final long serialVersionUID = 1L;
  28.         private SpoutOutputCollector collector;
  29.         Random rand;
  30.  
  31.         /**
  32.          * 这是第一个方法要执行的方法（只执行一次），常用于初始化的（如打开文件、打开连接等）
  33.          * 里面接收了三个参数，第一个是创建Topology时的配置，
  34.          * 第二个是所有的Topology数据，第三个是用来把Spout的数据发射给bolt
  35.          */
  36.         @Override
  37.         public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
  38.             System.out.println("in open()*****************");
  39.             this.collector = collector;
  40.             rand = new Random();
  41.         }
  42.  
  43.         public void close() {
  44.             // TODO Auto-generated method stub
  45.  
  46.         }
  47.  
  48.         public void activate() {
  49.             // TODO Auto-generated method stub
  50.  
  51.         }
  52.  
  53.         public void deactivate() {
  54.             // TODO Auto-generated method stub
  55.  
  56.         }
  57.  
  58.         /**
  59.          * 这个方法会不断被调用，为了降低它对CPU的消耗，当任务完成时让它sleep一下
  60.          */
  61.         @Override
  62.         public void nextTuple() {
  63.             Utils.sleep(10000);
  64.             String[] sentences = new String[] {
  65.                     "this is a test",
  66.                     "hello world",
  67.                     "I am fine",
  68.                     "china duan"
  69.             };
  70.             String s = sentences[rand.nextInt(sentences.length)];
  71.  
  72.             System.out.println("in nextTuple():" + s);
  73.             collector.emit(new Values(s));
  74.         }
  75.  
  76.         @Override
  77.         public void ack(Object msgId) {
  78.             System.out.println("OK:" + msgId);
  79.         } 
  80.  
  81.         @Override
  82.         public void fail(Object msgId) {
  83.             System.out.println("FAIL:" + msgId);
  84.         }  
  85.  
  86.         public void declareOutputFields(OutputFieldsDeclarer declarer) {
  87.             declarer.declare(new Fields("sentence"));
  88.         }
  89.  
  90.         public Map<String, Object> getComponentConfiguration() {
  91.             // TODO Auto-generated method stub
  92.             return null;
  93.         }
  94.  
  95.     }
  96.     public static class SplitSentence implements IRichBolt {
  97.         private OutputCollector collector;
  98.  
  99.         /**
  100.          * Topology执行完毕的清理工作，比如关闭连接、释放资源等操作都会写在这里
  101.          * 因为这只是个Demo，我们用它来打印我们的计数器
  102.          */
  103.         @Override
  104.         public void cleanup() {
  105.             System.out.println("cleanup()");
  106.         }
  107.  
  108.         /**
  109.          * 这是bolt中最重要的方法，每当接收到一个tuple时，此方法便被调用
  110.          * 这个方法的作用就是把文本文件中的每一行切分成一个个单词，并把这些单词发射出去（给下一个bolt处理）
  111.          */
  112.         @Override
  113.         public void execute(Tuple input) {
  114.             String s = input.getString(0);
  115.             String[] words = s.split(" ");
  116.             for(String w : words) {
  117.                 w = w.trim();
  118.                 if(!w.isEmpty()) {
  119.                     collector.emit(new Values(w));
  120.                 }
  121.             }
  122.             //确认成功处理一个tuple
  123.             collector.ack(input);
  124.         }
  125.  
  126.         public void prepare(Map arg0, TopologyContext arg1, OutputCollector arg2) {
  127.             this.collector = arg2;
  128.         }
  129.  
  130.         public void declareOutputFields(OutputFieldsDeclarer declarer) {
  131.             declarer.declare(new Fields("word"));
  132.  
  133.         }
  134.  
  135.         public Map<String, Object> getComponentConfiguration() {
  136.             // TODO Auto-generated method stub
  137.             return null;
  138.         }
  139.  
  140.     }
  141.     public static class WordCounter implements IRichBolt {
  142.         private OutputCollector collector;
  143.         Map<String,Integer> counts = new HashMap<String, Integer>();
  144.  
  145.         /**
  146.          * Topology执行完毕的清理工作，比如关闭连接、释放资源等操作都会写在这里
  147.          * 因为这只是个Demo，我们用它来打印我们的计数器
  148.          */
  149.         @Override
  150.         public void cleanup() {
  151.             System.out.println("cleanup()");
  152.         }
  153.  
  154.         /**
  155.          * 这是bolt中最重要的方法，每当接收到一个tuple时，此方法便被调用
  156.          * 这个方法的作用就是统计单词次数
  157.          */
  158.         @Override
  159.         public void execute(Tuple input) {
  160.             String w = input.getString(0);
  161.             Integer count = counts.get(w);
  162.             if(count==null) {
  163.                 count = 1;
  164.             } else {
  165.                 count++;
  166.             }
  167.             counts.put(w, count);
  168.             collector.emit(new Values("word",count));
  169.  
  170.             System.out.println(w + ":" + Integer.toString(count));
  171.             //确认成功处理一个tuple
  172.             collector.ack(input);
  173.         }
  174.  
  175.         public void prepare(Map arg0, TopologyContext arg1, OutputCollector arg2) {
  176.             this.collector = arg2;
  177.         }
  178.  
  179.         public void declareOutputFields(OutputFieldsDeclarer declarer) {
  180.             declarer.declare(new Fields("word","count"));
  181.         }
  182.  
  183.         public Map<String, Object> getComponentConfiguration() {
  184.             // TODO Auto-generated method stub
  185.             return null;
  186.         }
  187.  
  188.     }
  189.  
  190.     public static void main(String[] args) throws AlreadyAliveException, InvalidTopologyException {
  191.         //定义一个Topology
  192.         TopologyBuilder builder = new TopologyBuilder();
  193.  
  194.         //为topology设置spout(节点)
  195.         builder.setSpout("input", new WordProduct(),1);
  196.  
  197.         //为topology设置bolt(节点)
  198.         builder.setBolt("bolt_sentence", new SplitSentence(),1).shuffleGrouping("input");
  199.         builder.setBolt("bolt_wordcounter", new WordCounter(),1).fieldsGrouping("bolt_sentence", new Fields("word"));
  200.  
  201.         //配置
  202.         Config config = new Config();
  203.         //TOPOLOGY_DEBUG(setDebug), 当它被设置成true的话， storm会记录下每个组件所发射的每条消息。这在本地环境调试topology很有用， 但是在线上这么做的话会影响性能的。
  204.         config.setDebug(true);
  205.         //定义你希望集群分配多少个工作进程给你来执行这个topology.
  206.         config.setNumWorkers(2); 
  207.  
  208.         if(args!=null && args.length>0) {
  209.             config.setNumWorkers(3);
  210.             StormSubmitter.submitTopology(args[0], config, builder.createTopology());
  211.         } else {
  212.             //创建一个本地模式cluster
  213.             LocalCluster cluster = new LocalCluster();
  214.             System.out.println("start word count");
  215.             cluster.submitTopology("word count", config, builder.createTopology());
  216.             //运行60s
  217.             Utils.sleep(60000);
  218.  
  219.             System.out.println("begin kill topology");
  220.             cluster.killTopology("word count"); //终止topology任务
  221.             cluster.shutdown();
  222.         }
  223.     }
  224.  
  225. }

结果：

  1. 5693 [main-EventThread] INFO  org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED
  2. 5701 [main] INFO  backtype.storm.daemon.supervisor - Starting supervisor with id 1a1baffd-69d0-42f9-89de-35a60496fbd1 at host SF0001107252A.sf.com
  3. start word count
  4. 5768 [main] INFO  backtype.storm.daemon.nimbus - Received topology submission for word count with conf {"topology.max.task.parallelism" nil, "topology.acker.executors" nil, "topology.kryo.register" nil, "topology.kryo.decorators" (), "topology.name" "word count", "storm.id" "word count-1-1484476433", "topology.workers" 2, "topology.debug" true}
  5. 5805 [main] INFO  backtype.storm.daemon.nimbus - Activating word count: word count-1-1484476433
  6. 5927 [main] INFO  backtype.storm.scheduler.EvenScheduler - Available slots: (["1a1baffd-69d0-42f9-89de-35a60496fbd1" 1027] ["1a1baffd-69d0-42f9-89de-35a60496fbd1" 1028] ["1a1baffd-69d0-42f9-89de-35a60496fbd1" 1029] ["1989826c-4520-4bbc-bb34-e09f5155ce91" 1024] ["1989826c-4520-4bbc-bb34-e09f5155ce91" 1025] ["1989826c-4520-4bbc-bb34-e09f5155ce91" 1026])
  7. 5951 [main] INFO  backtype.storm.daemon.nimbus - Setting new assignment for topology id word count-1-1484476433: #backtype.storm.daemon.common.Assignment{:master-code-dir "C:\\Users\\01107252\\AppData\\Local\\Temp\\12d6d452-2f70-4f4f-8451-e21c36371261\\nimbus\\stormdist\\word count-1-1484476433", :node->host {"1989826c-4520-4bbc-bb34-e09f5155ce91" "SF0001107252A.sf.com", "1a1baffd-69d0-42f9-89de-35a60496fbd1" "SF0001107252A.sf.com"}, :executor->node+port {[4 4] ["1989826c-4520-4bbc-bb34-e09f5155ce91" 1024], [3 3] ["1a1baffd-69d0-42f9-89de-35a60496fbd1" 1027], [2 2] ["1989826c-4520-4bbc-bb34-e09f5155ce91" 1024], [1 1] ["1a1baffd-69d0-42f9-89de-35a60496fbd1" 1027], [5 5] ["1a1baffd-69d0-42f9-89de-35a60496fbd1" 1027]}, :executor->start-time-secs {[5 5] 1484476433, [3 3] 1484476433, [1 1] 1484476433, [4 4] 1484476433, [2 2] 1484476433}}
  8. 6637 [Thread-3] INFO  backtype.storm.daemon.supervisor - Downloading code for storm id word count-1-1484476433 from C:\Users\01107252\AppData\Local\Temp\12d6d452-2f70-4f4f-8451-e21c36371261\nimbus\stormdist\word count-1-1484476433
  9. 6749 [Thread-5] INFO  backtype.storm.daemon.supervisor - Downloading code for storm id word count-1-1484476433 from C:\Users\01107252\AppData\Local\Temp\12d6d452-2f70-4f4f-8451-e21c36371261\nimbus\stormdist\word count-1-1484476433
  10. 7255 [Thread-5] INFO  backtype.storm.daemon.supervisor - Finished downloading code for storm id word count-1-1484476433 from C:\Users\01107252\AppData\Local\Temp\12d6d452-2f70-4f4f-8451-e21c36371261\nimbus\stormdist\word count-1-1484476433
  11. 7313 [Thread-3] INFO  backtype.storm.daemon.supervisor - Finished downloading code for storm id word count-1-1484476433 from C:\Users\01107252\AppData\Local\Temp\12d6d452-2f70-4f4f-8451-e21c36371261\nimbus\stormdist\word count-1-1484476433
  12. 7317 [Thread-6] INFO  backtype.storm.daemon.supervisor - Launching worker with assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id "word count-1-1484476433", :executors ([3 3] [1 1] [5 5])} for this supervisor 1a1baffd-69d0-42f9-89de-35a60496fbd1 on port 1027 with id 7c37d53d-735c-43ee-8b1f-0ea0bf3ab564
  13. 7321 [Thread-6] INFO  backtype.storm.daemon.worker - Launching worker for word count-1-1484476433 on 1a1baffd-69d0-42f9-89de-35a60496fbd1:1027 with id 7c37d53d-735c-43ee-8b1f-0ea0bf3ab564 and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\01107252\\AppData\\Local\\Temp\\ad8f635a-51d8-41a8-bd21-722220739f15", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1027 1028 1029), "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}
  14. 7322 [Thread-6] INFO  backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
  15. 7322 [Thread-6] INFO  org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting
  16. 7323 [Thread-6] INFO  org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2000 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@51480654
  17. 7327 [Thread-6-SendThread(0:0:0:0:0:0:0:1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2000. Will not attempt to authenticate using SASL (unknown error)
  18. 7329 [Thread-6-SendThread(0:0:0:0:0:0:0:1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Socket connection established to 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2000, initiating session
  19. 7330 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /0:0:0:0:0:0:0:1:51109
  20. 7330 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /0:0:0:0:0:0:0:1:51109
  21. 7332 [SyncThread:0] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x159a1b0b919000a with negotiated timeout 20000 for client /0:0:0:0:0:0:0:1:51109
  22. 7332 [Thread-6-SendThread(0:0:0:0:0:0:0:1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2000, sessionid = 0x159a1b0b919000a, negotiated timeout = 20000
  23. 7333 [Thread-6-EventThread] INFO  org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED
  24. 7333 [Thread-6-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
  25. 7335 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b919000a
  26. 7337 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /0:0:0:0:0:0:0:1:51109 which had sessionid 0x159a1b0b919000a
  27. 7337 [Thread-6] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b919000a closed
  28. 7337 [Thread-6-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
  29. 7338 [Thread-6] INFO  backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
  30. 7339 [Thread-6] INFO  org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting
  31. 7339 [Thread-6] INFO  org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@14f9cec1
  32. 7341 [Thread-6-SendThread(127.0.0.1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
  33. 7343 [Thread-6-SendThread(127.0.0.1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2000, initiating session
  34. 7343 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:51112
  35. 7344 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:51112
  36. 7346 [SyncThread:0] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x159a1b0b919000b with negotiated timeout 20000 for client /127.0.0.1:51112
  37. 7347 [Thread-6-SendThread(127.0.0.1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2000, sessionid = 0x159a1b0b919000b, negotiated timeout = 20000
  38. 7347 [Thread-6-EventThread] INFO  org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED
  39. 7374 [Thread-4] INFO  backtype.storm.daemon.supervisor - Launching worker with assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id "word count-1-1484476433", :executors ([4 4] [2 2])} for this supervisor 1989826c-4520-4bbc-bb34-e09f5155ce91 on port 1024 with id f5fc123f-fb15-4515-a3c5-d658e983f9e3
  40. 7375 [Thread-4] INFO  backtype.storm.daemon.worker - Launching worker for word count-1-1484476433 on 1989826c-4520-4bbc-bb34-e09f5155ce91:1024 with id f5fc123f-fb15-4515-a3c5-d658e983f9e3 and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\01107252\\AppData\\Local\\Temp\\4a0fd3f4-bc1c-41b0-8319-2271783a8e8e", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1024 1025 1026), "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}
  41. 7375 [Thread-4] INFO  backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
  42. 7376 [Thread-4] INFO  org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting
  43. 7376 [Thread-4] INFO  org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2000 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@65667fc0
  44. 7380 [Thread-4-SendThread(127.0.0.1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
  45. 7381 [Thread-4-SendThread(127.0.0.1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2000, initiating session
  46. 7381 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:51115
  47. 7381 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:51115
  48. 7383 [SyncThread:0] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x159a1b0b919000c with negotiated timeout 20000 for client /127.0.0.1:51115
  49. 7384 [Thread-4-SendThread(127.0.0.1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2000, sessionid = 0x159a1b0b919000c, negotiated timeout = 20000
  50. 7384 [Thread-4-EventThread] INFO  org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED
  51. 7384 [Thread-4-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
  52. 7386 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b919000c
  53. 7388 [Thread-4] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b919000c closed
  54. 7388 [Thread-4-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
  55. 7389 [Thread-4] INFO  backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
  56. 7389 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:51115 which had sessionid 0x159a1b0b919000c
  57. 7390 [Thread-4] INFO  org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting
  58. 7390 [Thread-4] INFO  org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@6d895a70
  59. 7393 [Thread-4-SendThread(0:0:0:0:0:0:0:1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2000. Will not attempt to authenticate using SASL (unknown error)
  60. 7394 [Thread-4-SendThread(0:0:0:0:0:0:0:1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Socket connection established to 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2000, initiating session
  61. 7394 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /0:0:0:0:0:0:0:1:51118
  62. 7394 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /0:0:0:0:0:0:0:1:51118
  63. 7396 [SyncThread:0] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x159a1b0b919000d with negotiated timeout 20000 for client /0:0:0:0:0:0:0:1:51118
  64. 7396 [Thread-4-SendThread(0:0:0:0:0:0:0:1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2000, sessionid = 0x159a1b0b919000d, negotiated timeout = 20000
  65. 7396 [Thread-4-EventThread] INFO  org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED
  66. 7401 [Thread-4] INFO  backtype.storm.daemon.worker - Reading Assignments.
  67. 7834 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor __acker:[2 2]
  68. 7845 [Thread-4] INFO  backtype.storm.daemon.task - Emitting: __acker __system ["startup"]
  69. 7845 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks __acker:[2 2]
  70. 7863 [Thread-4] INFO  backtype.storm.daemon.executor - Timeouts disabled for executor __acker:[2 2]
  71. 7863 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor __acker:[2 2]
  72. 7871 [Thread-8-__acker] INFO  backtype.storm.daemon.executor - Preparing bolt __acker:(2)
  73. 7879 [Thread-8-__acker] INFO  backtype.storm.daemon.executor - Prepared bolt __acker:(2)
  74. 7888 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor bolt_wordcounter:[4 4]
  75. 7889 [Thread-4] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter __system ["startup"]
  76. 7890 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks bolt_wordcounter:[4 4]
  77. 7897 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor bolt_wordcounter:[4 4]
  78. 7900 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Preparing bolt bolt_wordcounter:(4)
  79. 7901 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Prepared bolt bolt_wordcounter:(4)
  80. 7907 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor __system:[-1 -1]
  81. 7908 [Thread-4] INFO  backtype.storm.daemon.task - Emitting: __system __system ["startup"]
  82. 7909 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks __system:[-1 -1]
  83. 7911 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor __system:[-1 -1]
  84. 7911 [Thread-4] INFO  backtype.storm.daemon.worker - Launching receive-thread for 1989826c-4520-4bbc-bb34-e09f5155ce91:1024
  85. 7911 [Thread-12-__system] INFO  backtype.storm.daemon.executor - Preparing bolt __system:(-1)
  86. 7929 [Thread-4] INFO  backtype.storm.daemon.worker - Worker has topology config {"storm.id" "word count-1-1484476433", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\01107252\\AppData\\Local\\Temp\\4a0fd3f4-bc1c-41b0-8319-2271783a8e8e", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.kryo.decorators" (), "topology.name" "word count", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1024 1025 1026), "topology.environment" nil, "topology.debug" true, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 2, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}
  87. 7929 [Thread-4] INFO  backtype.storm.daemon.worker - Worker f5fc123f-fb15-4515-a3c5-d658e983f9e3 for storm word count-1-1484476433 on 1989826c-4520-4bbc-bb34-e09f5155ce91:1024 has finished loading
  88. 7930 [Thread-13-worker-receiver-thread-0] INFO  backtype.storm.messaging.loader - Starting receive-thread: [stormId: word count-1-1484476433, port: 1024, thread-id: 0 ]
  89. 7937 [Thread-12-__system] INFO  backtype.storm.daemon.executor - Prepared bolt __system:(-1)
  90. 8355 [Thread-6] INFO  backtype.storm.daemon.worker - Reading Assignments.
  91. 8394 [Thread-6] INFO  backtype.storm.daemon.executor - Loading executor bolt_sentence:[3 3]
  92. 8395 [Thread-6] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence __system ["startup"]
  93. 8395 [Thread-6] INFO  backtype.storm.daemon.executor - Loaded executor tasks bolt_sentence:[3 3]
  94. 8398 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Preparing bolt bolt_sentence:(3)
  95. 8398 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Prepared bolt bolt_sentence:(3)
  96. 8398 [Thread-6] INFO  backtype.storm.daemon.executor - Finished loading executor bolt_sentence:[3 3]
  97. 8408 [Thread-6] INFO  backtype.storm.daemon.executor - Loading executor input:[5 5]
  98. 8410 [Thread-6] INFO  backtype.storm.daemon.task - Emitting: input __system ["startup"]
  99. 8410 [Thread-6] INFO  backtype.storm.daemon.executor - Loaded executor tasks input:[5 5]
  100. 8421 [Thread-18-input] INFO  backtype.storm.daemon.executor - Opening spout input:(5)
  101. 8422 [Thread-6] INFO  backtype.storm.daemon.executor - Finished loading executor input:[5 5]
  102. in open()*****************
  103. 8541 [Thread-18-input] INFO  backtype.storm.daemon.executor - Opened spout input:(5)
  104. 8544 [Thread-18-input] INFO  backtype.storm.daemon.executor - Activating spout input:(5)
  105. 8589 [Thread-6] INFO  backtype.storm.daemon.executor - Loading executor __system:[-1 -1]
  106. 8590 [Thread-6] INFO  backtype.storm.daemon.task - Emitting: __system __system ["startup"]
  107. 8591 [Thread-6] INFO  backtype.storm.daemon.executor - Loaded executor tasks __system:[-1 -1]
  108. 8593 [Thread-6] INFO  backtype.storm.daemon.executor - Finished loading executor __system:[-1 -1]
  109. 8595 [Thread-20-__system] INFO  backtype.storm.daemon.executor - Preparing bolt __system:(-1)
  110. 8595 [Thread-20-__system] INFO  backtype.storm.daemon.executor - Prepared bolt __system:(-1)
  111. 8609 [Thread-6] INFO  backtype.storm.daemon.executor - Loading executor __acker:[1 1]
  112. 8611 [Thread-6] INFO  backtype.storm.daemon.task - Emitting: __acker __system ["startup"]
  113. 8611 [Thread-6] INFO  backtype.storm.daemon.executor - Loaded executor tasks __acker:[1 1]
  114. 8613 [Thread-6] INFO  backtype.storm.daemon.executor - Timeouts disabled for executor __acker:[1 1]
  115. 8613 [Thread-6] INFO  backtype.storm.daemon.executor - Finished loading executor __acker:[1 1]
  116. 8613 [Thread-6] INFO  backtype.storm.daemon.worker - Launching receive-thread for 1a1baffd-69d0-42f9-89de-35a60496fbd1:1027
  117. 8614 [Thread-6] INFO  backtype.storm.daemon.worker - Worker has topology config {"storm.id" "word count-1-1484476433", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\01107252\\AppData\\Local\\Temp\\ad8f635a-51d8-41a8-bd21-722220739f15", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.kryo.decorators" (), "topology.name" "word count", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1027 1028 1029), "topology.environment" nil, "topology.debug" true, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 2, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}
  118. 8616 [Thread-6] INFO  backtype.storm.daemon.worker - Worker 7c37d53d-735c-43ee-8b1f-0ea0bf3ab564 for storm word count-1-1484476433 on 1a1baffd-69d0-42f9-89de-35a60496fbd1:1027 has finished loading
  119. 8615 [Thread-22-__acker] INFO  backtype.storm.daemon.executor - Preparing bolt __acker:(1)
  120. 8617 [Thread-22-__acker] INFO  backtype.storm.daemon.executor - Prepared bolt __acker:(1)
  121. 8618 [Thread-23-worker-receiver-thread-0] INFO  backtype.storm.messaging.loader - Starting receive-thread: [stormId: word count-1-1484476433, port: 1027, thread-id: 0 ]
  122. in nextTuple():china duan
  123. 18545 [Thread-18-input] INFO  backtype.storm.daemon.task - Emitting: input default [china duan]
  124. 18555 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Processing received message source: input:5, stream: default, id: {}, [china duan]
  125. 18555 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [china]
  126. 18560 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [duan]
  127. 18564 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [china]
  128. 18564 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
  129. china:1
  130. 18565 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [duan]
  131. 18565 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
  132. duan:1
  133. in nextTuple():this is a test
  134. 28559 [Thread-18-input] INFO  backtype.storm.daemon.task - Emitting: input default [this is a test]
  135. 28559 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Processing received message source: input:5, stream: default, id: {}, [this is a test]
  136. 28560 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [this]
  137. 28560 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [is]
  138. 28560 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [a]
  139. 28560 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [test]
  140. 28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [this]
  141. 28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
  142. this:1
  143. 28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [is]
  144. 28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
  145. is:1
  146. 28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [a]
  147. 28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
  148. a:1
  149. 28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [test]
  150. 28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
  151. test:1
  152. in nextTuple():I am fine
  153. 38561 [Thread-18-input] INFO  backtype.storm.daemon.task - Emitting: input default [I am fine]
  154. 38562 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Processing received message source: input:5, stream: default, id: {}, [I am fine]
  155. 38562 [Thread-18-input] INFO  backtype.storm.daemon.executor - Processing received message source: __system:-1, stream: __tick, id: {}, [30]
  156. 38562 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [I]
  157. 38562 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [am]
  158. 38562 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [fine]
  159. 38563 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [I]
  160. 38563 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
  161. I:1
  162. 38563 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [am]
  163. 38563 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
  164. am:1
  165. 38564 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [fine]
  166. 38564 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
  167. fine:1
  168. in nextTuple():I am fine
  169. 48564 [Thread-18-input] INFO  backtype.storm.daemon.task - Emitting: input default [I am fine]
  170. 48565 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Processing received message source: input:5, stream: default, id: {}, [I am fine]
  171. 48565 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [I]
  172. 48565 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [am]
  173. 48566 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [I]
  174. 48566 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 2]
  175. I:2
  176. 48566 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [fine]
  177. 48567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [am]
  178. 48567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 2]
  179. am:2
  180. 48567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [fine]
  181. 48567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 2]
  182. fine:2
  183. in nextTuple():china duan
  184. 58566 [Thread-18-input] INFO  backtype.storm.daemon.task - Emitting: input default [china duan]
  185. 58566 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Processing received message source: input:5, stream: default, id: {}, [china duan]
  186. 58566 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [china]
  187. 58566 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [duan]
  188. 58567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [china]
  189. 58567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 2]
  190. china:2
  191. 58567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [duan]
  192. 58567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 2]
  193. duan:2
  194. begin kill topology
  195. 65992 [main] INFO  backtype.storm.daemon.nimbus - Delaying event :remove for 30 secs for word count-1-1484476433
  196. 65999 [main] INFO  backtype.storm.daemon.nimbus - Updated word count-1-1484476433 with status {:type :killed, :kill-time-secs 30}
  197. 65999 [main] INFO  backtype.storm.daemon.nimbus - Shutting down master
  198. 66002 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b9190001
  199. 66004 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b9190001 closed
  200. 66004 [main-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
  201. 66005 [main] INFO  backtype.storm.daemon.nimbus - Shut down master
  202. 66012 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN  org.apache.storm.zookeeper.server.NIOServerCnxn - caught end of stream exception
  203. org.apache.storm.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read additional data from client sessionid 0x159a1b0b9190001, likely client has closed socket
  204.     at org.apache.storm.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) ~[storm-core-0.9.3.jar:0.9.3]
  205.     at org.apache.storm.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) [storm-core-0.9.3.jar:0.9.3]
  206.     at java.lang.Thread.run(Unknown Source) [na:1.8.0_31]
  207. 66012 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /0:0:0:0:0:0:0:1:51081 which had sessionid 0x159a1b0b9190001
  208. 66013 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b9190003
  209. 66014 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b9190003 closed
  210. 66014 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:51087 which had sessionid 0x159a1b0b9190003
  211. 66015 [main-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
  212. 66016 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b9190005
  213. 66018 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b9190005 closed
  214. 66018 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN  org.apache.storm.zookeeper.server.NIOServerCnxn - caught end of stream exception
  215. org.apache.storm.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read additional data from client sessionid 0x159a1b0b9190005, likely client has closed socket
  216.     at org.apache.storm.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) ~[storm-core-0.9.3.jar:0.9.3]
  217.     at org.apache.storm.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) [storm-core-0.9.3.jar:0.9.3]
  218.     at java.lang.Thread.run(Unknown Source) [na:1.8.0_31]
  219. 66019 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:51093 which had sessionid 0x159a1b0b9190005
  220. 66019 [main] INFO  backtype.storm.daemon.supervisor - Shutting down 1989826c-4520-4bbc-bb34-e09f5155ce91:f5fc123f-fb15-4515-a3c5-d658e983f9e3
  221. 66020 [main-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
  222. 66020 [main] INFO  backtype.storm.process-simulator - Killing process a311407c-b041-4fc2-a42b-52a959bf5f15
  223. 66021 [main] INFO  backtype.storm.daemon.worker - Shutting down worker word count-1-1484476433 1989826c-4520-4bbc-bb34-e09f5155ce91 1024
  224. 66021 [main] INFO  backtype.storm.daemon.worker - Shutting down receive thread
  225. 66021 [main] INFO  backtype.storm.messaging.loader - Shutting down receiving-thread: [word count-1-1484476433, 1024]
  226. 66021 [main] INFO  backtype.storm.messaging.loader - Waiting for receiving-thread:[word count-1-1484476433, 1024] to die
  227. 66021 [Thread-13-worker-receiver-thread-0] INFO  backtype.storm.messaging.loader - Receiving-thread:[word count-1-1484476433, 1024] received shutdown notice
  228. 66023 [main] INFO  backtype.storm.messaging.loader - Shutdown receiving-thread: [word count-1-1484476433, 1024]
  229. 66023 [main] INFO  backtype.storm.daemon.worker - Shut down receive thread
  230. 66023 [main] INFO  backtype.storm.daemon.worker - Terminating messaging context
  231. 66023 [main] INFO  backtype.storm.daemon.worker - Shutting down executors
  232. 66023 [main] INFO  backtype.storm.daemon.executor - Shutting down executor __acker:[2 2]
  233. 66023 [Thread-7-disruptor-executor[2 2]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
  234. 66023 [Thread-8-__acker] INFO  backtype.storm.util - Async loop interrupted!
  235. 66024 [main] INFO  backtype.storm.daemon.executor - Shut down executor __acker:[2 2]
  236. 66024 [main] INFO  backtype.storm.daemon.executor - Shutting down executor bolt_wordcounter:[4 4]
  237. 66024 [Thread-10-bolt_wordcounter] INFO  backtype.storm.util - Async loop interrupted!
  238. 66024 [Thread-9-disruptor-executor[4 4]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
  239. cleanup()
  240. 66025 [main] INFO  backtype.storm.daemon.executor - Shut down executor bolt_wordcounter:[4 4]
  241. 66025 [main] INFO  backtype.storm.daemon.executor - Shutting down executor __system:[-1 -1]
  242. 66025 [Thread-12-__system] INFO  backtype.storm.util - Async loop interrupted!
  243. 66025 [Thread-11-disruptor-executor[-1 -1]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
  244. 66025 [main] INFO  backtype.storm.daemon.executor - Shut down executor __system:[-1 -1]
  245. 66025 [main] INFO  backtype.storm.daemon.worker - Shut down executors
  246. 66026 [main] INFO  backtype.storm.daemon.worker - Shutting down transfer thread
  247. 66026 [Thread-14-disruptor-worker-transfer-queue] INFO  backtype.storm.util - Async loop interrupted!
  248. 66026 [main] INFO  backtype.storm.daemon.worker - Shut down transfer thread
  249. 66026 [main] INFO  backtype.storm.daemon.worker - Shutting down default resources
  250. 66027 [main] INFO  backtype.storm.daemon.worker - Shut down default resources
  251. 66036 [main] INFO  backtype.storm.daemon.worker - Disconnecting from storm cluster state context
  252. 66037 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b919000d
  253. 66038 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /0:0:0:0:0:0:0:1:51118 which had sessionid 0x159a1b0b919000d
  254. 66039 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b919000d closed
  255. 66039 [main] INFO  backtype.storm.daemon.worker - Shut down worker word count-1-1484476433 1989826c-4520-4bbc-bb34-e09f5155ce91 1024
  256. 66039 [Thread-4-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
  257. 66316 [main] INFO  backtype.storm.daemon.supervisor - Shut down 1989826c-4520-4bbc-bb34-e09f5155ce91:f5fc123f-fb15-4515-a3c5-d658e983f9e3
  258. 66316 [main] INFO  backtype.storm.daemon.supervisor - Shutting down supervisor 1989826c-4520-4bbc-bb34-e09f5155ce91
  259. 66317 [Thread-3] INFO  backtype.storm.event - Event manager interrupted
  260. 66317 [Thread-4] INFO  backtype.storm.event - Event manager interrupted
  261. 66318 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b9190007
  262. 66319 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b9190007 closed
  263. 66319 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:51099 which had sessionid 0x159a1b0b9190007
  264. 66319 [main-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
  265. 66321 [main] INFO  backtype.storm.daemon.supervisor - Shutting down 1a1baffd-69d0-42f9-89de-35a60496fbd1:7c37d53d-735c-43ee-8b1f-0ea0bf3ab564
  266. 66321 [main] INFO  backtype.storm.process-simulator - Killing process 36f44b82-647b-4812-8f60-70aa7aac4fa7
  267. 66321 [main] INFO  backtype.storm.daemon.worker - Shutting down worker word count-1-1484476433 1a1baffd-69d0-42f9-89de-35a60496fbd1 1027
  268. 66322 [main] INFO  backtype.storm.daemon.worker - Shutting down receive thread
  269. 66322 [main] INFO  backtype.storm.messaging.loader - Shutting down receiving-thread: [word count-1-1484476433, 1027]
  270. 66322 [Thread-23-worker-receiver-thread-0] INFO  backtype.storm.messaging.loader - Receiving-thread:[word count-1-1484476433, 1027] received shutdown notice
  271. 66322 [main] INFO  backtype.storm.messaging.loader - Waiting for receiving-thread:[word count-1-1484476433, 1027] to die
  272. 66322 [main] INFO  backtype.storm.messaging.loader - Shutdown receiving-thread: [word count-1-1484476433, 1027]
  273. 66322 [main] INFO  backtype.storm.daemon.worker - Shut down receive thread
  274. 66322 [main] INFO  backtype.storm.daemon.worker - Terminating messaging context
  275. 66322 [main] INFO  backtype.storm.daemon.worker - Shutting down executors
  276. 66323 [main] INFO  backtype.storm.daemon.executor - Shutting down executor bolt_sentence:[3 3]
  277. 66323 [Thread-16-bolt_sentence] INFO  backtype.storm.util - Async loop interrupted!
  278. 66323 [Thread-15-disruptor-executor[3 3]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
  279. cleanup()
  280. 66323 [main] INFO  backtype.storm.daemon.executor - Shut down executor bolt_sentence:[3 3]
  281. 66323 [main] INFO  backtype.storm.daemon.executor - Shutting down executor input:[5 5]
  282. 66324 [Thread-17-disruptor-executor[5 5]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
  283. 66324 [Thread-18-input] INFO  backtype.storm.util - Async loop interrupted!
  284. 66324 [main] INFO  backtype.storm.daemon.executor - Shut down executor input:[5 5]
  285. 66324 [main] INFO  backtype.storm.daemon.executor - Shutting down executor __system:[-1 -1]
  286. 66324 [Thread-20-__system] INFO  backtype.storm.util - Async loop interrupted!
  287. 66324 [Thread-19-disruptor-executor[-1 -1]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
  288. 66324 [main] INFO  backtype.storm.daemon.executor - Shut down executor __system:[-1 -1]
  289. 66324 [main] INFO  backtype.storm.daemon.executor - Shutting down executor __acker:[1 1]
  290. 66325 [Thread-22-__acker] INFO  backtype.storm.util - Async loop interrupted!
  291. 66325 [Thread-21-disruptor-executor[1 1]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
  292. 66325 [main] INFO  backtype.storm.daemon.executor - Shut down executor __acker:[1 1]
  293. 66325 [main] INFO  backtype.storm.daemon.worker - Shut down executors
  294. 66325 [main] INFO  backtype.storm.daemon.worker - Shutting down transfer thread
  295. 66325 [Thread-24-disruptor-worker-transfer-queue] INFO  backtype.storm.util - Async loop interrupted!
  296. 66325 [main] INFO  backtype.storm.daemon.worker - Shut down transfer thread
  297. 66326 [main] INFO  backtype.storm.daemon.worker - Shutting down default resources
  298. 66326 [main] INFO  backtype.storm.daemon.worker - Shut down default resources
  299. 66330 [main] INFO  backtype.storm.daemon.worker - Disconnecting from storm cluster state context
  300. 66330 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b919000b
  301. 66332 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b919000b closed
  302. 66332 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:51112 which had sessionid 0x159a1b0b919000b
  303. 66332 [main] INFO  backtype.storm.daemon.worker - Shut down worker word count-1-1484476433 1a1baffd-69d0-42f9-89de-35a60496fbd1 1027
  304. 66332 [Thread-6-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
  305. 66594 [main] INFO  backtype.storm.daemon.supervisor - Shut down 1a1baffd-69d0-42f9-89de-35a60496fbd1:7c37d53d-735c-43ee-8b1f-0ea0bf3ab564
  306. 66595 [main] INFO  backtype.storm.daemon.supervisor - Shutting down supervisor 1a1baffd-69d0-42f9-89de-35a60496fbd1
  307. 66595 [Thread-5] INFO  backtype.storm.event - Event manager interrupted
  308. 66595 [Thread-6] INFO  backtype.storm.event - Event manager interrupted
  309. 66596 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b9190009
  310. 66597 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b9190009 closed
  311. 66598 [main-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
  312. 66598 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /0:0:0:0:0:0:0:1:51105 which had sessionid 0x159a1b0b9190009
  313. 66598 [main] INFO  backtype.storm.testing - Shutting down in process zookeeper
  314. 66598 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxnFactory - NIOServerCnxn factory exited run method
  315. 66598 [main] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - shutting down
  316. 66598 [main] INFO  org.apache.storm.zookeeper.server.SessionTrackerImpl - Shutting down
  317. 66598 [main] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Shutting down
  318. 66599 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - PrepRequestProcessor exited loop!
  319. 66599 [main] INFO  org.apache.storm.zookeeper.server.SyncRequestProcessor - Shutting down
  320. 66599 [SyncThread:0] INFO  org.apache.storm.zookeeper.server.SyncRequestProcessor - SyncRequestProcessor exited!
  321. 66599 [main] INFO  org.apache.storm.zookeeper.server.FinalRequestProcessor - shutdown of request processor complete
  322. 66599 [main] INFO  backtype.storm.testing - Done shutting down in process zookeeper
  323. 66599 [main] INFO  backtype.storm.testing - Deleting temporary path C:\Users\01107252\AppData\Local\Temp\12d6d452-2f70-4f4f-8451-e21c36371261
  324. 66614 [main] INFO  backtype.storm.testing - Deleting temporary path C:\Users\01107252\AppData\Local\Temp\00f65db7-e527-4998-b253-110ce0e5ab71
  325. 66618 [main] INFO  backtype.storm.testing - Unable to delete file: C:\Users\01107252\AppData\Local\Temp\00f65db7-e527-4998-b253-110ce0e5ab71\version-2\log.1
  326. 66618 [main] INFO  backtype.storm.testing - Deleting temporary path C:\Users\01107252\AppData\Local\Temp\4a0fd3f4-bc1c-41b0-8319-2271783a8e8e
  327. 66660 [main] INFO  backtype.storm.testing - Deleting temporary path C:\Users\01107252\AppData\Local\Temp\ad8f635a-51d8-41a8-bd21-722220739f15
  328. 68397 [SessionTracker] INFO  org.apache.storm.zookeeper.server.SessionTrackerImpl - SessionTrackerImpl exited loop!

关键的方法是 execute方法。 正如你可以看到，它将句子拆分成单词，并发出每个单词作为一个新的元组。 另一个重要的方法是declareOutputFields，其中宣布bolts输出元组的架构。 在这里宣布，它发出一个域为word的元组。

setBolt的最后一个参数是你想为bolts的并行量。 SplitSentence bolts 是10个并发，这将导致在storm集群中有十个线程并行执行。 你所要做的的是增加bolts的并行量在遇到topology的瓶颈时。

setBolt方法返回一个对象，用来定义bolts的输入。 例如，SplitSentence螺栓订阅组件“1”使用随机分组的输出流。 “1”是指已经定义KestrelSpout。 我将解释在某一时刻的随机分组的一部分。 到目前为止，最要紧的是，SplitSentence bolts会消耗KestrelSpout发出的每一个元组。

storm的运行有两种模式: 本地模式和分布式模式.

1) 本地模式：

storm用一个进程里面的线程来模拟所有的spout和bolt. 本地模式对开发和测试来说比较有用。 你运行storm-starter里面的topology的时候它们就是以本地模式运行的， 你可以看到topology里面的每一个组件在发射什么消息。

2) 分布式模式：

storm由一堆机器组成。当你提交topology给master的时候， 你同时也把topology的代码提交了。master负责分发你的代码并且负责给你的topolgoy分配工作进程。如果一个工作进程挂掉了， master节点会把认为重新分配到其它节点。

首先， 这个代码定义通过定义一个LocalCluster对象来定义一个进程内的集群。提交topology给这个虚拟的集群和提交topology给分布式集群是一样的。通过调用submitTopology方法来提交topology， 它接受三个参数：要运行的topology的名字，一个配置对象以及要运行的topology本身。

topology的名字是用来唯一区别一个topology的，这样你然后可以用这个名字来杀死这个topology的。前面已经说过了， 你必须显式的杀掉一个topology， 否则它会一直运行。

Conf对象可以配置很多东西， 下面两个是最常见的：

TOPOLOGY_WORKERS(setNumWorkers) 定义你希望集群分配多少个工作进程给你来执行这个topology. topology里面的每个组件会被需要线程来执行。每个组件到底用多少个线程是通过setBolt和setSpout来指定的。这些线程都运行在工作进程里面. 每一个工作进程包含一些节点的一些工作线程。比如， 如果你指定300个线程，60个进程， 那么每个工作进程里面要执行6个线程， 而这6个线程可能属于不同的组件(Spout, Bolt)。你可以通过调整每个组件的并行度以及这些线程所在的进程数量来调整topology的性能。

TOPOLOGY_DEBUG(setDebug), 当它被设置成true的话， storm会记录下每个组件所发射的每条消息。这在本地环境调试topology很有用， 但是在线上这么做的话会影响性能的。

结论：

本章从storm的基本对象的定义，到广泛的介绍了storm的开发环境，从一个简单的例子讲解了topology的构建和定义。希望大家可以从本章的内容对storm有一个基本的理解和概念，并且已经可以构建一个简单的topology！！

Storm 01之 Storm基本概念及第一个demo的更多相关文章

  1. Springboot 1.简介 及第一个demo

    按照官网上的新建一个maven项目,然后将类引入pom.xml文件中 <?xml version="1.0" encoding="UTF-8"?> ...

  2. 测试框架httpclent 1.HttpClient简介及第一个demo

    httpclient就是一个模拟 发送http请求的一个工具. 首先在pom.xml文件里面添加工具类 <dependencies> <dependency> <grou ...

  3. lua入门之环境搭建、第一个demo

    前言 前段时间因为有些项目功能需要,自己研究了下lua,今天整理下,并以一个demo为示例演示 手机上的运行效果 分为几个步骤来逐步讲解. 1.lua介绍,为什么选择它? 2.环境安装 3.撸一个简单 ...

  4. 理解 Storm 拓扑的并行度(parallelism)概念

    组成:一个运行中的拓扑是由什么构成的:工作进程(worker processes),执行器(executors)和任务(tasks)! 在一个 Storm 集群中,Storm 主要通过以下三个部件来运 ...

  5. 【转载】Apache Storm 官方文档 —— 基础概念

    [转载自https://yq.aliyun.com/articles/87510?t=t1] Storm 系统中包含以下几个基本概念: 拓扑(Topologies) 流(Streams) 数据源(Sp ...

  6. apache Storm学习之二-基本概念介绍

    2.1 Storm基本概念 在运行一个Storm任务之前,需要了解一些概念: Topologies Streams Spouts Bolts Stream groupings Reliability ...

  7. Storm概念学习系列之核心概念（Tuple、Spout、Blot、Stream、Stream Grouping、Worker、Task、Executor、Topology）（博主推荐）

    不多说,直接上干货! 以下都是非常重要的storm概念知识. (Tuple元组数据载体 .Spout数据源.Blot消息处理者.Stream消息流 和 Stream Grouping 消息流组.Wor ...

  8. Storm系列一: Storm初步

    初入Storm 前言 学习Storm已经有两周左右的时间,但是认真来说学习过程确实是零零散散,遇到问题去百度一下,找到新概念再次学习,在这样的一个循环又不成体系的过程中不断学习Storm. 前人栽树, ...

  9. storm笔记：Storm+Kafka简单应用

    storm笔记:Storm+Kafka简单应用 这几天工作须要使用storm+kafka,基本场景是应用出现错误,发送日志到kafka的某个topic.storm订阅该topic.然后进行兴许处理.场 ...

随机推荐

  1. beautifulsoup的一些使用

    自动补全代码: import requests from bs4 import BeautifulSoup response=requests.get('https://www.ithome.com/ ...

  2. windows forget jenkins password.

    1. goto the directory of jenkins. for example my installation is in D:\Program Files (x86)\Jenkins. ...

  3. navicat for mysql 快捷键(原创)

    navicat for mysql 快捷键(原创) 在谷歌,百度上基本搜索不出来这方面的内容,我总结了一下,方便新手,节省一些探索的时间. 1.ctrl+q           打开查询窗口2.ctr ...

  4. 为Redmine的项目加上起止时间

    没有时间约束的项目不是好项目. 要给项目配置起止时间,须要用到自己定义属性. 我们须要管理员身份登录.才干够定义自己定义属性. 自己定义属性 看图吧,先是点击页面导航条(最上面那排菜单,有主页.我的工 ...

  5. 笔记04 WPF对象引用

    转自:http://www.fx114.net/qa-261-90254.aspx 我们应该都知道,XAML是一种声明式语言,XAML的标签声明的就是对象.一个XAML标签会对应着一个对象,这个对象一 ...

  6. RDLC报表 报表数据 栏 快捷键

    血淋淋的教训啊,这段时间用RDLC报表,创建报表后会在右边加载[服务器资源管理器][工具栏][报表数据]一些栏目 如图 [服务器资源管理器][工具栏]还好[报表数据]栏在开发的时候不小心点了X给关掉了 ...

  7. Android 四大组件学习之BroadcastReceiver一

    本节课学习四大组件最后一个, 广播接受者. 顾名思义广播接受者就是接受广播呗.比方在现实社会中,曾经每一个人家都有一台收音机,这可就能够去接受广播发出来的消息.大家都知道.程序世界也是參照的显示生活设 ...

  8. session.use_cookies有什么作用，

    session.use_cookies有什么作用, @ini_set('session.use_cookies', 1);什么意思 若session.use_cookies = 1 sessionid ...

  9. Erlang二进制模式匹配

    Erlang的模式匹配用来处理二进制数据可谓是得心应手.不仅直观,而且超乎想象的简单.在C++中,处理二进制数据首先要管理缓冲区.然后再按字节进行操作,如果要处理的数据不是按字节对齐,则需要进行位移等 ...

  10. ESXi 5.5 RTL9168E网卡驱动 找到网卡

    如果你对专业服务器的价格望而却步,恰巧又想在普通的PC上安装ESXi,恰巧又是ESXi 5.5版本,那么这篇文章中提及的问题你可能会遇到,并能给你提供一些帮助. 1.成功安装重启以后提示“no boo ...