Storm工程创建
1.创建maven项目:
pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion></modelVersion> <groupId>storm.book</groupId> <artifactId>Getting-Started</artifactId> <version>-SNAPSHOT</version> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version></version> <configuration> <source>1.6</source> <target>1.6</target> <compilerVersion>1.6</compilerVersion> </configuration> </plugin> </plugins> </build> <repositories> <!-- Repository where we can found the storm dependencies --> <repository> <id>clojars.org</id> <url>http://clojars.org/repo</url> </repository> </repositories> <dependencies> <!-- Storm Dependency --> <dependency> <groupId>storm</groupId> <artifactId>storm</artifactId> <version></version> </dependency> </dependencies> </project>
相关类:
WordReader:
package spouts; import java.io.BufferedReader; import java.io.FileNotFoundException; import java.io.FileReader; import java.util.Map; import backtype.storm.spout.SpoutOutputCollector; import backtype.storm.task.TopologyContext; import backtype.storm.topology.OutputFieldsDeclarer; import backtype.storm.topology.base.BaseRichSpout; import backtype.storm.tuple.Fields; import backtype.storm.tuple.Values; public class WordReader extends BaseRichSpout { private SpoutOutputCollector collector; private FileReader fileReader; private boolean completed = false; public void ack(Object msgId) { System.out.println("OK:"+msgId); } public void close() {} public void fail(Object msgId) { System.out.println("FAIL:"+msgId); } /** * The only thing that the methods will do It is emit each * file line */ public void nextTuple() { /** * The nextuple it is called forever, so if we have been readed the file * we will wait and then return */ if(completed){ try { Thread.sleep(); } catch (InterruptedException e) { //Do nothing } return; } String str; //Open the reader BufferedReader reader = new BufferedReader(fileReader); try{ //Read all lines while((str = reader.readLine()) != null){ /** * By each line emmit a new value with the line as a their */ this.collector.emit(new Values(str),str); } }catch(Exception e){ throw new RuntimeException("Error reading tuple",e); }finally{ completed = true; } } /** * We will create the file and get the collector object */ public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) { try { this.fileReader = new FileReader(conf.get("wordsFile").toString()); } catch (FileNotFoundException e) { throw new RuntimeException("Error reading file ["+conf.get("wordFile")+"]"); } this.collector = collector; } /** * Declare the output field "word" */ public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("line")); } }
WordCounter:
package bolts; import java.util.HashMap; import java.util.Map; import backtype.storm.task.TopologyContext; import backtype.storm.topology.BasicOutputCollector; import backtype.storm.topology.OutputFieldsDeclarer; import backtype.storm.topology.base.BaseBasicBolt; import backtype.storm.tuple.Tuple; public class WordCounter extends BaseBasicBolt { Integer id; String name; Map<String, Integer> counters; /** * At the end of the spout (when the cluster is shutdown * We will show the word counters */ @Override public void cleanup() { System.out.println("-- Word Counter ["+name+"-"+id+"] --"); for(Map.Entry<String, Integer> entry : counters.entrySet()){ System.out.println(entry.getKey()+": "+entry.getValue()); } } /** * On create */ @Override public void prepare(Map stormConf, TopologyContext context) { this.counters = new HashMap<String, Integer>(); this.name = context.getThisComponentId(); this.id = context.getThisTaskId(); } @Override public void declareOutputFields(OutputFieldsDeclarer declarer) {} @Override public void execute(Tuple input, BasicOutputCollector collector) { String str = input.getString(); /** * If the word dosn't exist in the map we will create * this, if not We will add 1 */ if(!counters.containsKey(str)){ counters.put(str, ); }else{ Integer c = counters.; counters.put(str, c); } } }
WordNormalizer:
package bolts; import backtype.storm.topology.BasicOutputCollector; import backtype.storm.topology.OutputFieldsDeclarer; import backtype.storm.topology.base.BaseBasicBolt; import backtype.storm.tuple.Fields; import backtype.storm.tuple.Tuple; import backtype.storm.tuple.Values; public class WordNormalizer extends BaseBasicBolt { public void cleanup() {} /** * The bolt will receive the line from the * words file and process it to Normalize this line * * The normalize will be put the words in lower case * and split the line to get all words in this */ public void execute(Tuple input, BasicOutputCollector collector) { String sentence = input.getString(); String[] words = sentence.split(" "); for(String word : words){ word = word.trim(); if(!word.isEmpty()){ word = word.toLowerCase(); collector.emit(new Values(word)); } } } /** * The bolt will only emit the field "word" */ public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word")); } }
TopologyMain:
import spouts.WordReader; import backtype.storm.Config; import backtype.storm.LocalCluster; import backtype.storm.topology.TopologyBuilder; import backtype.storm.tuple.Fields; import bolts.WordCounter; import bolts.WordNormalizer; public class TopologyMain { public static void main(String[] args) throws InterruptedException { //Topology definition TopologyBuilder builder = new TopologyBuilder(); builder.setSpout("word-reader",new WordReader()); builder.setBolt("word-normalizer", new WordNormalizer()) .shuffleGrouping("word-reader"); builder.setBolt() .fieldsGrouping("word-normalizer", new Fields("word")); //Configuration Config conf = new Config(); conf.put(]); conf.setDebug(false); //Topology run conf.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, ); LocalCluster cluster = new LocalCluster(); cluster.submitTopology("Getting-Started-Toplogie", conf, builder.createTopology()); Thread.sleep(); cluster.shutdown(); } }
工程架构
运行:
结果
强大的流处理!
github:https://github.com/super-d2/strom_demo
Storm工程创建的更多相关文章
- Android开发之基于AndroidStudio环境搭建和工程创建
断断续续的学习安卓也有一段时间了.因为之前是搞iOS开发的, 之前有关iOS的博客请看<我的iOS开发系列博文>.<我的Objective-C系列文章>和<窥探Swift ...
- 161130、Dubbo+SpringMVC工程创建详解
Dubbo出现的目的是为了应对现在高并发,高数据量请求的问题.目前的垂直应用架构已经无法满足现在大数据的冲击,SOA就应运而生,而Dubbo在国内使用的还是比较多,稳定性也比较不错. 架构 节点角色说 ...
- VC++2008 用空工程创建 DLL
VC++2008 用空工程创建 DLL 一.创建 DLL 工程项目: 1)点击菜单[File] -> [New] -> [Project...] 弹出 “New Project” 对话框: ...
- Django初探——工程创建以及models数据库相关配置
Python的WEB框架有Django.Tornado.Flask 等多种,Django相较与其他WEB框架其优势为:大而全,框架本身集成了ORM.模型绑定.模板引擎.缓存.Session等诸多功能. ...
- N76E003 工程创建教程
一.准备工作: 1.下载编译工具keil c51 2.下载N76E003提供的板级支持包(BSP),可到nuvoton上下载 二.开发环境搭建 1.安装keil c51,然后和谐...不能随便发链 ...
- 第1天:Django框架简介与工程创建
Django简介 Django特点 环境搭建 创建工程 启动服务 使用pycharm打开工程 创建子应用 Django简介 Django,是用Python语言写的开源web开发框架,并遵循MVC设计. ...
- java web 工程创建及servlet简单使用
1.java web工程创建 (1)File--->new--->project (2)选择java enterprise,按照下图操作 (3)点击next后,会进入如下界面,修改工程名后 ...
- Django工程创建
方法一: 1.win+r进入cmd命令窗口: 2.找到Django的安装地址: 3.cmd窗口中利用cd 进入相应的文件夹,再输入命令如下: django-admin.exe startproject ...
- Django—工程创建以及models数据库易错点
Python的WEB框架有Django.Tornado.Flask 等多种,Django相较与其他WEB框架其优势为:大而全,框架本身集成了ORM.模型绑定.模板引擎.缓存.Session等诸多功能. ...
随机推荐
- DataTable列上多值运算
1.从网上找了个中缀算法(也不知道什么前缀后缀,抱歉),可以对字符串表达式进行运算 2.有些时候还是会用到ASCII码表的 char c = expression[k];//expression为一字 ...
- 《JAVA与模式》之享元模式
<JAVA与模式>之享元模式 在阎宏博士的<JAVA与模式>一书中开头是这样描述享元(Flyweight)模式的: Flyweight在拳击比赛中指最轻量级,即“蝇量级”或“雨 ...
- LiLinux系统下如何修改主机名
1,用root用户登录,或者切换root用户,先查看当前的主机名:hostname (如果之前没有修改过,一般默认为localhost.localdomain): 2,vi /etc/sysconf ...
- 手机wifi密码的保存位置
subjects: adj. 受制于...的, 被统治的; n. 主题,学科, 国民 the subjects had to kneel down before the king. kneel -& ...
- Linux下链接mysql数据库的命令
一.MySQL 连接本地数据库,用户名为“root”,密码“123”(注意:“-p”和“123” 之间不能有空格) C:\>mysql -h localhost -u root -p123 二. ...
- ThinkPHP3.2.3自带的分页用法--很简单实用
把解压后的Page.class.php放入ThinkPHP/Extend/Library/ORG/Util/(如果没有请手动创建)目录下面.thinkphp 自带的分页非常好用美观,先看一下如下代码片 ...
- (9)UI(容器)
1.基础容器 基础容器可以设置子容器布局.是否裁切子元素.填充颜色.背景图片资源等属性. 使用场景. 在官方示例中,大量使用了基础容器作布局管理,如下面的主界面中,用户名称,钻石和金币就使 ...
- 题目1006:ZOJ问题
时间限制:1 秒 内存限制:32 兆 特殊判题:否 提交:13212 解决:2214 题目描述: 对给定的字符串(只包含'z','o','j'三种字符),判断他是否能AC. 是否AC的规则如下:1. ...
- 浅析django的abstract,proxy, managed
django.db.models.Model 的 Meta参数 参数 类型 说明 继承 abstract boolean 是否建表 不继承,子类自动充值为默认值(False) managed bool ...
- 【Hibernate】Hibernate系列2之Session详解
Session详解 2.1.概述-一级缓存 2.2.操作session缓存方法 2.3.数据库隔离级别 2.4.持久化状态 2.5.状态转换 2.6.存储过程与触发器