storm trident的filter和函数

目的：通过kafka输出的信息进行过滤，添加指定的字段后，进行打印

SentenceSpout:

package Trident;

import java.util.HashMap;

import java.util.List;

import java.util.Map;

import java.util.Properties;

import org.apache.storm.spout.SpoutOutputCollector;

import org.apache.storm.task.TopologyContext;

import org.apache.storm.topology.OutputFieldsDeclarer;

import org.apache.storm.topology.base.BaseRichSpout;

import org.apache.storm.tuple.Fields;

import org.apache.storm.tuple.Values;

import kafka.consumer.Consumer;

import kafka.consumer.ConsumerConfig;

import kafka.consumer.ConsumerIterator;

import kafka.consumer.KafkaStream;

import kafka.javaapi.consumer.ConsumerConnector;

import kafka.serializer.StringDecoder;

import kafka.utils.VerifiableProperties;

/**

 * 从kafka获取数据 spout发射

 * @author BFD-593

 *

 */

public class SentenceSpout extends BaseRichSpout{

	//TODO

	private SpoutOutputCollector collector;

	private ConsumerConnector consumer;

	private int index=0;

	@Override

	public void nextTuple() {

		Map<String, Integer> topicCountMap = new HashMap<String, Integer>();

        topicCountMap.put("helloworld", new Integer(1));  

        StringDecoder keyDecoder = new StringDecoder(new VerifiableProperties());

        StringDecoder valueDecoder = new StringDecoder(new VerifiableProperties());

        Map<String, List<KafkaStream<String, String>>> consumerMap =

        consumer.createMessageStreams(topicCountMap,keyDecoder,valueDecoder);

        KafkaStream<String, String> stream = consumerMap.get("helloworld").get(0);

        ConsumerIterator<String, String> it = stream.iterator();  

        int messageCount = 0;

        while (it.hasNext()){

        	String string = it.next().message().toString()+" 1"+" 2";

        	String name = string.split(" ")[0];

        	String value = string.split(" ")[1]==null?"":string.split(" ")[1];

        	String value2= string.split(" ")[2]==null?"":string.split(" ")[2];

            this.collector.emit(new Values(name,value,value2));

        }

	}

	@Override

	public void open(Map map, TopologyContext context, SpoutOutputCollector collector) {

		this.collector =  collector;

		Properties props = new Properties();

		 // zookeeper 配置

        props.put("zookeeper.connect", "192.168.170.185:2181");  

        // 消费者所在组

        props.put("group.id", "testgroup");  

        // zk连接超时

        props.put("zookeeper.session.timeout.ms", "4000");

        props.put("zookeeper.sync.time.ms", "200");

        props.put("auto.commit.interval.ms", "1000");

        props.put("auto.offset.reset", "smallest");  

        // 序列化类

        props.put("serializer.class", "kafka.serializer.StringEncoder");  

        ConsumerConfig config = new ConsumerConfig(props);

		this.consumer = Consumer.createJavaConsumerConnector(config);

	}

	@Override

	public void declareOutputFields(OutputFieldsDeclarer declarer) {

		Fields field = new Fields("name", "sentence","sentence2");

		declarer.declare(field);

	}

}

FunctionBolt:

	package Trident;

	import org.apache.storm.trident.operation.BaseFunction;

	import org.apache.storm.trident.operation.TridentCollector;

	import org.apache.storm.trident.tuple.TridentTuple;

	import org.apache.storm.tuple.Values;

	/**

	 * trident的函数操作：将spout发射的数据，添加一个fileds   gender的

	 * 它不会替换掉原来的tuple

	 * @author BFD-593

	 *

	 */

	public class FunctionBolt extends BaseFunction{

		@Override

		public void execute(TridentTuple tuple, TridentCollector collector) {

			String str = tuple.getStringByField("name");

			if(str.equals("a")){

				collector.emit(new Values("男"));

			}else{

				collector.emit(new Values("女"));

			}

		}

	}

MyFilter:

package Trident;

import java.util.Map;

import org.apache.storm.trident.operation.BaseFilter;

import org.apache.storm.trident.operation.TridentOperationContext;

import org.apache.storm.trident.tuple.TridentTuple;

/**

 * trident的过滤操作：将spout的发送的tuple，过滤掉fields0是a并且fields1是b的tuple

 * @author BFD-593

 *

 */

public class MyFilter extends BaseFilter{

	private TridentOperationContext context;

	@Override

	public void prepare(Map conf, TridentOperationContext context) {

		super.prepare(conf, context);

		this.context = context;

	}

	@Override

	public boolean isKeep(TridentTuple tuple) {

		String name = tuple.getStringByField("name");

		String value = tuple.getStringByField("sentence");

		return (!"a".equals(name))||(!"b".equals(value));

	}

}

PrintFilter:

package Trident;

import java.util.Iterator;

import java.util.Map;

import org.apache.storm.trident.operation.BaseFilter;

import org.apache.storm.trident.operation.TridentOperationContext;

import org.apache.storm.trident.tuple.TridentTuple;

import org.apache.storm.tuple.Fields;

/**

 * 过滤打印所有的fields以及值

 * @author BFD-593

 *

 */

public class PrintFilter extends BaseFilter{

	private TridentOperationContext context = null;

	@Override

	public void prepare(Map conf, TridentOperationContext context) {

		super.prepare(conf, context);

		this.context = context;

	}

	@Override

	public boolean isKeep(TridentTuple tuple) {

		Fields fields = tuple.getFields();

		Iterator<String> iterator = fields.iterator();

		String str = "";

		while(iterator.hasNext()){

			String next = iterator.next();

			Object value = tuple.getValueByField(next);

			str = str + next +":"+ value+",";

		}

		System.out.println(str);

		return true;

	}

}

TopologyTrident:

package Trident;

import org.apache.kafka.common.utils.Utils;

import org.apache.storm.Config;

import org.apache.storm.LocalCluster;

import org.apache.storm.trident.TridentTopology;

import org.apache.storm.trident.operation.builtin.Count;

import org.apache.storm.tuple.Fields;

/**

 * trident的过滤操作、函数操作、分驱聚合操作

 * @author BFD-593

 *

 */

public class TopologyTrident {

	public static void main(String[] args) {

		SentenceSpout spout = new SentenceSpout();

		TridentTopology topology = new TridentTopology();

		topology.newStream("spout", spout).each(new Fields("name"),new FunctionBolt(),new Fields("gender")).each(new Fields("name","sentence"), new MyFilter())

		.each(new Fields("name","sentence","sentence2","gender"), new PrintFilter());

		Config conf = new Config();

		LocalCluster clu = new LocalCluster();

		clu.submitTopology("mytopology", conf, topology.build());

		Utils.sleep(100000000);

		clu.killTopology("mytopology");

		clu.shutdown();

	}

}

package Trident;
import java.util.HashMap;import java.util.List;import java.util.Map;import java.util.Properties;
import org.apache.storm.spout.SpoutOutputCollector;import org.apache.storm.task.TopologyContext;import org.apache.storm.topology.OutputFieldsDeclarer;import org.apache.storm.topology.base.BaseRichSpout;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Values;
import kafka.consumer.Consumer;import kafka.consumer.ConsumerConfig;import kafka.consumer.ConsumerIterator;import kafka.consumer.KafkaStream;import kafka.javaapi.consumer.ConsumerConnector;import kafka.serializer.StringDecoder;import kafka.utils.VerifiableProperties;
/** * 从kafka获取数据 spout发射 * @author BFD-593 * */public class SentenceSpout extends BaseRichSpout{//TODOprivate SpoutOutputCollector collector;private ConsumerConnector consumer;private int index=0;@Overridepublic void nextTuple() {Map<String, Integer> topicCountMap = new HashMap<String, Integer>(); topicCountMap.put("helloworld", new Integer(1)); StringDecoder keyDecoder = new StringDecoder(new VerifiableProperties()); StringDecoder valueDecoder = new StringDecoder(new VerifiableProperties()); Map<String, List<KafkaStream<String, String>>> consumerMap = consumer.createMessageStreams(topicCountMap,keyDecoder,valueDecoder); KafkaStream<String, String> stream = consumerMap.get("helloworld").get(0); ConsumerIterator<String, String> it = stream.iterator(); int messageCount = 0; while (it.hasNext()){ String string = it.next().message().toString()+" 1"+" 2"; String name = string.split(" ")[0]; String value = string.split(" ")[1]==null?"":string.split(" ")[1]; String value2= string.split(" ")[2]==null?"":string.split(" ")[2]; this.collector.emit(new Values(name,value,value2)); } }
@Overridepublic void open(Map map, TopologyContext context, SpoutOutputCollector collector) {this.collector = collector;Properties props = new Properties(); // zookeeper 配置 props.put("zookeeper.connect", "192.168.170.185:2181"); // 消费者所在组 props.put("group.id", "testgroup"); // zk连接超时 props.put("zookeeper.session.timeout.ms", "4000"); props.put("zookeeper.sync.time.ms", "200"); props.put("auto.commit.interval.ms", "1000"); props.put("auto.offset.reset", "smallest"); // 序列化类 props.put("serializer.class", "kafka.serializer.StringEncoder"); ConsumerConfig config = new ConsumerConfig(props); this.consumer = Consumer.createJavaConsumerConnector(config);}
@Overridepublic void declareOutputFields(OutputFieldsDeclarer declarer) {Fields field = new Fields("name", "sentence","sentence2");declarer.declare(field);}
}

storm trident的filter和函数的更多相关文章

storm trident function函数
package cn.crxy.trident; import java.util.List; import backtype.storm.Config; import backtype.storm. ...
Strom-7 Storm Trident 详细介绍
一.概要 1.1 Storm(简介) Storm是一个实时的可靠地分布式流计算框架. 具体就不多说了,举个例子,它的一个典型的大数据实时计算应用场景:从Kafka消息队列读取消息( ...
Storm Trident API
在Storm Trident中有五种操作类型 Apply Locally:本地操作,所有操作应用在本地节点数据上,不会产生网络传输 Repartitioning:数据流重定向,单纯的改变数据流向,不会 ...
Storm专题二：Storm Trident API 使用具体解释
一.概述 Storm Trident中的核心数据模型就是"Stream",也就是说,Storm Trident处理的是Stream.可是实际上Stream是被成批处理的. ...
storm trident 示例
Storm Trident的核心数据模型是一批一批被处理的“流”,“流”在集群的分区在集群的节点上,对“流”的操作也是并行的在每个分区上进行. Trident有五种对“流”的操作: 1. 不 ...
storm trident merger
import java.util.List; import backtype.storm.Config; import backtype.storm.LocalCluster; import back ...
Python【day 14-5】sorted filter map函数应用和练习
'''''' ''' 内置函数或者和匿名函数结合输出 4,用map来处理字符串列表,把列表中所有人都变成sb,比方alex_sb name=[‘oldboy’,'alex','wusir'] 5,用m ...
lambda匿名函数sorted排序函数filter过滤函数map映射函数
lambda函数:表示匿名函数,不需要def来声明,一句话就能搞定. 语法:函数名=lamda 参数:返回值求10的10次方 f=lambda n:n**n print(f(10)) 注意: 函数名 ...
storm trident 的介绍与使用
一.trident 的介绍 trident 的英文意思是三叉戟,在这里我的理解是因为之前我们通过之前的学习topology spout bolt 去处理数据是没有问题的,但trident 的对spou ...

随机推荐

BZOJ_5418_[Noi2018]屠龙勇士_exgcd+excrt
BZOJ_5418_[Noi2018]屠龙勇士_exgcd+excrt Description www.lydsy.com/JudgeOnline/upload/noi2018day2.pdf 每次用 ...
js 改变对象的引用地址
在业务处理中我们经常会碰到列表中有编辑和新增按钮,为了能够提高代码的公用性,我们经常会使用同一组件处理. 这样会出现一个问题就是编辑的时候直接把对象传过去,直接赋值,引用地址是同一个,所以不管修改了那 ...
Linux wpa_cli 调试方法
记录一下如何使用wpa_cli来进行wifi调试. 1.启动WLAN (1)加载驱动打开wifi的时候会加载驱动,关闭则会卸载wifi驱动.手动调试的时候,先调用insmod/rmmod命令加载/ ...
pytest用例setup和teardown
函数式以下两种: setup_function/teardown_function 每个用例开始和结束调用一次 setup_module/teardown_module setup_modu ...
C# ActiveX 中static变量缓存的问题
最近在忙活一个绘图程序,按照要求需要以ActiveX的方式发布在网站中,这个绘图程序的大概功能就是从数据库获取数据,成图.发布后用户反映,数据变化后,图形没有发生变化,好像有缓存,如果把浏览器全部关闭 ...
TwinCAT3的c++和标准c++(c++11)特性区别
1.vector不能使用花括号初始化 2.不支持cmath,需要使用TcMath.h
HDOJ-2045
不容易系列之(3)—— LELE的RPG难题 Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 65536/32768 K (Java/O ...
Flutter实战视频-移动电商-16.补充_保持页面状态
16.补充_保持页面状态修正一个地方: 设置了item的高度为380 横向列表为380.最终build的高度也增加了50为430. 增加了上面的高度以后,下面那个横线划掉的价格可以显示出来了. 但是 ...
JS实现获取汉字首字母拼音、全拼音及混拼音的方法
本文实例讲述了JS实现获取汉字首字母拼音.全拼音及混拼音的方法.分享给大家供大家参考,具体如下: 这里需要用到一个js获取汉字拼音的插件,可点击此处本站下载. 运行效果如下: 完整示例代码: ? 1 ...
table对象

storm trident的filter和函数

storm trident的filter和函数的更多相关文章

随机推荐

热门专题