storm集成kafka的应用,从kafka读取,写入kafka
storm集成kafka的应用,从kafka读取,写入kafka
by 小闪电
0前言
storm的主要作用是进行流式的实时计算,对于一直产生的数据流处理是非常迅速的,然而大部分数据并不是均匀的数据流,而是时而多时而少。对于这种情况下进行批处理是不合适的,因此引入了kafka作为消息队列,与storm完美配合,这样可以实现稳定的流式计算。下面是一个简单的示例实现从kafka读取数据,并写入到kafka,以此来掌握storm与kafka之间的交互。
1程序框图
实质上就是storm的kafkaspout作为一个consumer,kafkabolt作为一个producer。
框图如下:
2 pom.xml
建立一个maven项目,将storm,kafka,zookeeper的外部依赖叠加起来。
- <?xml version="1.0" encoding="UTF-8"?>
- <project xmlns="http://maven.apache.org/POM/4.0.0"
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
- <modelVersion>4.0.0</modelVersion>
- <groupId>org.tony</groupId>
- <artifactId>storm-example</artifactId>
- <version>1.0-SNAPSHOT</version>
- <dependencies>
- <dependency>
- <groupId>org.apache.storm</groupId>
- <artifactId>storm-core</artifactId>
- <version>0.9.3</version>
- <!--<scope>provided</scope>-->
- </dependency>
- <dependency>
- <groupId>org.apache.storm</groupId>
- <artifactId>storm-kafka</artifactId>
- <version>0.9.3</version>
- <!--<scope>provided</scope>-->
- </dependency>
- <dependency>
- <groupId>com.google.protobuf</groupId>
- <artifactId>protobuf-java</artifactId>
- <version>2.5.0</version>
- </dependency>
- <!-- storm-kafka模块需要的依赖 -->
- <dependency>
- <groupId>org.apache.curator</groupId>
- <artifactId>curator-framework</artifactId>
- <version>2.5.0</version>
- <exclusions>
- <exclusion>
- <groupId>log4j</groupId>
- <artifactId>log4j</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-log4j12</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <!-- kafka -->
- <dependency>
- <groupId>org.apache.kafka</groupId>
- <artifactId>kafka_2.10</artifactId>
- <version>0.8.1.1</version>
- <exclusions>
- <exclusion>
- <groupId>org.apache.zookeeper</groupId>
- <artifactId>zookeeper</artifactId>
- </exclusion>
- <exclusion>
- <groupId>log4j</groupId>
- <artifactId>log4j</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- </dependencies>
- <repositories>
- <repository>
- <id>central</id>
- <url>http://repo1.maven.org/maven2/</url>
- <snapshots>
- <enabled>false</enabled>
- </snapshots>
- <releases>
- <enabled>true</enabled>
- </releases>
- </repository>
- <repository>
- <id>clojars</id>
- <url>https://clojars.org/repo/</url>
- <snapshots>
- <enabled>true</enabled>
- </snapshots>
- <releases>
- <enabled>true</enabled>
- </releases>
- </repository>
- <repository>
- <id>scala-tools</id>
- <url>http://scala-tools.org/repo-releases</url>
- <snapshots>
- <enabled>true</enabled>
- </snapshots>
- <releases>
- <enabled>true</enabled>
- </releases>
- </repository>
- <repository>
- <id>conjars</id>
- <url>http://conjars.org/repo/</url>
- <snapshots>
- <enabled>true</enabled>
- </snapshots>
- <releases>
- <enabled>true</enabled>
- </releases>
- </repository>
- </repositories>
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
- <version>3.1</version>
- <configuration>
- <source>1.6</source>
- <target>1.6</target>
- <encoding>UTF-8</encoding>
- <showDeprecation>true</showDeprecation>
- <showWarnings>true</showWarnings>
- </configuration>
- </plugin>
- <plugin>
- <artifactId>maven-assembly-plugin</artifactId>
- <configuration>
- <descriptorRefs>
- <descriptorRef>jar-with-dependencies</descriptorRef>
- </descriptorRefs>
- <archive>
- <manifest>
- <mainClass></mainClass>
- </manifest>
- </archive>
- </configuration>
- <executions>
- <execution>
- <id>make-assembly</id>
- <phase>package</phase>
- <goals>
- <goal>single</goal>
- </goals>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
- </project>
3 kafkaspout的消费逻辑,修改MessageScheme类,其中定义了俩个字段,key和message,方便分发到kafkabolt。代码如下
- package com.tony.storm_kafka.util;
- import java.io.UnsupportedEncodingException;
- import java.util.List;
- import backtype.storm.spout.Scheme;
- import backtype.storm.tuple.Fields;
- import backtype.storm.tuple.Values;
- /*
- *author: hi
- *public class MessageScheme{ }
- **/
- public class MessageScheme implements Scheme {
- @Override
- public List<Object> deserialize(byte[] arg0) {
- try{
- String msg = new String(arg0, "UTF-8");
- String msg_0 = "hello";
- return new Values(msg_0,msg);
- }
- catch (UnsupportedEncodingException e) {
- // TODO: handle exception
- e.printStackTrace();
- }
- return null;
- }
- @Override
- public Fields getOutputFields() {
- return new Fields("key","message");
- }
- }
4.编写topology主类,配置kafka,提交topology到storm的代码,其中kafkaspout的zkhost有动态和静态俩种配置,尽量使用动态自寻的方式。
- package org.tony.storm_kafka.common;
- import backtype.storm.Config;
- import backtype.storm.LocalCluster;
- import backtype.storm.StormSubmitter;
- import backtype.storm.generated.AlreadyAliveException;
- import backtype.storm.generated.InvalidTopologyException;
- import backtype.storm.generated.StormTopology;
- import backtype.storm.spout.SchemeAsMultiScheme;
- import backtype.storm.topology.BasicOutputCollector;
- import backtype.storm.topology.OutputFieldsDeclarer;
- import backtype.storm.topology.TopologyBuilder;
- import backtype.storm.topology.base.BaseBasicBolt;
- import backtype.storm.tuple.Tuple;
- import storm.kafka.BrokerHosts;
- import storm.kafka.KafkaSpout;
- import storm.kafka.SpoutConfig;
- import storm.kafka.ZkHosts;
- import storm.kafka.trident.TridentKafkaState;
- import java.util.Arrays;
- import java.util.Properties;
- import org.tony.storm_kafka.bolt.ToKafkaBolt;
- import com.tony.storm_kafka.util.MessageScheme;
- public class KafkaBoltTestTopology {
- //配置kafka spout参数
- public static String kafka_zk_port = null;
- public static String topic = null;
- public static String kafka_zk_rootpath = null;
- public static BrokerHosts brokerHosts;
- public static String spout_name = "spout";
- public static String kafka_consume_from_start = null;
- public static class PrinterBolt extends BaseBasicBolt {
- /**
- *
- */
- private static final long serialVersionUID = 9114512339402566580L;
- // @Override
- public void declareOutputFields(OutputFieldsDeclarer declarer) {
- }
- // @Override
- public void execute(Tuple tuple, BasicOutputCollector collector) {
- System.out.println("-----"+(tuple.getValue(1)).toString());
- }
- }
- public StormTopology buildTopology(){
- //kafkaspout 配置文件
- kafka_consume_from_start = "true";
- kafka_zk_rootpath = "/kafka08";
- String spout_id = spout_name;
- brokerHosts = new ZkHosts("192.168.201.190:2191,192.168.201.191:2191,192.168.201.192:2191", kafka_zk_rootpath+"/brokers");
- kafka_zk_port = "2191";
- SpoutConfig spoutConf = new SpoutConfig(brokerHosts, "testfromkafka", kafka_zk_rootpath, spout_id);
- spoutConf.scheme = new SchemeAsMultiScheme(new MessageScheme());
- spoutConf.zkPort = Integer.parseInt(kafka_zk_port);
- spoutConf.zkRoot = kafka_zk_rootpath;
- spoutConf.zkServers = Arrays.asList(new String[] {"10.9.201.190", "10.9.201.191", "10.9.201.192"});
- //是否從kafka第一條數據開始讀取
- if (kafka_consume_from_start == null) {
- kafka_consume_from_start = "false";
- }
- boolean kafka_consume_frome_start_b = Boolean.valueOf(kafka_consume_from_start);
- if (kafka_consume_frome_start_b != true && kafka_consume_frome_start_b != false) {
- System.out.println("kafka_comsume_from_start must be true or false!");
- }
- System.out.println("kafka_consume_from_start: " + kafka_consume_frome_start_b);
- spoutConf.forceFromStart=kafka_consume_frome_start_b;
- TopologyBuilder builder = new TopologyBuilder();
- builder.setSpout("spout", new KafkaSpout(spoutConf));
- builder.setBolt("forwardToKafka", new ToKafkaBolt<String, String>()).shuffleGrouping("spout");
- return builder.createTopology();
- }
- public static void main(String[] args) {
- KafkaBoltTestTopology kafkaBoltTestTopology = new KafkaBoltTestTopology();
- StormTopology stormTopology = kafkaBoltTestTopology.buildTopology();
- Config conf = new Config();
- //设置kafka producer的配置
- Properties props = new Properties();
- props.put("metadata.broker.list", "192.10.43.150:9092");
- props.put("producer.type","async");
- props.put("request.required.acks", "0"); // 0 ,-1 ,1
- props.put("serializer.class", "kafka.serializer.StringEncoder");
- conf.put(TridentKafkaState.KAFKA_BROKER_PROPERTIES, props);
- conf.put("topic","testTokafka");
- if(args.length > 0){
- // cluster submit.
- try {
- StormSubmitter.submitTopology("kafkaboltTest", conf, stormTopology);
- } catch (AlreadyAliveException e) {
- e.printStackTrace();
- } catch (InvalidTopologyException e) {
- e.printStackTrace();
- }
- }else{
- new LocalCluster().submitTopology("kafkaboltTest", conf, stormTopology);
- }
- }
- }
5 示例结果,testfromkafka topic里面的数据可以通过另外写个类来进行持续的生产。
topic testfromkafka的数据
topic testTokafka的数据
6 补充ToKfakaBolt,集成基础的Bolt类,主要改写Excute,同时加上Ack机制。
- import java.util.Map;
- import java.util.Properties;
- import kafka.javaapi.producer.Producer;
- import kafka.producer.KeyedMessage;
- import kafka.producer.ProducerConfig;
- import org.slf4j.Logger;
- import org.slf4j.LoggerFactory;
- import storm.kafka.bolt.mapper.FieldNameBasedTupleToKafkaMapper;
- import storm.kafka.bolt.mapper.TupleToKafkaMapper;
- import storm.kafka.bolt.selector.KafkaTopicSelector;
- import storm.kafka.bolt.selector.DefaultTopicSelector;
- import backtype.storm.task.OutputCollector;
- import backtype.storm.task.TopologyContext;
- import backtype.storm.topology.OutputFieldsDeclarer;
- import backtype.storm.topology.base.BaseRichBolt;
- import backtype.storm.tuple.Tuple;
- /*
- *author: yue
- *public class ToKafkaBolt{ }
- **/
- public class ToKafkaBolt<K,V> extends BaseRichBolt{
- private static final Logger Log = LoggerFactory.getLogger(ToKafkaBolt.class);
- public static final String TOPIC = "topic";
- public static final String KAFKA_BROKER_PROPERTIES = "kafka.broker.properties";
- private Producer<K, V> producer;
- private OutputCollector collector;
- private TupleToKafkaMapper<K, V> Mapper;
- private KafkaTopicSelector topicselector;
- public ToKafkaBolt<K,V> withTupleToKafkaMapper(TupleToKafkaMapper<K, V> mapper){
- this.Mapper = mapper;
- return this;
- }
- public ToKafkaBolt<K, V> withTopicSelector(KafkaTopicSelector topicSelector){
- this.topicselector = topicSelector;
- return this;
- }
- @Override
- public void prepare(Map stormConf, TopologyContext context,
- OutputCollector collector) {
- if (Mapper == null) {
- this.Mapper = new FieldNameBasedTupleToKafkaMapper<K, V>();
- }
- if (topicselector == null) {
- this.topicselector = new DefaultTopicSelector((String)stormConf.get(TOPIC));
- }
- Map configMap = (Map) stormConf.get(KAFKA_BROKER_PROPERTIES);
- Properties properties = new Properties();
- properties.putAll(configMap);
- ProducerConfig config = new ProducerConfig(properties);
- producer = new Producer<K, V>(config);
- this.collector = collector;
- }
- @Override
- public void execute(Tuple input) {
- // String iString = input.getString(0);
- K key = null;
- V message = null;
- String topic = null;
- try {
- key = Mapper.getKeyFromTuple(input);
- message = Mapper.getMessageFromTuple(input);
- topic = topicselector.getTopic(input);
- if (topic != null) {
- producer.send(new KeyedMessage<K, V>(topic,message));
- }else {
- Log.warn("skipping key = "+key+ ",topic selector returned null.");
- }
- } catch ( Exception e) {
- // TODO: handle exception
- Log.error("Could not send message with key = " + key
- + " and value = " + message + " to topic = " + topic, e);
- }finally{
- collector.ack(input);
- }
- }
- @Override
- public void declareOutputFields(OutputFieldsDeclarer declarer) {
- }
- }
作 者:小闪电
出处:http://www.cnblogs.com/yueyanyu/
本文版权归作者和博客园共有,欢迎转载、交流,但未经作者同意必须保留此段声明,且在文章页面明显位置给出原文链接。如果觉得本文对您有益,欢迎点赞、欢迎探讨。本博客来源于互联网的资源,若侵犯到您的权利,请联系博主予以删除。
storm集成kafka的应用,从kafka读取,写入kafka的更多相关文章
- Storm集成Kafka应用的开发
我们知道storm的作用主要是进行流式计算,对于源源不断的均匀数据流流入处理是非常有效的,而现实生活中大部分场景并不是均匀的数据流,而是时而多时而少的数据流入,这种情况下显然用批量处理是不合适的,如果 ...
- Storm集成Kafka编程模型
原创文章,转载请注明: 转载自http://www.cnblogs.com/tovin/p/3974417.html 本文主要介绍如何在Storm编程实现与Kafka的集成 一.实现模型 数据流程: ...
- 5、Storm集成Kafka
1.pom文件依赖 <!--storm相关jar --> <dependency> <groupId>org.apache.storm</groupId> ...
- Flume 读取RabbitMq消息队列消息,并将消息写入kafka
首先是关于flume的基础介绍 组件名称 功能介绍 Agent代理 使用JVM 运行Flume.每台机器运行一个agent,但是可以在一个agent中包含多个sources和sinks. Client ...
- Spark(二十一)【SparkSQL读取Kudu,写入Kafka】
目录 SparkSQL读取Kudu,写出到Kafka 1. pom.xml 依赖 2.将KafkaProducer利用lazy val的方式进行包装, 创建KafkaSink 3.利用广播变量,将Ka ...
- Springboot集成mybatis(mysql),mail,mongodb,cassandra,scheduler,redis,kafka,shiro,websocket
https://blog.csdn.net/a123demi/article/details/78234023 : Springboot集成mybatis(mysql),mail,mongodb,c ...
- Kafka设计解析(十八)Kafka与Flink集成
转载自 huxihx,原文链接 Kafka与Flink集成 Apache Flink是新一代的分布式流式数据处理框架,它统一的处理引擎既可以处理批数据(batch data)也可以处理流式数据(str ...
- spark读取 kafka nginx网站日志消息 并写入HDFS中(转)
原文链接:spark读取 kafka nginx网站日志消息 并写入HDFS中 spark 版本为1.0 kafka 版本为0.8 首先来看看kafka的架构图 详细了解请参考官方 我这边有三台机器用 ...
- Mysql增量写入Hdfs(一) --将Mysql数据写入Kafka Topic
一. 概述 在大数据的静态数据处理中,目前普遍采用的是用Spark+Hdfs(Hive/Hbase)的技术架构来对数据进行处理. 但有时候有其他的需求,需要从其他不同数据源不间断得采集数据,然后存储到 ...
随机推荐
- Matlab forward Euler demo
% forward Euler demo % take two steps in the solution of % dy/dt = y, y(0) = 1 % exact solution is y ...
- Web应用架构入门之11个基本要素
译者: 读完这篇博客,你就可以回答一个经典的面试题:当你访问Google时,到底发生了什么? 原文:Web Architecture 101 译者:Fundebug 为了保证可读性,本文采用意译而非直 ...
- jQuery中是事件绑定方式--on、bind、live、delegate
概述:jQuery是我们最常用的js库,对于事件的绑定也是有很多种,on.one.live.bind.delegate等等,接下来我们逐一来进行讲解. 本片文章中事件所带的为版本号,例:v1.7+为1 ...
- 微信小程序地图报错——ret is not defined
刚刚在使用微信的map做地图时候 发现如下报错: 后来找了一会发现错误为经纬度写反了导致经纬度超出了范围 正确取值范围: latitude 纬度 浮点数,范围 -90 ~ 90 longitud ...
- mysql 之库, 表的简易操作
一. 库的操作 1.创建数据库 创建数据库: create database 库名 charset utf8; charset uft8 可选项 1.2 数据库命名规范: 可以由字母.数字.下划 ...
- SWFUpload多文件上传使用指南
SWFUpload是一个flash和js相结合而成的文件上传插件,其功能非常强大.以前在项目中用过几次,但它的配置参数太多了,用过后就忘记怎么用了,到以后要用时又得到官网上看它的文档,真是太烦了.所以 ...
- EasyUI tabs指定要显示的tab
<div id="DivBox" class="easyui-tabs" style="width: 100%; height: 100%;& ...
- jQ 移动端返回顶部代码整理
//返回顶部 $('#btn-scroll').on('touchend',function(){ var T = $(window).scrollTop(); var t = setInterval ...
- Django admin 的模仿流程
- 使用hint优化Oracle的运行计划 以及 SQL Tune Advisor的使用
背景: 某表忽然出现查询很缓慢的情况.cost 100+ 秒以上:严重影响生产. 原SQL: explain plan for select * from ( select ID id,RET_NO ...