Using KafkaBolt to write to a kafka topic
https://community.hortonworks.com/questions/27187/using-kafkabolt-to-write-to-a-kafka-topic.html
--------------------------------------------------------------------------------------------------------------
I'm building a kafka and storm based streaming application based on the usecase:
1. Application produces JSON message to a kafka topic
2. Kafka storm spout ingests the message and does processing
3. Produce output to another kafka topic using a storm bolt
Here are components and versions I'm using:
Storm: 0.10.0
Kafka_2.10: 0.9.0.1
HDP: 2.3.4
storm-kafka: 0.10.0
I accomplished steps 1 and 2 using "storm.kafka.KafkaSpout" that comes with storm-kafka
I'm trying to use "storm.kafka.KafkaBolt" to write processed data to a topic and I couldn't figure out exactly how to do it. Here is the code snippet:
Properties props = new Properties();
props.put("metadata.broker.list", "192.168.56.102:9092");
props.put("request.required.acks", "1");
props.put("serializer.class", "kafka.serializer.StringEncoder"); Config conf = new Config();
conf.put(KafkaBolt.KAFKA_BROKER_PROPERTIES, props); KafkaBolt kafkaBolt = new KafkaBolt().withTopicSelector(new DefaultTopicSelector("OUTBOUND_TOPIC"))
.withTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper()); builder.setBolt("kafka_outbound_bolt", kafkaBolt, 3).shuffleGrouping("process_bolt");
"process_bolt" sends out a tuple that is a serializable domain model object (SubModel.java). I would like to send SubModel.toString() data to the outbound topic. I'm running storm in local cluster mode connected to kafka in a local VM.
- Am I doing anything wrong ?
- How do I use "FieldNameBasedTupleToKafkaMapper"?
Using KafkaBolt to write to a kafka topic的更多相关文章
- Kafka Topic Partition Replica Assignment实现原理及资源隔离方案
本文共分为三个部分: Kafka Topic创建方式 Kafka Topic Partitions Assignment实现原理 Kafka资源隔离方案 1. Kafka Topic创建方式 ...
- Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure is thrown while creating kafka topic
Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentica ...
- 用canal同步binlog到kafka,spark streaming消费kafka topic乱码问题
canal 1.1.1版本之后, 默认支持将canal server接收到的binlog数据直接投递到MQ, 目前默认支持的MQ系统有kafka和RocketMQ. 在投递的时候我们使用的是非压平的消 ...
- Kafka Topic Api
Pom: <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.10 ...
- [Flume][Kafka]Flume 与 Kakfa结合例子(Kakfa 作为flume 的sink 输出到 Kafka topic)
Flume 与 Kakfa结合例子(Kakfa 作为flume 的sink 输出到 Kafka topic) 进行准备工作: $sudo mkdir -p /flume/web_spooldir$su ...
- 手动删除Kafka Topic
一.删除Kafka topic 运行./bin/kafka-topics --delete --zookeeper [zookeeper server] --topic [topic name]: ...
- kafka topic制定规则
kafka topic的制定,我们要考虑的问题有很多,比如生产环境中用几备份.partition数目多少合适.用几台机器支撑数据量,这些方面如何去考量?笔者根据实际的维护经验,写一些思考,希望大家指正 ...
- kafka topic 完全删除
kafka topic 完全删除 1.自动删除脚本(得配置server.properties 中 delete.topic.enable=true) ./kafka-topics.sh --zoo ...
- kafka topic查看删除
1,查看kafka topic列表,使用--list参数 >bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --list __consumer_of ...
随机推荐
- K-means algorithm----PRML读书笔记
The K-means algorithm is based on the use of squared Euclidean distance as the measure of dissimila ...
- 三个命令解决ASTGO服务器重启后各种问题
SSH 命令方式登录到服务器,依次执行下面三个命令. service httpd restart service mysqld restart safe_asterisk 前面两个命令提示无效,尝试从 ...
- Docker For MYSQL 8.0 特别注意修复数据库新的验证方式
从Docker登录MySQL的终端 docker exec -it wordpress-mysql /bin/bash 登录数据库 mysql -u root -p 使用MYSQL数据库 use my ...
- Python 38 初识数据库
数据库 1.什么是mysql,什么是数据库? 文件处理就可以将数据永久存储 问题 1.管理不方便 2.文件操作效率问题 3.一个程序不太可能仅运行在同一台电脑上 提高计算机性能的方式 1.垂直扩展 ...
- BZOJ 3831 单调队列DP
思路: 这好像是我刚学单调性的时候做的题 (我是不会告诉你 我被这题教做人了的...) i-stk[head]>k 删队头 f[stk[tail]]>f[i]||(f[stk[tail]] ...
- POJ 3630 trie树
Phone List Time Limit: 1000MS Memory Limit: 65536K Total Submissions: 26559 Accepted: 8000 Descripti ...
- http接口 两种调用http接口的方法
import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.NameValuePair; ...
- AFN请求后台返回数据为NSInlineData类型的处理
在利用AFN进行数据解析时出现返回数据为 <7b227374 61747573 223a302c 226d6573 73616765 223a22e6 82a8e79a 84e6898b e69 ...
- js-内置对象及相关语法
1:如图(视频截取的) this指的是当前标签的对象. var ary=new Array("mark","jay","leslie"); ...
- AI:IPPR的数学表示-CNN方法
前言: 随着超量类别PR和高精度的需求,人工特征方法局限性凸显出来,固定的特征hash压缩映射因其压缩损失.表现为特定的特征hash方法,在海量的同类数据集上近邻特性变差,而在不同类别的数据上面隔离性 ...