首先安装 Confluent Platform

Quick Start for Confluent Platform (Local install)

Use this quick start to get up and running with Confluent Platform and its main components in a development environment. This quick start uses Confluent Control Center included in Confluent Platform for topic management and event stream processing using ksqlDB.

In this quick start, you create Apache Kafka topics, use Kafka Connect to generate mock data to those topics, and create ksqlDB streaming queries on those topics. You then go to Control Center to monitor and analyze the streaming queries.

See also

You can also run an automated version of this quick start designed for Confluent Platform local installs.

Step 1: Download and Start Confluent Platform

  1. Go to the downloads page.

  2. Scroll to the Download Confluent Platform section and provide the following:

    • Email: Your email address
    • Format: deb, rpm, tar, or zip
  3. Click DOWNLOAD FREE.

Tip

You have installation options for Ansible, Docker, and Kubernetes. Also, you can download a previous version from Previous Versions.

  1. Decompress the file. You should have the directories, such as bin and etc.

  2. Set the environment variable for the Confluent Platform directory.

    export CONFLUENT_HOME=<path-to-confluent>
  3. Add the Confluent Platform bin directory to your PATH.

    export PATH=$PATH:$CONFLUENT_HOME/bin
  4. Install the Kafka Connect Datagen source connector using the Confluent Hub client. This connector generates mock data for demonstration purposes and is not suitable for production. Confluent Hub is an online library of pre-packaged and ready-to-install extensions or add-ons for Confluent Platform and Kafka.

    confluent-hub install \
    --no-prompt confluentinc/kafka-connect-datagen:latest
  5. Start Confluent Platform using the Confluent CLI confluent local services start command. This command starts all of the Confluent Platform components, including Kafka, ZooKeeper, Schema Registry, HTTP REST Proxy for Kafka, Kafka Connect, ksqlDB, and Control Center.

    Important

    The confluent local commands are intended for a single-node development environment and are not suitable for a production environment. The data that are produced are transient and are intended to be temporary. For production-ready workflows, see Install and Upgrade Confluent Platform.

    confluent local services start

    Your output should resemble:

    Starting Zookeeper
    Zookeeper is [UP]
    Starting Kafka
    Kafka is [UP]
    Starting Schema Registry
    Schema Registry is [UP]
    Starting Kafka REST
    Kafka REST is [UP]
    Starting Connect
    Connect is [UP]
    Starting KSQL Server
    KSQL Server is [UP]
    Starting Control Center
    Control Center is [UP]

Step 2: Create Kafka Topics

In this step, you create Kafka topics using Confluent Control Center. Confluent Control Center provides the functionality for building and monitoring production data pipelines and event streaming applications.

  1. Navigate to the Control Center web interface at http://localhost:9021.

    If you installed Confluent Platform on a different host, replace localhost with the host name in the address.

    It may take a minute or two for Control Center to come online.

Note

Control Center won’t connect to ksqlDB if Control Center isn’t open and running in a localhost browser session.

  1. Click the controlcenter.cluster tile.

  2. In the navigation bar, click Topics to open the topics list, and then click Add a topic.

  3. In the Topic name field, specify pageviews and click Create with defaults.

    Note that topic names are case-sensitive.

  4. In the navigation bar, click Topics to open the topics list, and then click Add a topic.

  5. In the Topic name field, specify users and click Create with defaults.

Step 3: Install a Kafka Connector and Generate Sample Data

In this step, you use Kafka Connect to run a demo source connector called kafka-connect-datagen that creates sample data for the Kafka topics pageviews and users.

Tip

The Kafka Connect Datagen connector was installed manually in Step 1: Download and Start Confluent Platform. If you encounter issues locating the Datagen Connector, refer to the Issue: Cannot locate the Datagen connector in the Troubleshooting section.

  1. Run the first instance of the Kafka Connect Datagen connector to produce Kafka data to the pageviews topic in AVRO format.

    1. In the navigation bar, click Connect.

    2. Click the connect-default cluster in the Connect Clusters list.

    3. Click Add connector.

    4. Select the DatagenConnector tile.

      Tip

      To narrow displayed connectors, click Filter by category and click Sources.

    5. In the Name field, enter datagen-pageviews as the name of the connector.

    6. Enter the following configuration values:

      • Key converter class: org.apache.kafka.connect.storage.StringConverter.
      • kafka.topic: pageviews.
      • max.interval: 100.
      • quickstart: pageviews.
    7. Click Next.

    8. Review the connector configuration and click Launch.

  2. Run the second instance of the Kafka Connect Datagen connector to produce Kafka data to the users topic in AVRO format.

    1. Click Add connector.

    2. Select the DatagenConnector tile.

      Tip

      To narrow displayed connectors, click Filter by category and click Sources.

    3. In the Name field, enter datagen-users as the name of the connector.

    4. Enter the following configuration values:

      • Key converter class: org.apache.kafka.connect.storage.StringConverter
      • kafka.topic: users
      • max.interval: 1000
      • quickstart: users
    5. Click Next.

    6. Review the connector configuration and click Launch.

Step 4: Create and Write to a Stream and Table using ksqlDB

Tip

You can also run these commands using the ksqlDB CLI from your terminal with this command: <path-to-confluent>/bin/ksql http://localhost:8088.

Create Streams and Tables

In this step, you use ksqlDB to create a stream for the pageviews topic and a table for the users topic.

  1. In the navigation bar, click ksqlDB.

  2. Select the ksqlDB application.

  3. Copy the following code into the editor window and click Run query to create the pageviews stream. Stream names are not case-sensitive.

    CREATE STREAM pageviews WITH (KAFKA_TOPIC='pageviews', VALUE_FORMAT='AVRO');
  4. Copy the following code into the editor window and click Run query to create the users table. Table names are not case-sensitive.

    CREATE TABLE users (id VARCHAR PRIMARY KEY)
    WITH (KAFKA_TOPIC='users', VALUE_FORMAT='AVRO');

Write Queries

In this step, you create ksqlDB queries against the stream and the table you created above.

  1. In the Editor tab, click Add query properties to add a custom query property.

  2. Set the auto.offset.reset parameter to Earliest.

    The setting instructs ksqlDB queries to read all available topic data from the beginning. This configuration is used for each subsequent query. For more information, see the ksqlDB Configuration Parameter Reference.

  3. Create the following queries.

    1. Click Stop to stop the current running query.

    2. Create a non-persistent query that returns data from a stream with the results limited to a maximum of three rows:

      Enter the following query in the editor:

      SELECT pageid FROM pageviews EMIT CHANGES LIMIT 3;
    3. Click Run query. Your output should resemble:

      Click the Card view or Table view icon to change the output layout.

    4. Create a persistent query (as a stream) that filters the PAGEVIEWS stream for female users. The results from this query are written to the Kafka PAGEVIEWS_FEMALE topic:

      Enter the following query in the editor:

      CREATE STREAM pageviews_female
      AS SELECT users.id AS userid, pageid, regionid
      FROM pageviews LEFT JOIN users ON pageviews.userid = users.id
      WHERE gender = 'FEMALE'
      EMIT CHANGES;
    5. Click Run query. Your output should resemble:

    6. Create a persistent query where REGIONID ends with 8 or 9. Results from this query are written to the Kafka topic named pageviews_enriched_r8_r9 as explicitly specified in the query:

      Enter the following query in the editor:

      CREATE STREAM pageviews_female_like_89
      WITH (KAFKA_TOPIC='pageviews_enriched_r8_r9', VALUE_FORMAT='AVRO')
      AS SELECT * FROM pageviews_female
      WHERE regionid LIKE '%_8' OR regionid LIKE '%_9'
      EMIT CHANGES;
    7. Click Run query. Your output should resemble:

    8. Create a persistent query that counts the PAGEVIEWS for each REGION and GENDER combination in a tumbling window of 30 seconds when the count is greater than 1. Because the procedure is grouping and counting, the result is now a table, rather than a stream. Results from this query are written to a Kafka topic called PAGEVIEWS_REGIONS:

      Enter the following query in the editor:

      CREATE TABLE pageviews_regions WITH (KEY_FORMAT='JSON')
      AS SELECT gender, regionid, COUNT(*) AS numusers
      FROM pageviews LEFT JOIN users ON pageviews.userid = users.id
      WINDOW TUMBLING (SIZE 30 SECOND)
      GROUP BY gender, regionid
      HAVING COUNT(*) > 1
      EMIT CHANGES;
    9. Click Run query. Your output should resemble:

    10. Click the Persistent queries tab. You should see the following persisted queries:

      • PAGEVIEWS_FEMALE
      • PAGEVIEWS_FEMALE_LIKE_89
      • PAGEVIEWS_REGIONS
    11. Click the Editor tab. The All available streams and tables pane shows all of the streams and tables that you can access.

    12. In the All available streams and tables section, click KSQL_PROCESSING_LOG to view the stream’s schema, including nested data structures.

Run Queries

In this step, you run the ksqlDB queries you save as streams and tables above in the previous section.

  1. In the Streams tab, select the PAGEVIEWS_FEMALE stream.

  2. Click Query stream.

    The editor opens, and streaming output of the query displays.

  3. Click Stop to stop the output generation.

  4. In the Tables tab, select PAGEVIEWS_REGIONS table.

  5. Click Query table.

    The editor opens, and streaming output of the query displays.

  6. Click Stop to stop the output generation.

Step 5: Monitor Consumer Lag

  1. In the navigation bar, click Consumers to view the consumers created by ksqlDB.

  2. Click the consumer group ID to view details for the _confluent-ksql-default_query_CSAS_PAGEVIEWS_FEMALE_5 consumer group.

    From the page, you can see the consumer lag and consumption values for your streaming query.

For more information, see the Control Center Consumers documentation.

Step 6: Stop Confluent Platform

When you are done working with the local install, you can stop Confluent Platform.

  1. Stop Confluent Platform using the Confluent CLI confluent local services connect stop command.

    confluent local services stop
  2. Destroy the data in the Confluent Platform instance with the confluent local destroy command.

    confluent local destroy

然后使用docker-compose运行Kafka

Kafka Commands Primer

Once you have Confluent Platform running, an intuitive next step is try out some basic Kafka commands to create topics and work with producers and consumers. This should help orient Kafka newbies and pros alike that all those familiar Kafka tools are readily available in Confluent Platform, and work the same way. These provide a means of testing and working with basic functionality, as well as configuring and monitoring deployments. The commands surface a subset of the APIs available to you.

Confluent Platform ships with Kafka commands and utilities in $CONFLUENT_HOME/etc/kafka/bin. This bin/ directory includes both Confluent proprietary and open source Kafka utilities.

A few things to note:

  • With Confluent Platform installed and running on your system, you can run Kafka commands from anywhere; for example, from your $HOME (~/) directory. You do not have to run these from within $CONFLUENT_HOME.
  • A full list is provided in CLI Tools for Confluent Platform. Those in the list that begin with kafka- are the Kafka open source command utilities, also covered in various sections of the Apache Kafka documentation.
  • Comprehensive command line help is available by typing any of the commands with no arguments; for example, kafka-topics or kafka-producer-perf-test.

To help get you started, the sections below provide examples for some of the most fundamental and widely-used commands.

Create, list and describe topics

You can use kafka-topics for operations on topics (create, list, describe, alter, delete, and so forth).

In a command window, run the following commands to experiment with topics.

  1. Create three topics, cool-topic, warm-topic, hot-topic.

    kafka-topics --create --topic cool-topic --bootstrap-server localhost:9092
    kafka-topics --create --topic warm-topic --bootstrap-server localhost:9092
    kafka-topics --create --topic hot-topic --partitions 2 --replication-factor 2 --bootstrap-server localhost:9092
  2. List all topics.

    kafka-topics --list --bootstrap-server localhost:9092

    Tip

System topics are prefaced by an underscore in the output. The topics you created are listed at the end.

  1. Describe a topic.

    This shows partitions, replication factor, and in-sync replicas for the topic.

    kafka-topics --describe --topic cool-topic --bootstrap-server localhost:9092

    Your output should resemble the following:

    Topic: cool-topic PartitionCount: 1       ReplicationFactor: 1    Configs: segment.bytes=1073741824
    Topic: cool-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0 Offline:

    Tip

If you run kafka-topics --describe with no specified topic, you get a detailed description of every topic on the cluster (system and user topics).

  1. Describe another topic, using one of the other brokers in the cluster as the bootstrap server.

    kafka-topics --describe --topic hot-topic --bootstrap-server localhost:9094

    Here is that example output:

    Topic: hot-topic  PartitionCount: 2       ReplicationFactor: 2    Configs: segment.bytes=1073741824
    Topic: hot-topic Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0 Offline:
    Topic: hot-topic Partition: 1 Leader: 0 Replicas: 0,2 Isr: 0,2 Offline:

    You can connect to any of the brokers in the cluster to run these commands because they all have the same data!

  2. Alter a topic’s cofiguration.

    For this example, change the partition count on hot-topic from 2 to 9.

    kafka-topics --alter --topic hot-topic --partitions 9 --bootstrap-server localhost:9092

    Tip

    Dynamic topic modification is inherently limited by the current configurations. For example, you cannot decrease the number of partitions or modify the replication factor for a topic, as that would require partition reassignment.

  3. Rerun --describe on the same topic.

    kafka-topics --describe --topic hot-topic --bootstrap-server localhost:9092

    Here is that example output, and verify that the partition count is updated to 9:

    Topic: hot-topic  PartitionCount: 9       ReplicationFactor: 2    Configs: segment.bytes=1073741824
    Topic: hot-topic Partition: 0 Leader: 2 Replicas: 2,1 Isr: 2,1 Offline:
    Topic: hot-topic Partition: 1 Leader: 1 Replicas: 1,0 Isr: 1,0 Offline:
    Topic: hot-topic Partition: 2 Leader: 1 Replicas: 1,2 Isr: 1,2 Offline:
    Topic: hot-topic Partition: 3 Leader: 2 Replicas: 2,1 Isr: 2,1 Offline:
    Topic: hot-topic Partition: 4 Leader: 0 Replicas: 0,2 Isr: 0,2 Offline:
    Topic: hot-topic Partition: 5 Leader: 1 Replicas: 1,0 Isr: 1,0 Offline:
    Topic: hot-topic Partition: 6 Leader: 2 Replicas: 2,0 Isr: 2,0 Offline:
    Topic: hot-topic Partition: 7 Leader: 0 Replicas: 0,1 Isr: 0,1 Offline:
    Topic: hot-topic Partition: 8 Leader: 1 Replicas: 1,2 Isr: 1,2 Offline:
  4. Delete a topic.

    kafka-topics --delete --topic warm-topic --bootstrap-server localhost:9092
  5. List all topics.

    kafka-topics --list --bootstrap-server localhost:9092

Run producers and consumers to send and read messages

The command utilities kafka-console-producer and kafka-console-consumer allow you to manually produce messages to and consume from a topic.

  1. Open two new command windows, one for a producer, and the other for a consumer.

  2. Run a producer to produce to cool-topic.

    kafka-console-producer --topic cool-topic --bootstrap-server localhost:9092
  3. Send some messages.

    Type your messages at the prompt (>), and hit Return after each one.

    Your command window will resemble the following:

    $ kafka-console-producer --broker-list localhost:9092 --topic cool-topic
    >hi cool topic
    >did you get this message?
    >first
    >second
    >third
    >yes! I love you cool topic
    >

    Tip

    You can use the --broker-list flag in place of --bootstrap-server for the producer, typically used to send data to specific brokers; shown here as an example.

  4. In the other command window, run a consumer to read messages from cool-topic. Specify that you want to start consuming from the beginning, as shown.

    kafka-console-consumer --topic cool-topic --from-beginning --bootstrap-server localhost:9092

    Your output will resemble the following:

    $ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic cool-topic
    hi cool topic on origin cluster
    is this getting to your replica?
    first
    second
    third
    yes! I love you cool topic
  5. When you want to stop the producer and consumer, type Ctl-C in their respective command windows.

    Tip

    You may want to leave at least the producer running for now, in case you want to send more messages when we revisit topics on the Control Center.

Produce auto-generated message data to topics

You can use kafka-consumer-perf-test in its own command window to generate test data to topics.

  • For example, open a new command window and type the following command to send data to hot-topic, with the specified throughput and record size.

    kafka-producer-perf-test \
    --producer-props bootstrap.servers=localhost:9092 \
    --topic hot-topic \
    --record-size 1000 \
    --throughput 1000 \
    --num-records 3600000

    The command provides status output on messages sent, as shown:

    4999 records sent, 999.8 records/sec (0.95 MB/sec), 1.1 ms avg latency, 240.0 ms max latency.
    5003 records sent, 1000.2 records/sec (0.95 MB/sec), 0.5 ms avg latency, 4.0 ms max latency.
    5003 records sent, 1000.2 records/sec (0.95 MB/sec), 0.6 ms avg latency, 5.0 ms max latency.
    5001 records sent, 1000.2 records/sec (0.95 MB/sec), 0.3 ms avg latency, 3.0 ms max latency.
    5001 records sent, 1000.0 records/sec (0.95 MB/sec), 0.3 ms avg latency, 4.0 ms max latency.
    5000 records sent, 1000.0 records/sec (0.95 MB/sec), 0.8 ms avg latency, 24.0 ms max latency.
    5001 records sent, 1000.2 records/sec (0.95 MB/sec), 0.6 ms avg latency, 3.0 ms max latency.
    ...
  • Open a new command window to consume the messages from hot-topic as they are sent (not from the beginning).

    kafka-console-consumer --topic hot-topic --bootstrap-server localhost:9092

    Type Ctl-C to stop the consumer.

    Tip

    You may want to leave the producer running for a moment, as you are about to revisit Topics on the Control Center.

To learn more, check out Benchmark Commands, Let’s Load test, Kafka!, and How to do Performance testing of Kafka Cluster

参考文档

Confluent Quick Start

Confluent Community Docker Image for Apache Kafka

Dockers启动Kafka的更多相关文章

  1. 启动kafka时报scala相关错误:java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc()

    1.报错现象: 启动kafka的时候启动失败,并且会报告下面的错误: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/ ...

  2. 安装启动kafka

    vim kafka/config/server.properties #确保唯一 broker.id=0 #允许删除主题 delete.topic.enable=true # 指定数据文件所在目录 l ...

  3. 首次启动Kafka报Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c0000000, 1073741824, 0) failed; error='Cannot allocate memory' (errno=12)

    首次启动Kafka报错如下: 原因:内存不足,查看启动配置 调小一些:

  4. 启动Kafka

    启动zookeeper 使用命令查看zookeeper是否启动成功: 启动kafka Brokerr 使用命令查看kafka Broker是否启动更成功 在kafka中创建topic 'test' b ...

  5. Mac单机模式安装启动Kafka

    1.下载kafka,网址: https://www.apache.org/dyn/closer.cgi?path=/kafka/2.0.0/kafka_2.12-2.0.0.tgz 2.移动tar包到 ...

  6. 启动kafka集群,关闭kafka集群脚本

    启动kafka集群,关闭kafka集群脚本 在$KAFKA_HOME/bin下新建如下脚本文件 start-kafka.sh #!/bin/bash BROKERS="mini41 mini ...

  7. 启动kafka报错

    启动kafka时 报错: kafka-console-consumer.sh --from-beginning --zookeeper node01:8121,node02:8121,node03:8 ...

  8. macOS启动Kafka

    目录 kafka目录结构 先启动zookeeper 后启动kafka 创建topic 创建一个生产者 创建一个消费者 kafka目录结构 # kafka安装目录 /usr/local/Cellar/k ...

  9. 【kafka】一键启动kafka脚本

    3.1 创建文件cd bin 跳转到bin文件夹里touch  start-kafka-cluster.sh --新建一键启动文件touch  stop-kafka-cluster.sh --新建一键 ...

随机推荐

  1. uni-app 提示 v-for 暂不支持循环数据

    这个问题由于目前博主只在APP端遇到过,解决办法是把v-for key值全部取循环的索引,如果解决了你的问题请给博主点个赞 <block v-for="(item,index) in ...

  2. Gson gson = new GsonBuilder().setDateFormat("yyyy-MM-dd HH:mm:ss").create()

    在Java中处理JSON格式的数据时,Google  Gson 是个不错的选择,用起来挺方便的,也有一定灵活性.我现在工作中在参与的两个项目里都有用它.不过它在处理Date格式时有个小陷阱,在不同环境 ...

  3. Django笔记&教程 3-3 模板常用语法

    Django 自学笔记兼学习教程第3章第3节--模板常用语法 点击查看教程总目录 本文主要参考:https://docs.djangoproject.com/en/2.2/ref/templates/ ...

  4. java 邮件 接收与发送

    ... package com.e6soft; import java.io.BufferedReader; import java.io.FileOutputStream; import java. ...

  5. CTF入门学习2->Web基础了解

    Web安全基础 00 Web介绍 00-00 Web本意是网,这里多指万维网(World Wide Web),是由许多互相连接的超文本系统组成的,通过互联网访问. Web是非常广泛的互联网应用,每天都 ...

  6. [bzoj1741]穿越小行星群

    将每一行/每一列作为一个点,对于一个障碍(x,y),要么第x行和第y列的状态(是否攻击)只需要有一个就可以了,将第x行和第y列连边,就是二分图的最小点覆盖=最大匹配数. 1 #include<b ...

  7. [loj3343]超现实树

    定义1:两棵树中的$x$和$y$对应当且仅当$x$到根的链与$y$到根的链同构 定义2:$x$和$y$的儿子状态相同当且仅当$x$与儿子所构成的树与$y$与儿子所构成的树同构 根据题中所给的定义,有以 ...

  8. 【Tool】Node.js 安装

    Node.js 安装 2019-07-29  14:56:14  by冲冲 1. 说明 电脑使用node.js存在两种方式: ① 第一种,下载(.exe)到本地电脑,不需要安装,下载后点击 node. ...

  9. 从零开始学Kotlin第一课

    Kotlin的方法: 一个简单的计算器: fun main(args:Array<String>){ //主函数main方法 var a=8; var b=9; println(plus( ...

  10. 洛谷 P4564 [CTSC2018]假面(期望+dp)

    题目传送门 题意: 有 \(n\) 个怪物,第 \(i\) 个怪物初始血量为 \(m_i\).有 \(Q\) 次操作: 0 x u v,有 \(p=\frac{u}{v}\) 的概率令 \(m_x\) ...