Kafka笔记——技术点汇总

Table of contents

• Table of contents
• Overview
  • Introduction
  • Use cases
• Manual setup
  • Assumption
  • Configuration
  • Startup & test
• Principle
  • Topic
  • Distribution
  • Producer
  • Consumer
• Operation
  • Adding topics
  • Modifying topics
  • Removing a topic
  • Graceful shutdown
  • Balancing leadership
  • Checking consumer position
  • Listing consumer groups
  • Infrequently used operations
• API
  • Overview
  • Producer API
  • Consumer API
• Performance tunning

Overview

Introduction

1. Apache kafka is distributed streaming platform.
2. Capabilities of streaming platform:
   • It lets you publish and subscribe to streams of records. It is similar to a message queue or enterprise messaging system.
   • It lets you store streams of records in a fault-tolerant way.
   • It lets you process streams of records as they occur.

Use cases

1. Messaging
   • In comparison to most messaging systems(e.g. ActiveMQ, RocketMQ) Kafka has better throughput, built-in partitioning, replication, and fault-tolerance which makes it a good solution for large scale message processing applications.
2. Website activity tracking
   • The original use case for Kafka was to be able to rebuild a user activity tracking pipeline as a set of real-time publish-subscribe feeds.
   • Site activity (page views, searches, or other actions users may take) is published to central topics with one topic per activity type. These feeds are available for subscription for a range of use cases including real-time processing, real-time monitoring, and loading into Hadoop or offline data warehousing systems for offline processing and reporting.
3. Metrics
   • Kafka is often used for operational monitoring data. This involves aggregating statistics from distributed applications to produce centralized feeds of operational data.
4. Log aggregation
   • Log aggregation typically collects physical log files off servers and puts them in a central place (a file server or HDFS perhaps) for processing. Kafka abstracts away the details of files and gives a cleaner abstraction of log or event data as a stream of messages. This allows for lower-latency processing and easier support for multiple data sources and distributed data consumption.
   • In comparison to log-centric systems like Scribe or Flume, Kafka offers equally good performance, stronger durability guarantees due to replication, and much lower end-to-end latency.
5. Stream processing
   • Many users of Kafka process data in processing pipelines consisting of multiple stages, where raw input data is consumed from Kafka topics and then aggregated, enriched, or otherwise transformed into new topics for further consumption or follow-up processing. Apart from Kafka Streams, alternative open source stream processing tools include Apache Storm and Apache Samza.
6. Event sourcing
   • Event sourcing is a style of application design where state changes are logged as a time-ordered sequence of records.
7. Commit log
   • Kafka can serve as a kind of external commit-log for a distributed system. The log helps replicate data between nodes and acts as a re-syncing mechanism for failed nodes to restore their data. The log compaction feature in Kafka helps support this usage. In this usage Kafka is similar to Apache BookKeeper project.

Manual setup

Assumption

We assume that JDK and Zookeeper have been installed.

Configuration

1. Configure on one server.

tar zxvf kafka_2.10-0.10.2.1.tgz -C /opt/app
cd /opt/app/kafka_2.10-0.10.2.1
vi config/server.properties

broker.id=1 # Unique
log.dirs=/opt/data/kafka.logs # Data directory
default.replication.factor=3
zookeeper.connect=centos1:2181,centos2:2181/kafka # Specify chroot at the end, default chroot is /
# Configurations for product server
num.partitions=8
delete.topic.enable=false
auto.create.topics.enable=false
log.retention.hours=168
min.insync.replicas=2
queued.max.requests=500

1. Copy Kafka directory to the other servers.

scp -r /opt/app/kafka_2.10-0.10.2.1 hadoop@centos2:/opt/app
scp -r /opt/app/kafka_2.10-0.10.2.1 hadoop@centos3:/opt/app

1. Remember to configure broker.id on the other servers.

vi config/server.properties

Startup & test

1. Startup daemon on all servers.

bin/kafka-server-start.sh -daemon config/server.properties
jps

Kafka

1. Test.

bin/kafka-topics.sh --create --zookeeper centos1:2181,centos2:2181,centos3:2181/kafka --replication-factor 2 --partitions 2 --topic test-topic
bin/kafka-topics.sh --list --zookeeper centos1:2181,centos2:2181,centos3:2181/kafka
bin/kafka-topics.sh --describe --zookeeper centos1:2181,centos2:2181,centos3:2181/kafka --topic test-topic
bin/kafka-console-producer.sh --broker-list centos1:9092,centos2:9092,centos3:9092 --topic test-topic
bin/kafka-console-consumer.sh --bootstrap-server centos1:9092,centos2:9092,centos3:9092 --topic test-topic --from-beginning

1. Shutdown daemon on all servers.

bin/kafka-server-stop.sh

Principle

Topic

1. A topic is a category or feed name to which records are published.
2. Partition
   • Each partition is an ordered, immutable sequence of records that is continually appended to—a structured commit log. The records in the partitions are each assigned a sequential id number called the offset that uniquely identifies each record within the partition.
   • The partitions in the log serve several purposes. First, they allow the log to scale beyond a size that will fit on a single server. Each individual partition must fit on the servers that host it, but a topic may have many partitions so it can handle an arbitrary amount of data. Second they act as the unit of parallelism—more on that in a bit.
     
     
3. Storage
   • The Kafka cluster retains all published records—whether or not they have been consumed—using a configurable retention period. For example, if the retention policy is set to two days, then for the two days after a record is published, it is available for consumption, after which it will be discarded to free up space. Kafka's performance is effectively constant with respect to data size so storing data for a long time is not a problem.
4. Offset control
   • In fact, the only metadata retained on a per-consumer basis is the offset or position of that consumer in the log. This offset is controlled by the consumer: normally a consumer will advance its offset linearly as it reads records, but, in fact, since the position is controlled by the consumer it can consume records in any order it likes. For example a consumer can reset to an older offset to reprocess data from the past or skip ahead to the most recent record and start consuming from "now".

Distribution

Each partition has one server which acts as the "leader" and zero or more servers which act as "followers". The leader handles all read and write requests for the partition while the followers passively replicate the leader. If the leader fails, one of the followers will automatically become the new leader. Each server acts as a leader for some of its partitions and a follower for others so load is well balanced within the cluster.

Producer

The producer is responsible for choosing which record to assign to which partition within the topic. This can be done in a round-robin fashion simply to balance load or it can be done according to some semantic partition function (say based on some key in the record).

Consumer

1. Consumers label themselves with a consumer group name, and each record published to a topic is delivered to one consumer instance within each subscribing consumer group. Consumer instances can be in separate processes or on separate machines.
2. If all the consumer instances have the same consumer group, then the records will effectively be load balanced over the consumer instances.
3. If all the consumer instances have different consumer groups, then each record will be broadcast to all the consumer processes.
   
   
4. This process of maintaining membership in the group is handled by the Kafka protocol dynamically. If new instances join the group they will take over some partitions from other members of the group; if an instance dies, its partitions will be distributed to the remaining instances.
5. Kafka only provides a total order over records within a partition, not between different partitions in a topic. Per-partition ordering combined with the ability to partition data by key is sufficient for most applications. However, if you require a total order over records this can be achieved with a topic that has only one partition, though this will mean only one consumer process per consumer group.

Operation

Adding topics

1. Command:

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --create --topic my_topic_name --partitions 20 --replication-factor 3 --config x=y

1. You have the option of either adding topics manually or having them be created automatically when data is first published to a non-existent topic.
2. The replication factor controls how many servers will replicate each message that is written. We recommend you use a replication factor of 2 or 3 so that you can transparently bounce machines without interrupting data consumption.
3. Each sharded partition log is placed into its own folder under the Kafka log directory. The name of such folders consists of the topic name, appended by a dash (-) and the partition id.

Modifying topics

1. Commands:
   • Add partitions

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name --partitions 40

* Add configs

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name --config x=y

* Remove a config

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name --delete-config x

1. Adding partitions doesn't change the partitioning of existing data so this may disturb consumers if they rely on that partition. That is if data is partitioned by hash(key) % number_of_partitions then this partitioning will potentially be shuffled by adding partitions but Kafka will not attempt to automatically redistribute data in any way.
2. Kafka does not currently support reducing the number of partitions for a topic.

Removing a topic

1. Command

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic my_topic_name

1. Topic deletion option is disabled by default. To enable it set the server configdelete.

delete.topic.enable=true

Graceful shutdown

1. The Kafka cluster will automatically detect any broker shutdown or failure and elect new leaders for the partitions on that machine.
2. Note that controlled shutdown will only succeed if all the partitions hosted on the broker have replicas (i.e. the replication factor is greater than 1 and at least one of these replicas is alive). This is generally what you want since shutting down the last replica would make that topic partition unavailable.
3. Controlled leadership migration requires using a special setting:

controlled.shutdown.enable=true

Balancing leadership

1. Whenever a broker stops or crashes leadership for that broker's partitions transfers to other replicas. To avoid this, Kafka has a notion of preferred replicas. If the list of replicas for a partition is 1,5,9 then node 1 is preferred as the leader to either node 5 or 9 because it is earlier in the replica list.
2. You can have the Kafka cluster try to restore leadership to the restored replicas by running the command:

bin/kafka-preferred-replica-election.sh --zookeeper zk_host:port/chroot

1. Configure Kafka to do this automatically:

auto.leader.rebalance.enable=true

Checking consumer position

1. Before 0.9.0.0

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper localhost:2181 --group test

1. Since 0.9.0.0

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group test-consumer-group

1. If you are using the old high-level consumer and storing the group metadata in ZooKeeper (i.e. offsets.storage=zookeeper)

bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group test-consumer-group

Listing consumer groups

1. List all consumer groups across all topics:

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --list

1. If you are using the old high-level consumer and storing the group metadata in ZooKeeper (i.e. offsets.storage=zookeeper)

bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list

Infrequently used operations

The following infrequently used operations refer to official documentation.

* Migrating data to new machines
* Increasing replication factor
* Limiting bandwidth usage during data migration
* Setting quotas

API

Overview

1. Four core APIs:
   • The Producer API allows an application to publish a stream of records to one or more Kafka topics.
   • The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them.
   • The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams.
   • The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table.
2. Maven dependency for producer API and consumer API.

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.10.2.0</version>
</dependency>

Producer API

1. The producer is thread safe and sharing a single producer instance across threads will generally be faster than having multiple instances.
2. The producer consists of a pool of buffer space that holds records that haven't yet been transmitted to the server as well as a background I/O thread that is responsible for turning these records into requests and transmitting them to the cluster. Failure to close the producer after use will leak these resources.
3. The send() method is asynchronous. When called it adds the record to a buffer of pending record sends and immediately returns. This allows the producer to batch together individual records for efficiency.
4. Producer provides software load balancing through an optionally user-specified kafka.producer.Partitioner.
5. An example.

import java.util.Properties;
import java.util.UUID;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;

public class TestProducer {

    public static void main(String[] args) {
        Properties conf = new Properties();
        conf.put("bootstrap.servers", "centos1:9092,centos2:9092");
        conf.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        conf.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        final String topic = "test-topic";
        Producer<String, String> producer = new KafkaProducer<>(conf);
        try {
            for (int index = 0; index < 10000; index++) {
                String recordKey = UUID.randomUUID().toString();
                String recordValue = UUID.randomUUID().toString();
                ProducerRecord<String, String> record = new ProducerRecord<>(topic, recordKey, recordValue);
                producer.send(record);
            }

        } finally {
            producer.close();
        }
    }

}

Consumer API

1. The position of the consumer gives the offset of the next record that will be given out. It will be one larger than the highest offset the consumer has seen in that partition. It automatically advances every time the consumer receives messages in a call to poll(long).
2. The committed position is the last offset that has been stored securely. Should the process fail and restart, this is the offset that the consumer will recover to. The consumer can either automatically commit offsets periodically; or it can choose to control this committed position manually by calling one of the commit APIs (e.g. commitSync and commitAsync).
3. Membership in a consumer group is maintained dynamically: if a process fails, the partitions assigned to it will be reassigned to other consumers in the same group. Similarly, if a new consumer joins the group, partitions will be moved from existing consumers to the new one. This is known as rebalancing the group and is discussed in more detail below. Group rebalancing is also used when new partitions are added to one of the subscribed topics or when a new topic matching a subscribed regex is created. The group will automatically detect the new partitions through periodic metadata refreshes and assign them to members of the group.
4. Automatic offset committing example.

import java.util.Arrays;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;

public class TestConsumer {

    public static void main(String[] args) {
        Properties conf = new Properties();
        conf.put("bootstrap.servers", "centos1:9092,centos2:9092");
        conf.put("group.id", "test-group");
        conf.put("enable.auto.commit", "true");
        conf.put("auto.commit.interval.ms", "1000");
        conf.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        conf.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        final String topic = "test-topic";
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(conf);
        try {
            consumer.subscribe(Arrays.asList(topic));
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(100);
                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
                }
            }
        } finally {
            consumer.close();
        }
    }

}

1. Manual offset committing example.

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;

public class TestConsumer {

    public static void main(String[] args) {
        Properties conf = new Properties();
        conf.put("bootstrap.servers", "centos1:9092,centos2:9092");
        conf.put("group.id", "test-group");
        conf.put("enable.auto.commit", "false");
        conf.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        conf.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        final String topic = "test-topic";
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(conf);
        try {
            consumer.subscribe(Arrays.asList(topic));
            final int minBatchSize = 200;
            List<ConsumerRecord<String, String>> buffer = new ArrayList<>();
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(100);
                for (ConsumerRecord<String, String> record : records) {
                    buffer.add(record);
                }
                if (buffer.size() >= minBatchSize) {
                    insertIntoDb(buffer);
                    consumer.commitSync();
                    buffer.clear();
                }
            }
        } finally {
            consumer.close();
        }
    }

    private static void insertIntoDb(List<ConsumerRecord<String, String>> records) {
        for (ConsumerRecord<String, String> record : records) {
            System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
        }
    }

}

1. Manual partition assignment example.

import java.util.Arrays;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.TopicPartition;

public class TestConsumer {

    public static void main(String[] args) {
        Properties conf = new Properties();
        conf.put("bootstrap.servers", "localhost:9092,localhost:9093");
        conf.put("group.id", "test-group");
        conf.put("enable.auto.commit", "true");
        conf.put("auto.commit.interval.ms", "1000");
        conf.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        conf.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        final String topic = "test-topic";
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(conf);
        try {
            TopicPartition partition0 = new TopicPartition(topic, 0);
//          TopicPartition partition1 = new TopicPartition(topic, 1);
            consumer.assign(Arrays.asList(partition0));
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(100);
                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("partition = %d, offset = %d, key = %s, value = %s%n", record.partition(), record.offset(), record.key(), record.value());
                }
            }
        } finally {
            consumer.close();
        }
    }

}

1. Controlling consumer's position example.

import java.util.Arrays;
import java.util.List;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.TopicPartition;

public class TestConsumer {

    public static void main(String[] args) {
        Properties conf = new Properties();
        conf.put("bootstrap.servers", "localhost:9092,localhost:9093");
        conf.put("group.id", "test-group");
        conf.put("enable.auto.commit", "true");
        conf.put("auto.commit.interval.ms", "1000");
        conf.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        conf.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        final String topic = "test-topic";
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(conf);
        try {
            TopicPartition partition0 = new TopicPartition(topic, 0);
            TopicPartition partition1 = new TopicPartition(topic, 1);
            List<TopicPartition> partitions = Arrays.asList(partition0, partition1);
            consumer.assign(partitions);
//          consumer.seekToBeginning(partitions);
            consumer.seek(partition0, 1000);
            consumer.seek(partition1, 2000);
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(100);
                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("partition = %d, offset = %d, key = %s, value = %s%n", record.partition(), record.offset(), record.key(), record.value());
                }
            }
        } finally {
            consumer.close();
        }
    }

}

Performance tunning

1. JDK: use the latest released version of JDK 1.8.
2. Memory: you need sufficient memory to buffer active readers and writers. You can do a back-of-the-envelope estimate of memory needs by assuming you want to be able to buffer for 30 seconds and compute your memory need as write_throughput*30.
3. Disk:
   • In general disk throughput is the performance bottleneck, and more disks is better.
   • You can either RAID these drives together into a single volume or format and mount each drive as its own directory. Since Kafka has replication the redundancy provided by RAID can also be provided at the application level.
4. OS:
   • File descriptor limits: We recommend at least 100000 allowed file descriptors for the broker processes as a starting point.
   • Max socket buffer size.
5. Filesystem: EXT4 has had more usage, but recent improvements to the XFS filesystem have shown it to have better performance characteristics for Kafka's workload with no compromise in stability.