why’s kafka so fast
As we all know that Kafka is very fast, much faster than most of its competitors. So what’s the reason here?
Avoid Random Disk Access
Kafka writes everything onto the disk in order and consumers fetch data in order too. So disk access always works sequentially instead of randomly. For traditional hard disks(HDD), sequential access is much faster than random access. Here is a comparison:
hardware | sequential writes | random writes |
---|---|---|
6 * 7200rpm SATA RAID-5 | 300MB/s | 50KB/s |
Kafka Writes Everything Onto The Disk Instead of Memory
Yes, you read that right. Kafka writes everything onto the disk instead of memory. But wait a moment, isn’t memory supposed to be faster than disks? Typically it’s the case, for Random Disk Access. But for sequential access, the difference is much smaller. Here is a comparison taken from https://queue.acm.org/detail.cfm?id=1563874.
As you can see, it’s not that different. But still, sequential memory access is faster than Sequential Disk Access, why not choose memory? Because Kafka runs on top of JVM, which gives us two disadvantages.
1.The memory overhead of objects is very high, often doubling the size of the data stored(or even higher).
2.Garbage Collection happens every now and then, so creating objects in memory is very expensive as in-heap data increases because we will need more time to collect unused data(which is garbage).
So writing to file systems may be better than writing to memory. Even better, we can utilize MMAP(memory mapped files) to make it faster.
Memory Mapped Files(MMAP)
Basically, MMAP(Memory Mapped Files) can map the file contents from the disk into memory. And when we write something into the mapped memory, the OS will flush the change onto the disk sometime later. So everything is faster because we are using memory actually, but in an indirect way. So here comes the question. Why would we use MMAP to write data onto disks, which later will be mapped into memory? It seems to be a roundabout route. Why not just write data into memory directly? As we have learned previously, Kafka runs on top of JVM, if we wrote data into memory directly, the memory overhead would be high and GC would happen frequently. So we use MMAP here to avoid the issue.
Zero Copy
Suppose that we are fetching data from the memory and sending them to the Internet. What is happening in the process is usually twofold.
1.To fetch data from the memory, we need to copy those data from the Kernel Context into the Application Context.
2.To send those data to the Internet, we need to copy the data from the Application Context into the Kernel Context.
As you can see, it’s redundant to copy data between the Kernel Context and the Application Context. Can we avoid it? Yes, using Zero Copy we can copy data directly from the Kernel Context to the Kernel Context.
Batch Data
Kafka only sends data when batch.size
is reached instead of one by one. Assuming the bandwidth is 10MB/s, sending 10MB data in one go is much faster than sending 10000 messages one by one(assuming each message takes 100 bytes).
"I would be fine and made no troubles."
why’s kafka so fast的更多相关文章
- Apache Kafka for Item Setup
At Walmart.com in the U.S. and at Walmart's 11 other websites around the world, we provide seamless ...
- Apache Kafka - Schema Registry
关于我们为什么需要Schema Registry? 参考, https://www.confluent.io/blog/how-i-learned-to-stop-worrying-and-love- ...
- Understanding, Operating and Monitoring Apache Kafka
Apache Kafka is an attractive service because it's conceptually simple and powerful. It's easy to un ...
- Build an ETL Pipeline With Kafka Connect via JDBC Connectors
This article is an in-depth tutorial for using Kafka to move data from PostgreSQL to Hadoop HDFS via ...
- How to choose the number of topics/partitions in a Kafka cluster?
This is a common question asked by many Kafka users. The goal of this post is to explain a few impor ...
- kafka producer源码
producer接口: /** * Licensed to the Apache Software Foundation (ASF) under one or more * contributor l ...
- Flume-ng+Kafka+storm的学习笔记
Flume-ng Flume是一个分布式.可靠.和高可用的海量日志采集.聚合和传输的系统. Flume的文档可以看http://flume.apache.org/FlumeUserGuide.html ...
- Apache Kafka: Next Generation Distributed Messaging System---reference
Introduction Apache Kafka is a distributed publish-subscribe messaging system. It was originally dev ...
- Exploring Message Brokers: RabbitMQ, Kafka, ActiveMQ, and Kestrel--reference
[This article was originally written by Yves Trudeau.] http://java.dzone.com/articles/exploring-mess ...
随机推荐
- SpringCloud的阿里巴巴相关开源组件
Sentinel 阿里巴巴开源产品,把流量作为切入点,从流量控制.熔断降级.系统负载保护等多个维度保护服务的稳定性. Nacos 阿里巴巴开源产品,一个更易于构建云原生应用的动态服务发现.配置管理和服 ...
- vue---v-model的详细解答
1.v-model:双向数据绑定的实现原理 等同于一个 v-bind 加 v-on <div id="app"> <!-- <input t ...
- flink WaterMark之TumblingEventWindow
1.WaterMark,翻译成水印或水位线,水印翻译更抽象,水位线翻译接地气. watermark是用于处理乱序事件的,通常用watermark机制结合window来实现. 流处理从事件产生,到流经s ...
- 利用Python模拟登录pastebin.com
任务 在https://pastebin.com网站注册一个账号,利用python实现用户的自动登录和创建paste.该任务需要分成如下两步利用python实现: 账号的自动登录 paste的自动创建 ...
- Python必备面试题
Python部分 1. __new__.__init__区别,如何实现单例模式,有什么优点 __new__是一个静态方法,__init__是一个实例方法 __new__返回一个创建的实例,__in ...
- 李宏毅-Network Compression课程笔记
一.方法总结 Network Pruning Knowledge Distillation Parameter Quantization Architecture Design Dynamic Com ...
- 实现 Trie (前缀树)
实现一个 Trie (前缀树),包含 insert, search, 和 startsWith 这三个操作. 示例: Trie trie = new Trie(); trie.insert(" ...
- AI AND THE BOTTOM LINE: 15 EXAMPLES OF ARTIFICIAL INTELLIGENCE IN FINANCE
https://builtin.com/artificial-intelligence/ai-finance-banking-applications-companies f there's one ...
- websocket广播式实例
1.引入相关依赖 <parent> <groupId>org.springframework.boot</groupId> <artifactId>sp ...
- HDU 6588 Function
[传送门] 求$$\sum_{i=1}^{n} \gcd(\lfloor \sqrt[3]{i} \rfloor, i)$$题解写的很清楚,自己重新推一推. $$\sum_{i=1}^{n} \gcd ...