January 11, 2018Apache Flink

Robert Metzger and Chris Ward

A favorite session from Flink Forward Berlin 2017 was Robert Metzger’s “Keep It Going: How to Reliably and Efficiently Operate Apache Flink”. One of the topics that Robert touches on is how to roughly size an Apache Flink cluster. Flink Forward attendees mentioned that his cluster sizing guidelines were helpful to them, and so we’ve converted that section of his talk into a blog post. Enjoy!

One of the most frequently-asked questions in the Flink community is how to size a cluster when moving from development to production. The definitive answer to this question, is, of course, “it depends,” but that’s not a helpful answer. This post outlines a series of questions to ask to arrive at some numbers you can use as guidance.

Do the Math and Establish a Baseline

The first step is to think through your application’s operational metrics to arrive at a baseline of required resources.

The key metrics to consider are:

  • The number of records per second and the size per record
  • The number of distinct keys you have and the state size per key
  • The number of state updates and the access patterns of your state backend

Finally, a more pragmatic concern is your service-level agreements (SLAs) around downtime, latency, and max throughput with your customers as these directly influence your capacity planning.

Next, look at what resources you have available based on your budget. For example:

  • The network capacity, taking into account any external services that also use the network, such as Kafka, HDFS, etc.
  • Your disk bandwidth, if you are relying on a disk-based state backend like RocksDB (and considering other disk use like Kafka or HDFS)
  • The number of machines and the CPU and memory they have available

Based on all these factors, you can now build a baseline for normal operation, plus a buffer of resources used for recovery catch-up or to handle load spikes. I recommend you also consider the resources used during checkpointing when establishing the baseline.

Example: Let’s run some numbers

I will now plan a job deployment on a hypothetical cluster to visualize the process of establishing a resource usage baseline. These numbers are rough “back-of-the-envelope” values, and they’re not comprehensive–at the end of the post, I’ll also identify some of the aspects that I ignored while making this calculation.

Example Flink Streaming Job and Hardware


Example Flink Streaming job topology

For this example, I am going to deploy a typical Flink streaming job that reads data from a Kafka topic using Flink’s Kafka consumer. The stream is then transformed using a keyed, aggregating window operator. The window operator performs aggregations on time windows of 5 minutes. As there is always fresh data, I’ll configure the window to be a sliding window with a 1-minute slide.

This means I’ll get the aggregates for the past 5 minutes updated every minute. The streaming job creates an aggregate per userId. The messages consumed from the Kafka topic have a size (on average) of 2 KB.

The throughput is 1 million messages per second. To understand the state size of the window operator, you need to know the number of distinct keys. In this case, it’s the number of userIds, which is 500,000,000 unique users. For each user, you are computing four numbers, stored as longs (8 bytes).

Let’s summarize the job’s key metrics:

  • Message size: 2KB
  • Throughput: 1,000,000 msg/sec
  • Distinct keys: 500,000,000 (aggregation in window: 4 longs per key)
  • Checkpointing: Once every minute.


 Hypothetical Hardware Setup

There are five machines running the job, each running a Flink TaskManager (Flink’s worker nodes). Disks are network-attached (common in cloud setups), and there is a 10 Gigabit Ethernet connection from the main switch to each machine running a TaskManager. The Kafka brokers are running on separate machines.

Each machine has 16 CPU cores. For simplicity, I won’t consider CPU and memory requirements. In the real world, depending on your application logic and the state backend in use, you would need to pay attention to memory. This example uses a RocksDB-based state backend, which is robust and has low memory requirements.

A Single Machine’s Perspective

To understand the resource requirements of the whole job deployment, it’s easiest to focus on the operations in one machine and one TaskManager first. You can then use the numbers derived from one machine to calculate the overall resource requirements.

By default (if all operators have the same parallelism and there are no special scheduling restrictions), all operators of a streaming job are running on each machine.

In this case, the Kafka source (or consumer), window operator, and Kafka sink (or producer) are all running on each of the five machines.

A machine perspective – TaskManager n

keyBy is a separate operator in the figure above so that calculating the resource requirements is easier. In reality, keyBy is an API construct and translates into a configuration attribute for the connection between the Kafka source and window operator.

I will now go through each of these operators from top to bottom to understand their network resource requirements.

The Kafka source

To calculate the amount of data received by an individual Kafka source, first, compute the aggregate Kafka input. The sources receive 1,000,000 messages per second that are 2KB each.

2KB x 1,000,000/s = 2GB/s

Dividing 2GB/s by the number of machines (5) leads to the following result:

2GB/s ÷ 5 machines = 400MB/s

Each of the 5 Kafka sources running in the cluster receives data with an average throughput of 400 MB/s.

The Kafka source calculation

The Shuffle / keyBy

Next, you need to ensure that all events with the same key (in this case the userId) end up on the same machine. The data in the Kafka topic you are reading from might be partitioned according to a different partitioning scheme.

The shuffling process sends all data with the same key to one machine, so you are splitting the 400MB/s stream of data coming from Kafka into a userId-partitioned stream:

400MB/s ÷ 5 machines = 80MB/s

On average, you have to send 80 MB/s of data to each of the machines. This analysis is from the perspective of a single machine which means that some of the data is already on the designated target machine, so subtract 80MB/s to account for that:

400MB/s – 80MB = 320MB/s

Each machine receives and sends user data at a rate of 320MB/s.

The shuffle calculation

Window Emit and Kafka Sink

The next question to ask is how much data the window operator emits and sends through to the Kafka sink. It’s 67MB/s, and let’s explain how we arrived at this number.

The window operator keeps an aggregate of 4 numbers (represented as longs) for each key. Once every minute, the operator emits the current aggregate values. Each key emits 2 ints (user_idwindow_ts) and 4 longs from the aggregation:

(2 x 4 bytes) + (4 x 8 bytes) = 40 bytes per key

Then factor in the keys (500,000,000 divided by the number of machines):

100,000,000 keys x 40 bytes = 4GB

…from each machine.

Then calculate the per-second size:

4GB/min ÷ 60 = 67MB/s

…emitted by each TaskManager.

This means that each TaskManager emits on average 67 MB/s of user data from the window operators. Since there is a Kafka sink running on each TaskManager (next to the window operator), and there’s no further repartitioning, this is the amount of data emitted from Flink to Kafka.

User data: From Kafka, shuffled to the window operators and back to Kafka

The emission of data from the window operators is expected to be “bursty,” because they are emitting the data once every minute. In practice, the operator will not send data at a constant rate of 67 MB/s, but rather max out the available bandwidth for a few seconds every minute.

This all totals to:

  • Data in: 720MB/s (400 + 320) per machine
  • Data out: 387MB/s (320 + 67) per machine

State Access and Checkpointing

That’s not everything. So far, I’ve only looked at the user data that Flink is processing. You need to include the overhead from disk access to RocksDB for storing state and checkpointing. To understand the disk access costs, you look at how the window operator accesses state. The Kafka source also keeps some state, but it is negligible compared to the window operator.

To understand the state size of the window operator, look at it from a different angle. Flink is computing five-minute windows with a 1-minute slide. Flink implements sliding windows by maintaining five windows, one for each “slide.” As mentioned earlier, you maintain 40 bytes of state for each window and each key for the aggregations when using a window implementation which is performing an eager aggregation. For every incoming event, you first need to retrieve the current aggregation values from disk (read 40 bytes), update the aggregates, and then write the new value back (write 40 bytes).

Window State

This means:

40 bytes of state x 5 windows x 200,000 msg/s per machine = 40MB/s

…of read or write disk access per machine. As said in the beginning, the disks are network attached, so I need to add these numbers to the overall throughput calculations.
The totals are now:

  • Data in: 760MB/s (400 MB/s data in + 320 MB/s shuffle + 40 MB/s state)
  • Data out: 427MB/s (320 MB/s shuffle + 67 MB/s data out + 40 MB/s state)


The above considerations are for the state access, which happens consistently as new events arrive at the window operator. You also have checkpointing enabled for fault-tolerance. If a machine or anything else fails, you want to restore your window contents and continue processing.

Checkpointing is set to an interval of one checkpoint per minute, and each checkpoint copies the entire state of the job into a network-attached file system.

Let’s quickly see how big the entire state on each machine is:

40bytes of state x 5 windows x 100,000,000 keys = 20GB

And, to get the per-second value:

20GB ÷ 60 = 333 MB/s.

Similar to the window operator, checkpointing has a bursty pattern, and once every minute, it tries to send its data at full speed to external storage. Checkpointing causes additional state access to RocksDB (which in this example is located on network attached disks). Since Flink 1.3, the RocksDB state backend supports incremental checkpointing, reducing the required network transfers on each checkpoint, by conceptually only sending the “diff” since the last checkpoint, but this feature is not used in this example.

This updates the totals to:

  • Data in: 760MB/s (400 + 320 + 40)
  • Data out: 760MB/s (320 + 67 + 40 + 333)


This means that the overall network traffic is:

760 + 760 x 5 + 400 + 2335 = 10335 MB/s


The 400 is total of the 80MB state access (read and write) process across the 5 machines, and 2335 is the total of the Kafka in and out processes across the cluster.

Or just over half the available network capacity in the hardware setup above.


Networking requirements

There’s a disclaimer I’d like to add. None of these calculations include protocol overhead such as TCP, Ethernet, and RPC calls from Flink, Kafka, or the file system. This is still good starting point to understand what sort of hardware you will need for a job and to have an indication of performance.

Scale Your Way

Based on my analysis, this example, with a 5-node cluster, and in typical operation, each machine would need to handle 760 MB/s of data, both in and out, from a total capacity of 1250 MB/s. That reserves about 40% of the network capacity for the complexities I glossed over, such as network protocol overheads, heavy load during event replay when recovering from a checkpoint, and uneven load balancing across the cluster caused by data skew.

There’s no one-size-fits-all answer to whether 40% is an appropriate amount of headroom, but this arithmetic should give you a good starting point. Try the calculations above, swapping out the number of machines, the number of keys, or the messages per second to get a selection of values to consider and then balance that with your budget and operational factors. Happy scaling!

Tags: capacity planningcluster sizingresource planningthroughput

How To Size Your Apache Flink® Cluster: A Back-of-the-Envelope Calculation的更多相关文章

  1. Flink监控:Monitoring Apache Flink Applications

    This post originally appeared on the Apache Flink blog. It was reproduced here under the Apache Lice ...

  2. Managing Large State in Apache Flink®: An Intro to Incremental Checkpointing

    January 23, 2018- Apache Flink, Flink Features Stefan Richter and Chris Ward Apache Flink was purpos ...

  3. Apache Flink 开发环境搭建和应用的配置、部署及运行

    https://mp.weixin.qq.com/s/noD2Jv6m-somEMtjWTJh3w 本文是根据 Apache Flink 系列直播课程整理而成,由阿里巴巴高级开发工程师沙晟阳分享,主要 ...

  4. apache flink源码挖坑 (未完待续)

    Apache Flink 源码解读(一) ​ By yyz940922原创 项目模块 (除去.git, .github, .idea, docs等): flink-annotations: flink ...

  5. Peeking into Apache Flink's Engine Room

    http://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html   Join Processin ...

  6. Apache Flink

    Flink 剖析 1.概述 在如今数据爆炸的时代,企业的数据量与日俱增,大数据产品层出不穷.今天给大家分享一款产品—— Apache Flink,目前,已是 Apache 顶级项目之一.那么,接下来, ...

  7. Apache Flink Quickstart

    Apache Flink 是新一代的基于 Kappa 架构的流处理框架,近期底层部署结构基于 FLIP-6 做了大规模的调整,我们来看一下在新的版本(1.6-SNAPSHOT)下怎样从源码快速编译执行 ...

  8. 新一代大数据处理引擎 Apache Flink

    https://www.ibm.com/developerworks/cn/opensource/os-cn-apache-flink/index.html 大数据计算引擎的发展 这几年大数据的飞速发 ...

  9. 腾讯大数据平台Oceanus: A one-stop platform for real time stream processing powered by Apache Flink

    January 25, 2019Use Cases, Apache Flink The Big Data Team at Tencent     In recent years, the increa ...

随机推荐

  1. 使用SDL2显示一张图片,SDL2上手贴

    SDL全名Simple DirectMedia Layer,是一个跨平台的底层音频.视频.键盘.鼠标操作库,操作实际通过更底层的OpenGL/Direct3D完成,在保留跨平台的兼容性之外提供了非常高 ...

  2. FloatingActionButton(悬浮按钮)使用学习<一>

      FloatingActionButton简称FAB.   一. 对于App或某个页面中是否要使用FloatingActionButton必要性:     FAB代表一个App或一个页面中最主要的操 ...

  3. .NET代码树执行时间计时器

    有很多时候我们需要对代码不同段计算一个执行时间,并希望通过节点树的方式表达现每段代码的执行时长.在.Net下似乎找不到这样一个功能类,所以花了一些时间实现这样一个代码运行计时器.首先看一下简单的需求. ...

  4. 15-Flink实战项目之实时热销排行

    戳更多文章: 1-Flink入门 2-本地环境搭建&构建第一个Flink应用 3-DataSet API 4-DataSteam API 5-集群部署 6-分布式缓存 7-重启策略 8-Fli ...

  5. 从0到1,了解NLP中的文本相似度

    本文由云+社区发表 作者:netkiddy 导语 AI在2018年应该是互联网界最火的名词,没有之一.时间来到了9102年,也是项目相关,涉及到了一些AI写作相关的功能,为客户生成一些素材文章.但是, ...

  6. 【Vue.js】vue基础: 3种Class和Style绑定语法

    凡是用到了v-bind,那就一定有变量的存在,下面是三种语法的展示: 1. 对象语法: v-bind:class="{active: isActive, 'text-danger': has ...

  7. Python UnboundLocalError和NameError错误根源解析

    如果代码风格相对而言不是那么的pythonic,或许很少碰到这类错误.当然并不是不鼓励使用一些python语言的技巧.如果遇到这这种类型的错误,说明我们对python中变量引用相关部分有不当的认识和理 ...

  8. Python3+Selenium2完整的自动化测试实现之旅(二):IE和Chrome浏览器驱动配置

    上一篇写了自动化测试需要的最基础环境配置,地址:https://www.cnblogs.com/tdp0108/p/10412073.html 当前市面上很多公司开发的应用软件都是BS架构,即基于浏览 ...

  9. java数组及数组的插入,删除,冒泡算法

    1.数组的定义 数组为相同类型的若干个数据,在一个数组里面,不能存放多种不同类型的数据,其中每个数据为该数组的一个元素,可以通过下标对改元素进行访问. 1.1 数组的特点 (1)数组被创建后,长度就已 ...

  10. 让Mongo在Spring中跑起来

    本文标题为<让Mongo在Spring中跑起来>,旨在Spring中如何成功连接MongoDB并对其进行增删改查等操作,由于笔者也是刚接触,对其中的一些原由也不甚了解,若有错误之处,敬请指 ...