Flink Event Time Processing and Watermarks(文末有翻译)
If you are building a Realtime streaming application, Event Time processing is one of the features that you will have to use sooner or later.
Since in most of the real-world use cases messages arrive out-of-order, there should be some way through which the system you build understands the fact that messages could arrive late and handle them accordingly.
In this blog post, we will see why we need Event Time processing and how we can enable it in ApacheFlink.
EventTime is the time at which an event occurred in the real-world and ProcessingTime is the time at which that event is processed by the Flink system.
To understand the importance of Event Time processing, we will first start by building a Processing Time based system and see it’s drawback.
We will create a SlidingWindow of size 10 seconds which slides every 5 seconds and at the end of the window, the system will emit the number of messages that were received during that time.
Once you understand how EventTime processing works with respect to a SlidingWindow, it will not be difficult to understand how it works for a TumblingWindow as well. So let’s get started.
ProcessingTime based system
For this example we expect messages to have the format value,timestamp where value is the message and timestamp is the time at which this message was generated at the source.
Since we are now building a Processing Time based system, the code below ignores the timestamp part.
It is an important aspect to understand that the messages should contain the information on when it was generated.
Flink or any other system is not a magic box that can somehow figure this out by itself. Later we will see that, Event Time processing extracts this timestamp information to handle late messages.
val text = senv.socketTextStream("localhost", )
val counts = text.map {(m: String) => (m.split(",")(), ) }
.keyBy()
.timeWindow(Time.seconds(), Time.seconds())
.sum()
counts.print
senv.execute("ProcessingTime processing example")
Case 1: Messages arrive without delay
Suppose the source generated three messages of the type a at times 13th second, 13th second and 16th second respectively.
(Hours and minutes are not important here since the window size is only 10 seconds).
These messages will fall into the windows as follows.
The first two messages that were generated at 13th sec will fall into both window1[5s-15s] and window2[10s-20s] and the third message generated at 16th second will fall into window2[10s-20s] and window3[15s-25s].
The final counts emitted by each window will be (a,2), (a,3) and (a,1) respectively.
This output can be considered as the expected behavior. Now we will look at what happens when one of the message arrives late into the system.
Case 2: Messages arrive in delay
Now suppose one of the messages (generated at 13th second) arrived at a delay of 6 seconds(at 19th second), may be due to some network congestion.
Can you guess which all windows would this message fall into?
The delayed message fell into window 2 and 3, since 19 is within the range 10-20 and 15-25.
It did not cause any problem to the calculation in window2 (because the message was anyways supposed to fall into that window) but it affected the result of window1 and window3.
We will now try to fix this problem by using EventTime processing.
EventTime based system
To enable EventTime processing, we need a timestamp extractor that extracts the event time information from the message.
Remember that the messages were of the format value,timestamp. The extractTimestamp method gets the timestamp part and returns it as a Long.
Ignore the getCurrentWatermark method for now, we will come back to it later.
class TimestampExtractor extends AssignerWithPeriodicWatermarks[String] with Serializable {
override def extractTimestamp(e: String, prevElementTimestamp: Long) = {
e.split(",")().toLong
}
override def getCurrentWatermark(): Watermark = {
new Watermark(System.currentTimeMillis)
}
}
注:这个例子使用的AssignerWithPeriodicWatermarks接口。其实,还有另一个接口 AssignerWithPunctuatedWatermarks。
官网描述:
As described in timestamps and watermark handling, Flink provides abstractions that allow the programmer to assign their own timestamps and emit their own watermarks.
More specifically, one can do so by implementing one of the AssignerWithPeriodicWatermarks
and AssignerWithPunctuatedWatermarks
interfaces, depending on the use case.
In a nutshell, the first will emit watermarks periodically, while the second does so based on some property of the incoming records, e.g. whenever a special element is encountered in the stream.
We now need to set this timestamp extractor and also set the TimeCharactersistic as EventTime.
Rest of the code remains the same as in the case of ProcessingTime.
senv.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
val text = senv.socketTextStream("localhost", )
.assignTimestampsAndWatermarks(new TimestampExtractor)
val counts = text.map {(m: String) => (m.split(",")(), ) }
.keyBy()
.timeWindow(Time.seconds(), Time.seconds())
.sum()
counts.print
senv.execute("EventTime processing example")
The result of running the above code is shown in the diagram below.
The results look better, the windows 2 and 3 now emitted correct result, but window1 is still wrong.
Flink did not assign the delayed message to window 3 because it now checked the message’s event time and understood that it did not fall in that window.
But why didn’t it assign the message to window 1?.
The reason is that by the time the delayed message reached the system(at 19th second), the evaluation of window 1 has already finished (at 15th second).
Let us now try to fix this issue by using the Watermark.
Note that in window 2, the delayed message was still placed at 19th second, not at 13th second(it's event time).
This depiction in the figure was intentional to indicate that the messages within a window are not sorted according to it's event time. (this might change in future)
Watermarks
Watermarks is a very important and interesting idea and I will try to give you a brief overview about it.
If you are interested in learning more, you can watch this awesome talk from Google and also read this blog from dataArtisans.
A Watermark is essentially a timestamp. When an Operator in Flink receives a watermark, it understands(assumes) that it is not going to see any message older than that timestamp.
Hence watermark can also be thought of as a way of telling Flink how far it is, in the “EventTime”.
For the purpose of this example, think of it as a way of telling Flink how much delayed a message can be.
In the last attempt, we set the watermark as the current system time. It was, therefore, not expecting any delayed messages.
We will now set the watermark as current time - 5 seconds, which tells Flink to expect messages to be a maximum of 5 seconds dealy - This is because each window will be evaluated only when the watermark passes through it. Since our watermark is current time - 5 seconds, the first window [5s-15s] will be evaluated only at 20th second. Similarly the window [10s-20s] will be evaluated at 25th second and so on.
override def getCurrentWatermark(): Watermark = {
new Watermark(System.currentTimeMillis - )
}
Here we are assuming that the eventtime is 5 seconds older than the current system time, but that is not always the case.
In many cases it will be better to hold the max timestamp received so far(which is extracted from the message) and subtract the expected delay from it.
The result of running the code after making above changes is:
Finally we have the correct result, all the three windows now emit counts as expected - which is (a,2), (a,3) and (a,1).
Allowed Lateness
In our earlier approach where we used “watermark - delay”, the window would not fire until the watermark is past window_length + delay.
If you want to accommodate late events, and want the window to fire on-time you can use Allowed Lateness.
If allowed lateness is set, Flink will not discard message unless it is past the window_end_time + allowed lateness.
Once a late message is received, Flink will extract it’s timestamp and check if it is within the allowed lateness, then it will check whether to FIRE the window or not (as per the Trigger set).
Hence, note that a window might fire multiple times in this approach, and you might want to make your sink idempotent - if you need exactly once processing.
Conclusion
The importance of real-time stream processing systems has grown lately and having to deal with delayed message is part of any such system you build.
In this blog post, we saw how late arriving messages can affect the results of your system and how ApacheFlink’s Event Time processing capabilities can be used to solve them.
That concludes the post, Thanks for reading!
Continue reading
中文译文:https://blog.csdn.net/a6822342/article/details/78064815
Flink Event Time Processing and Watermarks(文末有翻译)的更多相关文章
- Angular 2的12个经典面试问题汇总(文末附带Angular测试)
Angular作为目前最为流行的前端框架,受到了前端开发者的普遍欢迎.不论是初学Angular的新手,还是有一定Angular开发经验的开发者,了解本文中的12个经典面试问题,都将会是一个深入了解和学 ...
- 30分钟玩转Net MVC 基于WebUploader的大文件分片上传、断网续传、秒传(文末附带demo下载)
现在的项目开发基本上都用到了上传文件功能,或图片,或文档,或视频.我们常用的常规上传已经能够满足当前要求了, 然而有时会出现如下问题: 文件过大(比如1G以上),超出服务端的请求大小限制: 请求时间过 ...
- Visual Studio Code-批量在文末添加文本字段
小技巧一例,在vs code或notepad++文末批量添加文本字段信息,便于数据信息的完整,具体操作如下: Visual Studio Code批量添加"@azureyun.com&quo ...
- C# 30分钟完成百度人脸识别——进阶篇(文末附源码)
距离上次入门篇时隔两个月才出这进阶篇,小编惭愧,对不住关注我的卡哇伊的小伙伴们,为此小编用这篇博来谢罪. 前面的准备工作我就不说了,注册百度账号api,创建web网站项目,引入动态链接库引入. 不了解 ...
- 文末福利丨i春秋互联网安全校园行第1站精彩回顾
活动背景 为响应国家完善网络安全人才培养体系.推动网络安全教育的号召,i春秋特此发起“互联网安全校园行”系列活动.旨在通过活动和知识普及提升大学生信息安全意识,并通过线下交流.技能分享.安全小活动以及 ...
- i春秋官网4.0上线啦 文末有福利
爱瑞宝地(Everybody)期待了很久的 i春秋官网4.0上线啦 除了产品的功能更加完善 性能和体验也将大幅度提高 清新.舒适的视觉感受 搭配更加便捷的操作流程 只需一秒,扫码立即登录 即刻进入网络 ...
- Angular的12个经典问题,看看你能答对几个?(文末附带Angular测试)
Angular作为目前最为流行的前端框架,受到了前端开发者的普遍欢迎.不论是初学Angular的新手,还是有一定Angular开发经验的开发者,了解本文中的12个经典面试问题,都将会是一个深入了解和学 ...
- 文末有福利 | IT从业者应关注哪些技术热点?
7月14-15日,MPD工作坊北京站即将开幕,目前大会日程已经出炉,来自各大企业的技术专家,按照软件研发中心的岗位职能划分,从产品运营.团队管理.架构技术.自动化运维等领域进行干货分享,点击此[链接] ...
- Angular 2的12个经典面试问题汇总(文末附带Angular測试)
Angular作为眼下最为流行的前端框架,受到了前端开发者的普遍欢迎.不论是初学Angular的新手.还是有一定Angular开发经验的开发者,了解本文中的12个经典面试问题,都将会是一个深入了解和学 ...
随机推荐
- ASP.NET Core SignalR中的流式传输
什么是流式传输? 流式传输是这一种以稳定持续流的形式传输数据的技术. 流式传输的使用场景 有些场景中,服务器返回的数据量较大,等待时间较长,客户端不得不等待服务器返回所有数据后,再进行相应的操作.这时 ...
- Latex文件分别用Texwork和Winedt打开时,产生中文乱码的解决方法
中文兼容方法(能保证编译成功) \usepackage{CJK} \begin{document} \begin{CJK}{GBK}{kai} ... 中文 ... \end{CJK} \end{do ...
- 从锅炉工到AI专家(1)
序言 标题来自一个很著名的梗,起因是知乎上一个问题:<锅炉设计转行 AI,可行吗?>,后来就延展出了很多类似的问句,什么"快递转行AI可行吗?"."xxx转行 ...
- C++中 引用&与取地址&的区别
微信公众号[程序员江湖] 作者黄小斜,斜杠青年,某985硕士,阿里 Java 研发工程师,于 2018 年秋招拿到 BAT 头条.网易.滴滴等 8 个大厂 offer,目前致力于分享这几年的学习经验. ...
- mongodb分片扩展架构
目录 一.简介 二.分片集群 三.数据分布策略 四.Mongos访问模式 五.Config元数据 六.分片均衡 参考文档 一.简介 MongoDB目前3大核心优势:『灵活模式』+ 『高可用性』 + 『 ...
- vue开发项目详细教程(第一篇 搭建环境篇)
最近做vue做项目碰到了不少坑,看了三天文档便开始上手做项目了,不是我牛b,是因为项目紧,我没有时间去深入学习,所以只能一边学一边做了. 我要做的项目是一个官方网站(包括管理后台),也因为是我第一次使 ...
- [一] java8 函数式编程入门 什么是函数式编程 函数接口概念 流和收集器基本概念
本文是针对于java8引入函数式编程概念以及stream流相关的一些简单介绍 什么是函数式编程? java程序员第一反应可能会理解成类的成员方法一类的东西 此处并不是这个含义,更接近是数学上的 ...
- CAN总线学习记录之四:位定时与同步
一.位定时 1.1 比特率和波特率 1)位速率:又叫做比特率(bit rata).信息传输率,表示的是单位时间内,总线上传输的信息量,即每秒能够传输的二进制位的数量,单位是bit per second ...
- [日常] nginx记录post数据
1.使用log_format指令来更改日志格式,该指令只能放在http{}段 log_format 日志名 '日志内容'; server { access_log /var/log/nginx/d ...
- 轻松理解 Java HashMap 和 ConcurrentHashMap
前言 Map 这样的 Key Value 在软件开发中是非常经典的结构,常用于在内存中存放数据. 本篇主要想讨论 ConcurrentHashMap 这样一个并发容器,在正式开始之前我觉得有必要谈谈 ...