Structured streaming: A Declarative API for Real-Time Applications in Apache Spark(Abstract: 原文+注译）

题目中文：结构化流： Apache spark中，处理实时数据的声明式API

Abstract

with the ubiquity of real-time data, organizations need streaming systems that are scalable, easy to use, and easy to integrate into business applications. Structured Streaming is a new high-level streaming. Structured Streaming differs from other recent streaming APIs, such as Google Dataflow, in two main ways. First, it is a purely declarative API based on automatically incrementalizing a static relational query(expressed using SQL or DataFrames), in contrast to APIs that ask the user to build a DAG of physical operators. Second, Structured Streaming aims to support end-to-end real time applications that integrate streaming with batch and interactive analysis. We found that this integration was often a key challenge in practice. Structured Streaming achieves high performance via Spark SQL's code generation engine and can outperformance via Spark SQL's code generation engine and can outperform Apache Flink by up to 2* and Apache Kafka streams by 90*. It also offers rich operational features such as roolbacks, code updates, and mixed streaming/batch execution. We describtion deployments on Databricks, the largest of which process over 1PB of data per month.

　　随着实时数据的普遍性，企业也更加需要“流式计算系统”具有更好的可扩展性、容易使用，并且容易整合进业务系统中去。结构化流（Structured Streaming)是一个基于我们开发spark streaming经验而开发出来的一个高级流式API。结构化流和其他最近的一些流式API（比如：Google Dataflow）主要在两个方面有所不一样。

　　第一. 它是一个纯粹的声明式API。它基于自动增量化关系查询（这个查询方法，通常使用SQL或者DataFrames)。这一点上，它和那些要求用户创建一个物理operators的DAG很不一样。

　　第二. 结构化流目的在于支持端到端的实时应用，并且集成了批处理和交互式分析。

　　我们实践时才发现，这样的集成却是真正关键的挑战。结构化流媒体通过Spark SQL代码生成器能够取得很好的表现。我们测试时得到的效果是：它的性能是Flink的两倍，是Apache Kafka Stream的90倍（主要应该是吞吐量上，可参考第9节的具体讲解）。它也提供了丰富的操作特性，比如：回滚，代码更新，混合批处理和流处理。

　　我们通过Databricks上上百个生产部署的案例来描述系统的设计和使用。其中，最大的每个月处理超过1PB的数据。