Spark2.x(六十):在Structured Streaming流处理中是如何查找kafka的DataSourceProvider?
本章节根据源代码分析Spark Structured Streaming(Spark2.4)在进行DataSourceProvider查找的流程,首先,我们看下读取流数据源kafka的代码:
- SparkSession sparkSession = SparkSession.builder().getOrCreate();
- Dataset<Row> sourceDataset = sparkSession.readStream().format("kafka").option("xxx", "xxx").load();
sparkSession.readStream()返回的对象是DataSourceReader
DataSourceReader(https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala),其中上边代码中的load()方法,正是DataSourceReader的方法。
format参数kafka在DataSourceReader中作为source属性:
- @InterfaceStability.Evolving
- final class DataStreamReader private[sql](sparkSession: SparkSession) extends Logging {
- /**
- * Specifies the input data source format.
- *
- * @since 2.0.0
- */
- def format(source: String): DataStreamReader = {
- this.source = source
- this
- }
- 。。。
- }
DataSourceReader#format(source:String)中参数往往是csv/text/json/jdbc/kafka/console/socket等
DataSourceReader#load()方法
- /**
- * Loads input data stream in as a `DataFrame`, for data streams that don't require a path
- * (e.g. external key-value stores).
- *
- * @since 2.0.0
- */
- def load(): DataFrame = {
- if (source.toLowerCase(Locale.ROOT) == DDLUtils.HIVE_PROVIDER) {
- throw new AnalysisException("Hive data source can only be used with tables, you can not " +
- "read files of Hive data source directly.")
- }
- val ds = DataSource.lookupDataSource(source, sparkSession.sqlContext.conf).newInstance()
- // We need to generate the V1 data source so we can pass it to the V2 relation as a shim.
- // We can't be sure at this point whether we'll actually want to use V2, since we don't know the
- // writer or whether the query is continuous.
- val v1DataSource = DataSource(
- sparkSession,
- userSpecifiedSchema = userSpecifiedSchema,
- className = source,
- options = extraOptions.toMap)
- val v1Relation = ds match {
- case _: StreamSourceProvider => Some(StreamingRelation(v1DataSource))
- case _ => None
- }
- ds match {
- case s: MicroBatchReadSupport =>
- val sessionOptions = DataSourceV2Utils.extractSessionConfigs(
- ds = s, conf = sparkSession.sessionState.conf)
- val options = sessionOptions ++ extraOptions
- val dataSourceOptions = new DataSourceOptions(options.asJava)
- var tempReader: MicroBatchReader = null
- val schema = try {
- tempReader = s.createMicroBatchReader(
- Optional.ofNullable(userSpecifiedSchema.orNull),
- Utils.createTempDir(namePrefix = s"temporaryReader").getCanonicalPath,
- dataSourceOptions)
- tempReader.readSchema()
- } finally {
- // Stop tempReader to avoid side-effect thing
- if (tempReader != null) {
- tempReader.stop()
- tempReader = null
- }
- }
- Dataset.ofRows(
- sparkSession,
- StreamingRelationV2(
- s, source, options,
- schema.toAttributes, v1Relation)(sparkSession))
- case s: ContinuousReadSupport =>
- val sessionOptions = DataSourceV2Utils.extractSessionConfigs(
- ds = s, conf = sparkSession.sessionState.conf)
- val options = sessionOptions ++ extraOptions
- val dataSourceOptions = new DataSourceOptions(options.asJava)
- val tempReader = s.createContinuousReader(
- Optional.ofNullable(userSpecifiedSchema.orNull),
- Utils.createTempDir(namePrefix = s"temporaryReader").getCanonicalPath,
- dataSourceOptions)
- Dataset.ofRows(
- sparkSession,
- StreamingRelationV2(
- s, source, options,
- tempReader.readSchema().toAttributes, v1Relation)(sparkSession))
- case _ =>
- // Code path for data source v1.
- Dataset.ofRows(sparkSession, StreamingRelation(v1DataSource))
- }
- }
val ds=DataSoruce.lookupDataSource(source ,….).newInstance()用到了该source变量,要想知道ds是什么(Dataset还是其他),需要查看DataSource.lookupDataSource(source,。。。)方法实现。
DataSource.lookupDataSource(source, sparkSession.sqlContext.conf)解析
DataSource源代码文件:https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
其中lookupDataSource方法是DataSource类的object对象中定义的:
- object DataSource extends Logging {
- 。。。。。/**
- * Class that were removed in Spark 2.0. Used to detect incompatibility libraries for Spark 2.0.
- */
- private val spark2RemovedClasses = Set(
- "org.apache.spark.sql.DataFrame",
- "org.apache.spark.sql.sources.HadoopFsRelationProvider",
- "org.apache.spark.Logging")
- /** Given a provider name, look up the data source class definition. */
- def lookupDataSource(provider: String, conf: SQLConf): Class[_] = {
- val provider1 = backwardCompatibilityMap.getOrElse(provider, provider) match {
- case name if name.equalsIgnoreCase("orc") &&
- conf.getConf(SQLConf.ORC_IMPLEMENTATION) == "native" =>
- classOf[OrcFileFormat].getCanonicalName
- case name if name.equalsIgnoreCase("orc") &&
- conf.getConf(SQLConf.ORC_IMPLEMENTATION) == "hive" =>
- "org.apache.spark.sql.hive.orc.OrcFileFormat"
- case "com.databricks.spark.avro" if conf.replaceDatabricksSparkAvroEnabled =>
- "org.apache.spark.sql.avro.AvroFileFormat"
- case name => name
- }
- val provider2 = s"$provider1.DefaultSource"
- val loader = Utils.getContextOrSparkClassLoader
- val serviceLoader = ServiceLoader.load(classOf[DataSourceRegister], loader)
- try {
- serviceLoader.asScala.filter(_.shortName().equalsIgnoreCase(provider1)).toList match {
- // the provider format did not match any given registered aliases
- case Nil =>
- try {
- Try(loader.loadClass(provider1)).orElse(Try(loader.loadClass(provider2))) match {
- case Success(dataSource) =>
- // Found the data source using fully qualified path
- dataSource
- case Failure(error) =>
- if (provider1.startsWith("org.apache.spark.sql.hive.orc")) {
- throw new AnalysisException(
- "Hive built-in ORC data source must be used with Hive support enabled. " +
- "Please use the native ORC data source by setting 'spark.sql.orc.impl' to " +
- "'native'")
- } else if (provider1.toLowerCase(Locale.ROOT) == "avro" ||
- provider1 == "com.databricks.spark.avro" ||
- provider1 == "org.apache.spark.sql.avro") {
- throw new AnalysisException(
- s"Failed to find data source: $provider1. Avro is built-in but external data " +
- "source module since Spark 2.4. Please deploy the application as per " +
- "the deployment section of \"Apache Avro Data Source Guide\".")
- } else if (provider1.toLowerCase(Locale.ROOT) == "kafka") {
- throw new AnalysisException(
- s"Failed to find data source: $provider1. Please deploy the application as " +
- "per the deployment section of " +
- "\"Structured Streaming + Kafka Integration Guide\".")
- } else {
- throw new ClassNotFoundException(
- s"Failed to find data source: $provider1. Please find packages at " +
- "http://spark.apache.org/third-party-projects.html",
- error)
- }
- }
- } catch {
- case e: NoClassDefFoundError => // This one won't be caught by Scala NonFatal
- // NoClassDefFoundError's class name uses "/" rather than "." for packages
- val className = e.getMessage.replaceAll("/", ".")
- if (spark2RemovedClasses.contains(className)) {
- throw new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
- "Please check if your library is compatible with Spark 2.0", e)
- } else {
- throw e
- }
- }
- case head :: Nil =>
- // there is exactly one registered alias
- head.getClass
- case sources =>
- // There are multiple registered aliases for the input. If there is single datasource
- // that has "org.apache.spark" package in the prefix, we use it considering it is an
- // internal datasource within Spark.
- val sourceNames = sources.map(_.getClass.getName)
- val internalSources = sources.filter(_.getClass.getName.startsWith("org.apache.spark"))
- if (internalSources.size == 1) {
- logWarning(s"Multiple sources found for $provider1 (${sourceNames.mkString(", ")}), " +
- s"defaulting to the internal datasource (${internalSources.head.getClass.getName}).")
- internalSources.head.getClass
- } else {
- throw new AnalysisException(s"Multiple sources found for $provider1 " +
- s"(${sourceNames.mkString(", ")}), please specify the fully qualified class name.")
- }
- }
- } catch {
- case e: ServiceConfigurationError if e.getCause.isInstanceOf[NoClassDefFoundError] =>
- // NoClassDefFoundError's class name uses "/" rather than "." for packages
- val className = e.getCause.getMessage.replaceAll("/", ".")
- if (spark2RemovedClasses.contains(className)) {
- throw new ClassNotFoundException(s"Detected an incompatible DataSourceRegister. " +
- "Please remove the incompatible library from classpath or upgrade it. " +
- s"Error: ${e.getMessage}", e)
- } else {
- throw e
- }
- }
- }
- 、、、
- }
其业务流程:
1)优先从object DataSource预定义backwardCompatibilityMap中查找provider;
2)查找失败,返回原名字;
3)使用serviceLoader加载DataSourceRegister的子类集合;
4)过滤3)中集合中shortName与provider相等的provider;
5)返回providerClass。
其中的backwardCompatibilityMap也是DataSource的object对象中的定义的,相当于是一个预定义provider的集合。
- object DataSource extends Logging {
- /** A map to maintain backward compatibility in case we move data sources around. */
- private val backwardCompatibilityMap: Map[String, String] = {
- val jdbc = classOf[JdbcRelationProvider].getCanonicalName
- val json = classOf[JsonFileFormat].getCanonicalName
- val parquet = classOf[ParquetFileFormat].getCanonicalName
- val csv = classOf[CSVFileFormat].getCanonicalName
- val libsvm = "org.apache.spark.ml.source.libsvm.LibSVMFileFormat"
- val orc = "org.apache.spark.sql.hive.orc.OrcFileFormat"
- val nativeOrc = classOf[OrcFileFormat].getCanonicalName
- val socket = classOf[TextSocketSourceProvider].getCanonicalName
- val rate = classOf[RateStreamProvider].getCanonicalName
- Map(
- "org.apache.spark.sql.jdbc" -> jdbc,
- "org.apache.spark.sql.jdbc.DefaultSource" -> jdbc,
- "org.apache.spark.sql.execution.datasources.jdbc.DefaultSource" -> jdbc,
- "org.apache.spark.sql.execution.datasources.jdbc" -> jdbc,
- "org.apache.spark.sql.json" -> json,
- "org.apache.spark.sql.json.DefaultSource" -> json,
- "org.apache.spark.sql.execution.datasources.json" -> json,
- "org.apache.spark.sql.execution.datasources.json.DefaultSource" -> json,
- "org.apache.spark.sql.parquet" -> parquet,
- "org.apache.spark.sql.parquet.DefaultSource" -> parquet,
- "org.apache.spark.sql.execution.datasources.parquet" -> parquet,
- "org.apache.spark.sql.execution.datasources.parquet.DefaultSource" -> parquet,
- "org.apache.spark.sql.hive.orc.DefaultSource" -> orc,
- "org.apache.spark.sql.hive.orc" -> orc,
- "org.apache.spark.sql.execution.datasources.orc.DefaultSource" -> nativeOrc,
- "org.apache.spark.sql.execution.datasources.orc" -> nativeOrc,
- "org.apache.spark.ml.source.libsvm.DefaultSource" -> libsvm,
- "org.apache.spark.ml.source.libsvm" -> libsvm,
- "com.databricks.spark.csv" -> csv,
- "org.apache.spark.sql.execution.streaming.TextSocketSourceProvider" -> socket,
- "org.apache.spark.sql.execution.streaming.RateSourceProvider" -> rate
- )
- }
- 。。。
- }
shortName为kafka且实现了DataSourceRegister接口的类:
满足“shortName为kafka且实现了DataSourceRegister接口的类”就是:KafkaSourceProvider(https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala)
- /**
- * The provider class for all Kafka readers and writers. It is designed such that it throws
- * IllegalArgumentException when the Kafka Dataset is created, so that it can catch
- * missing options even before the query is started.
- */
- private[kafka010] class KafkaSourceProvider extends DataSourceRegister
- with StreamSourceProvider
- with StreamSinkProvider
- with RelationProvider
- with CreatableRelationProvider
- with TableProvider
- with Logging {
- import KafkaSourceProvider._
- override def shortName(): String = "kafka"
- 。。。。
- }
DataSourceRegister类定义
- /**
- * Data sources should implement this trait so that they can register an alias to their data source.
- * This allows users to give the data source alias as the format type over the fully qualified
- * class name.
- *
- * A new instance of this class will be instantiated each time a DDL call is made.
- *
- * @since 1.5.0
- */
- @InterfaceStability.Stable
- trait DataSourceRegister {
- /**
- * The string that represents the format that this data source provider uses. This is
- * overridden by children to provide a nice alias for the data source. For example:
- *
- * {{{
- * override def shortName(): String = "parquet"
- * }}}
- *
- * @since 1.5.0
- */
- def shortName(): String
- }
继承了DataSourceRegister的类有哪些?
继承了DataSourceRegister的类包含:
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
https://github.com/apache/spark/blob/branch-2.4/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
https://github.com/apache/spark/blob/branch-2.4/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormat.scala
https://github.com/apache/spark/blob/branch-2.4/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamProvider.scala
https://github.com/apache/spark/blob/branch-2.4/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala
https://github.com/apache/spark/blob/branch-2.4/sql/hive/src/test/scala/org/apache/spark/sql/sources/SimpleTextRelation.scala
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
https://github.com/apache/spark/blob/branch-2.4/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/test/scala/org/apache/spark/sql/sources/fakeExternalSources.scala
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/test/scala/org/apache/spark/sql/sources/DDLSourceLoadSuite.scala
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2.scala
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/noop/NoopDataSource.scala
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/console.scala
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcRelationProvider.scala
https://github.com/apache/spark/blob/branch-2.4/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageFileFormat.scala
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/TextSocketSourceProvider.scala
Spark2.x(六十):在Structured Streaming流处理中是如何查找kafka的DataSourceProvider?的更多相关文章
- C#高级编程六十九天----DLR简介 .在.NET中使用DLR(转载) 我也来说说Dynamic
DLR 一.近年来,在TIOBE公司每个月发布的编程语言排行榜中,C#总是能挤进前十名,而在最近十年来,C#总体上呈现上升的趋势.C#能取得这样的成绩,有很多因素,其中它在语言特性上的锐意进取让人印象 ...
- Spark2.x(六十一):在Spark2.4 Structured Streaming中Dataset是如何执行加载数据源的?
本章主要讨论,在Spark2.4 Structured Streaming读取kafka数据源时,kafka的topic数据是如何被执行的过程进行分析. 以下边例子展开分析: SparkSession ...
- Spark学习进度11-Spark Streaming&Structured Streaming
Spark Streaming Spark Streaming 介绍 批量计算 流计算 Spark Streaming 入门 Netcat 的使用 项目实例 目标:使用 Spark Streaming ...
- Kafka:ZK+Kafka+Spark Streaming集群环境搭建(二十三)Structured Streaming遇到问题:Set(TopicName-0) are gone. Some data may have been missed
事情经过:之前该topic(M_A)已经存在,而且正常使用structured streaming消费了一段时间,后来删除了topic(M_A),重新创建了topic(M-A),程序使用新创建的top ...
- Spark2.3(四十二):Spark Streaming和Spark Structured Streaming更新broadcast总结(二)
本次此时是在SPARK2,3 structured streaming下测试,不过这种方案,在spark2.2 structured streaming下应该也可行(请自行测试).以下是我测试结果: ...
- Spark2.2(三十八):Spark Structured Streaming2.4之前版本使用agg和dropduplication消耗内存比较多的问题(Memory issue with spark structured streaming)调研
在spark中<Memory usage of state in Spark Structured Streaming>讲解Spark内存分配情况,以及提到了HDFSBackedState ...
- Spark2.3(三十五)Spark Structured Streaming源代码剖析(从CSDN和Github中看到别人分析的源代码的文章值得收藏)
从CSDN中读取到关于spark structured streaming源代码分析不错的几篇文章 spark源码分析--事件总线LiveListenerBus spark事件总线的核心是LiveLi ...
- Spark2.x(五十五):在spark structured streaming下sink file(parquet,csv等),正常运行一段时间后:清理掉checkpoint,重新启动app,无法sink记录(file)到hdfs。
场景: 在spark structured streaming读取kafka上的topic,然后将统计结果写入到hdfs,hdfs保存目录按照month,day,hour进行分区: 1)程序放到spa ...
- Spark2.3(三十四):Spark Structured Streaming之withWaterMark和windows窗口是否可以实现最近一小时统计
WaterMark除了可以限定来迟数据范围,是否可以实现最近一小时统计? WaterMark目的用来限定参数计算数据的范围:比如当前计算数据内max timestamp是12::00,waterMar ...
随机推荐
- linux技能五 文件权限
文件权限:-rw-r--r--. 1 fileInUser fileInGroup 1623 5月 4 19:33 fileName -:第一个-是文件类型 rw-:文件的所有者权限 r--:文件的 ...
- ningx配置本地https环境
由于项目改成了https访问,所以本地开发的时候也要通过https验证,避免页面发送http请求. 例如原来是这样访问:http://192.168.88.88:8080/ 或 http://loca ...
- liunx 安装nc/netcat centos安装netcat
如果本文对你有用,请爱心点个赞,提高排名,帮助更多的人.谢谢大家!❤ 如果解决不了,可以在文末进群交流. 1:下载源码包,wget https://sourceforge.net/projects/n ...
- React: 无状态组件生成真实DOM结点
在上一篇文章中,我们总结并模拟了 JSX 生成真实 DOM 结点的过程,今天接着来介绍一下无状态组件的生成过程. 先以下面一段简单的代码举例: const Greeting = function ({ ...
- MySQL Charset--UTF8和UTF8MB4对比测试
UTF8和UTF8MB4 在早期MySQL版本中,使用只支持最长三字节的UTF8字符集便可以存放所有Unicode字符.随着Unicode的完善,Unicode字符集收录的字符数量越来越多,最新版本的 ...
- Java开发环境之MyEclipse
查看更多Java开发环境配置,请点击<Java开发环境配置大全> 拾贰章:MyEclipse安装教程 1)下载MyEclipse安装包 http://www.myeclipsecn.com ...
- Android开发之常用Intent.Action【转】
1.从google搜索内容 Intent intent = new Intent(); intent.setAction(Intent.ACTION_WEB_SEARCH); intent.putEx ...
- 10 分钟上手 Vue 组件 Vue-Draggable
Vue 综合了 Angualr 和 React 的优点,因其易上手,轻量级,受到了广泛应用.成为了是时下火热的前端框架,吸引着越来越多的前端开发者! 本文将通过一个最简单的拖拽例子带领大家快速上手 V ...
- Mac电脑永久路由的添加方法是是什么? Mac校园网连接教程
学校校园网面向全校师生开放,无奈Windows用户基数大,学校只为Windows平台制作了内网连接工具,Mac平台资源较少,本人查阅相关资料后,总结整理出以下步骤,方便本校学生连接校园网. 有永久路由 ...
- Windows Cmd 命令管理服务
今天在Windows 干净环境上安装软件过程中,安装完成后,发现部署在IIS 上的网站无法使用,提示 "您提交的参数有误!,请重新提交" 纯净的windows 7 x64位环境, ...