Spark2.3（三十四）：Spark Structured Streaming之withWaterMark和windows窗口是否可以实现最近一小时统计
WaterMark除了可以限定来迟数据范围，是否可以实现最近一小时统计？
WaterMark目的用来限定参数计算数据的范围：比如当前计算数据内max timestamp是12::00，waterMark限定数据分为是60 minutes，那么如果此时输入11:00之前的数据就会被舍弃不参与统计，视为来迟范围超出了60minutes限定范围。
那么，是否可以借助它实现最近一小时的数据统计呢？
代码示例：
package com.dx.streaming import java.sql.Timestamp
import java.text.SimpleDateFormat import org.apache.spark.sql.streaming.OutputMode
import org.apache.spark.sql.{Encoders, SparkSession}
import org.apache.log4j.{Level, Logger} case class MyEntity(id: String, timestamp: Timestamp, value: Integer) object Main {
Logger.getLogger("org.apache.spark").setLevel(Level.WARN);
Logger.getLogger("akka").setLevel(Level.ERROR);
Logger.getLogger("kafka").setLevel(Level.ERROR); def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().appName("test").master("local[*]").getOrCreate()
val lines = spark.readStream.format("socket").option("host", "192.168.0.141").option("port", 19999).load() var sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
import spark.implicits._
lines.as(Encoders.STRING)
.map(row => {
val fields = row.split(",")
MyEntity(fields(0), new Timestamp(sdf.parse(fields(1)).getTime), Integer.valueOf(fields(2)))
})
.createOrReplaceTempView("tv_entity") spark.sql("select id,timestamp,value from tv_entity")
.withWatermark("timestamp", "60 minutes")
.createOrReplaceTempView("tv_entity_watermark") val resultDf = spark.sql(
s"""
|select id,sum(value) as sum_value
|from tv_entity_watermark
|group id
|""".stripMargin) val query = resultDf.writeStream.format("console").outputMode(OutputMode.Update()).start() query.awaitTermination()
query.stop()
}
}
当通过nc -lk 19999中依次（每组输入间隔几秒时间即可）输入如下数据时：
1,2018-12-01 12:00:01,100
2,2018-12-01 12:00:01,100 1,2018-12-01 12:05:01,100
2,2018-12-01 12:05:01,100 1,2018-12-01 12:15:01,100
2,2018-12-01 12:15:01,100 1,2018-12-01 12:25:01,100
2,2018-12-01 12:25:01,100 1,2018-12-01 12:35:01,100
2,2018-12-01 12:35:01,100 1,2018-12-01 12:45:01,100
2,2018-12-01 12:45:01,100 1,2018-12-01 12:55:01,100
2,2018-12-01 12:55:01,100 1,2018-12-01 13:05:02,100
2,2018-12-01 13:05:02,100 1,2018-12-01 13:15:01,100
2,2018-12-01 13:15:01,100
发现最终统计结果为：
id , sum_value
,
,
而不是期望的
id , sum_value
,
,
既然是不能限定数据统计范围是60minutes，是否需要借助于窗口函数window就可以实现呢？
是否需要借助于watermark和窗口函数window就可以实现最近1小时数据统计呢？
spark.sql("select id,timestamp,value from tv_entity")
.withWatermark("timestamp", "60 minutes")
.createOrReplaceTempView("tv_entity_watermark")
val resultDf = spark.sql(
s"""
|select id,sum(value) as sum_value
|from tv_entity_watermark
|group window(timestamp,'60 minutes','60 minutes'),id
|""".stripMargin)
val query = resultDf.writeStream.format("console").outputMode(OutputMode.Update()).start()
依然输入上边的测试数据，会发现超过1小时候数据会重新开辟（归零后重新统计）一个统计结果，而不是滚动的一小时统计。
就是把上边的测试数据分为了两组来分别统计：
第一组（小时）参与统计数据：
,-- ::,
,-- ::, ,-- ::,
,-- ::, ,-- ::,
,-- ::, ,-- ::,
,-- ::, ,-- ::,
,-- ::, ,-- ::,
,-- ::, ,-- ::,
,-- ::,
第二组（小时）参与统计数据：
,-- ::,
,-- ::, ,-- ::,
,-- ::,
猜测总结：
根据上边测试结果可以推出一个猜测结论：
在spark structured streaming中是不存储参数统计的数据的，只是对数据进行了maxTimestamp.avgTimestamp,minTimestamp存储，同时只是对数据的统计结果进行存储，下次再次触发统计时只是在原有的统计结果之上进行累加等操作，而参与统计的数据应该是没有存储，否则这类需求应该是可以实现。
但是以下代码尝试确实是可以实现，缺点太耗费资源：
package com.dx.streaming import java.sql.Timestamp
import java.text.SimpleDateFormat import org.apache.spark.sql.streaming.OutputMode
import org.apache.spark.sql.{Encoders, SparkSession}
import org.apache.log4j.{Level, Logger} case class MyEntity(id: String, timestamp: Timestamp, value: Integer) object Main {
Logger.getLogger("org.apache.spark").setLevel(Level.WARN)
Logger.getLogger("akka").setLevel(Level.ERROR)
Logger.getLogger("kafka").setLevel(Level.ERROR) def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().appName("test").master("local[*]").getOrCreate()
val lines = spark.readStream.format("socket").option("host", "192.168.0.141").option("port", 19999).load() var sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
import spark.implicits._
lines.as(Encoders.STRING)
.map(row => {
val fields = row.split(",")
MyEntity(fields(0), new Timestamp(sdf.parse(fields(1)).getTime), Integer.valueOf(fields(2)))
})
.createOrReplaceTempView("tv_entity") spark.sql("select id,timestamp,value from tv_entity")
.withWatermark("timestamp", "60 minutes")
.createOrReplaceTempView("tv_entity_watermark") var resultDf = spark.sql(
s"""
|select id,min(timestamp) min_timestamp,max(timestamp) max_timestamp,sum(value) as sum_value
|from tv_entity_watermark
|group by window(timestamp,'3600 seconds','60 seconds'),id
|""".stripMargin) val query = resultDf.writeStream.format("console").outputMode(OutputMode.Update()).start() query.awaitTermination()
query.stop()
}
}
使用spark streaming把历史结果保存到内存中实现最近一小时统计：
pom.xml
<!--Spark -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.2.0</version>
</dependency>
java code:
package com.dx.streaming; import java.io.Serializable;
import java.sql.Timestamp;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map; import org.apache.log4j.Level;
import org.apache.log4j.LogManager;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.VoidFunction;
import org.apache.spark.streaming.Durations;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaReceiverInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext; public class Main {
private static List<MyEntity> store = new ArrayList<MyEntity>();
private static JavaStreamingContext jssc; public static void main(String[] args) throws Exception {
// set log4j programmatically
LogManager.getLogger("org.apache.spark").setLevel(Level.WARN);
LogManager.getLogger("akka").setLevel(Level.ERROR);
LogManager.getLogger("kafka").setLevel(Level.ERROR); SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
//
System.out.println(sdf.parse("2018-12-04 11:00:00").getTime() - sdf.parse("2018-12-04 10:00:00").getTime()); SparkConf conf = new SparkConf().setMaster("local[*]").setAppName("NetworkWordCount");
JavaSparkContext sc = new JavaSparkContext(conf);
// jssc = new JavaStreamingContext(conf, Durations.seconds(10));
jssc = new JavaStreamingContext(sc, Durations.seconds(10)); JavaReceiverInputDStream<String> lines = jssc.socketTextStream("192.168.0.141", 19999); JavaDStream<MyEntity> dStream = lines.map(new Function<String, MyEntity>() {
private static final long serialVersionUID = 1L; public MyEntity call(String line) throws Exception {
String[] fields = line.split(",");
MyEntity myEntity = new MyEntity();
myEntity.setId(Integer.valueOf(fields[0]));
myEntity.setTimestamp(Timestamp.valueOf(fields[1]));
myEntity.setValue(Long.valueOf(fields[2]));
return myEntity;
}
}); // 不确定是否必须repartition(1)，目的避免外边这层循环多次循环，确保只执行一次大循环。
dStream.repartition(1).foreachRDD(new VoidFunction<JavaRDD<MyEntity>>() {
public void call(JavaRDD<MyEntity> tItems) throws Exception {
System.out.println("print...");
tItems.foreach(new VoidFunction<MyEntity>() {
public void call(MyEntity t) throws Exception {
System.out.println(">>>>>>>>>>>>>" + t.toString());
store.add(t);
System.out.println(store.size());
}
}); System.out.println("@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@");
for (MyEntity myEntity : store) {
System.out.println("++++++++++++++++++++++" + myEntity.toString());
} // 第一步：从store中超過1小時之前的數據剔除；
MyEntity first = store.get(0);
MyEntity last = store.get(store.size() - 1);
// 超過一小時（这里为什么这么做，假设数据本身就是按照时间循序有序插入的，实际业务中如果相同可以这样做）
while (last.getTimestamp().getTime() - first.getTimestamp().getTime() > 3600000) {
store.remove(0);
first = store.get(0);
} // 第二步：執行業務統計代碼
Map<Integer, Long> statistics = new HashMap<Integer, Long>();
for (MyEntity myEntity : store) {
if (false == statistics.containsKey(myEntity.getId())) {
statistics.put(myEntity.getId(), myEntity.getValue());
} else {
statistics.put(myEntity.getId(), myEntity.getValue() + statistics.get(myEntity.getId()));
}
} // 第三步：将结果写入关系数据库
System.out.println("#######################print result##########################");
for (Map.Entry<Integer, Long> kv : statistics.entrySet()) {
System.out.println(kv.getKey() + "," + kv.getValue());
}
}
}); jssc.start(); // Start the computation
jssc.awaitTermination(); // Wait for the computation to terminate
}
} class MyEntity implements Serializable {
private final SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
private int id;
private Timestamp timestamp;
private long value; public int getId() {
return id;
} public void setId(int id) {
this.id = id;
} public Timestamp getTimestamp() {
return timestamp;
} public void setTimestamp(Timestamp timestamp) {
this.timestamp = timestamp;
} public long getValue() {
return value;
} public void setValue(long value) {
this.value = value;
} @Override
public String toString() {
return getId() + "," + sdf.format(new Date(getTimestamp().getTime())) + "," + getValue();
}
}
输出日志
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@62d73ead{/streaming/batch,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@228cea97{/streaming/batch/json,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3db663d0{/static/streaming,null,AVAILABLE,@Spark}
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
print...
>>>>>>>>>>>>>,-- ::,
>>>>>>>>>>>>>,-- ::,
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
#######################print result##########################
,
,
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
print...
[Stage :> ( + ) / ]>>>>>>>>>>>>>,-- ::,
>>>>>>>>>>>>>,-- ::,
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
#######################print result##########################
,
,
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
print...
[Stage :> ( + ) / ]>>>>>>>>>>>>>,-- ::,
>>>>>>>>>>>>>,-- ::,
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
#######################print result##########################
,
,
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
print...
>>>>>>>>>>>>>,-- ::,
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
#######################print result##########################
,
,
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
print...
>>>>>>>>>>>>>,-- ::,
>>>>>>>>>>>>>,-- ::,
>>>>>>>>>>>>>,-- ::,
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
#######################print result##########################
,
,
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
print...
>>>>>>>>>>>>>,-- ::,
>>>>>>>>>>>>>,-- ::,
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
#######################print result##########################
,
,
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
print...
>>>>>>>>>>>>>,-- ::,
>>>>>>>>>>>>>,-- ::,
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
#######################print result##########################
,
,
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
print...
>>>>>>>>>>>>>,-- ::,
>>>>>>>>>>>>>,-- ::,
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
#######################print result##########################
,
,
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
// :: WARN storage.RandomBlockReplicationPolicy: Expecting replicas with only peer/s.
// :: WARN storage.BlockManager: Block input-- replicated to only peer(s) instead of peers
print...
>>>>>>>>>>>>>,-- ::,
>>>>>>>>>>>>>,-- ::,
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
#######################print result##########################
,
,
print...
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
++++++++++++++++++++++,-- ::,
#######################print result##########################
,
,
Spark2.3（三十四）：Spark Structured Streaming之withWaterMark和windows窗口是否可以实现最近一小时统计的更多相关文章
- Kafka：ZK+Kafka+Spark Streaming集群环境搭建（二十四）Structured Streaming：Encoder
一般情况下我们在使用Dataset<Row>进行groupByKey时,你会发现这个方法最后一个参数需要一个encoder,那么这些encoder如何定义呢? 一般数据类型 static ...
- Spark2.2（三十三）：Spark Streaming和Spark Structured Streaming更新broadcast总结（一）
背景: 需要在spark2.2.0更新broadcast中的内容,网上也搜索了不少文章,都在讲解spark streaming中如何更新,但没有spark structured streaming更新 ...
- Spark2.3（四十二）：Spark Streaming和Spark Structured Streaming更新broadcast总结（二）
本次此时是在SPARK2,3 structured streaming下测试,不过这种方案,在spark2.2 structured streaming下应该也可行(请自行测试).以下是我测试结果: ...
- Spark2.2(三十八)：Spark Structured Streaming2.4之前版本使用agg和dropduplication消耗内存比较多的问题（Memory issue with spark structured streaming）调研
在spark中<Memory usage of state in Spark Structured Streaming>讲解Spark内存分配情况,以及提到了HDFSBackedState ...
- Spark2.3（三十五）Spark Structured Streaming源代码剖析(从CSDN和Github中看到别人分析的源代码的文章值得收藏)
从CSDN中读取到关于spark structured streaming源代码分析不错的几篇文章 spark源码分析--事件总线LiveListenerBus spark事件总线的核心是LiveLi ...
- Spark2.x（五十五）：在spark structured streaming下sink file(parquet,csv等)，正常运行一段时间后：清理掉checkpoint，重新启动app，无法sink记录（file）到hdfs。
场景: 在spark structured streaming读取kafka上的topic,然后将统计结果写入到hdfs,hdfs保存目录按照month,day,hour进行分区: 1)程序放到spa ...
- Spark2.2(三十九)：如何根据appName监控spark任务,当任务不存在则启动（任务存在当超过多久没有活动状态则kill，等待下次启动）
业务需求 实现一个根据spark任务的appName来监控任务是否存在,及任务是否卡死的监控. 1)给定一个appName,根据appName从yarn application -list中验证任务是 ...
- Kafka：ZK+Kafka+Spark Streaming集群环境搭建（二十九）：推送avro格式数据到topic，并使用spark structured streaming接收topic解析avro数据
推送avro格式数据到topic 源代码:https://github.com/Neuw84/structured-streaming-avro-demo/blob/master/src/main/j ...
- Spark Structured Streaming框架（2）之数据输入源详解
Spark Structured Streaming目前的2.1.0版本只支持输入源:File.kafka和socket. 1. Socket Socket方式是最简单的数据输入源,如Quick ex ...
随机推荐
- 微信公众号开发JS-SDK（1.2）
概述 微信js-SDK是微信公众平台面向网页开发者提供的基于微信内的网页开发工具包. 通过使用微信JS-SDK,网页开发者可借助微信高效地使用拍照.选图.语音.位置等手机系统的能力,同时可以直接使用微 ...
- python 全栈开发，Day70(模板自定义标签和过滤器,模板继承 (extend),Django的模型层-ORM简介)
昨日内容回顾 视图函数: request对象 request.path 请求路径 request.GET GET请求数据 QueryDict {} request.POST POST请求数据 Quer ...
- [转] HTML5 FormData 方法介绍以及实现文件上传
XMLHttpRequest 是一个浏览器接口,通过它,我们可以使得 Javascript 进行 HTTP (S) 通信.XMLHttpRequest 在现在浏览器中是一种常用的前后台交互数据的方式. ...
- Ubuntu 安装 OpenMPI
1. 下载OpenMPI 在官网上下载最新版本的安装包,如:openmpi-1.8.4.tar.gz 2. 解压并进行配置 tar -zxvf openmpi-3.0.0.tar.gz cd open ...
- A. 【UR #16】破坏发射台
题解: 首先看n是偶数的 那么就是不需要满足对面这个性质的 这样就可以dp了 f[i][0/1]表示dp到第i位,当前数等于或不等于第一位的方案数 然后显然可以用矩阵优化 再考虑n为奇数 用一样的思路 ...
- python全栈开发day34-线程Thread
一.昨日内容回顾 1. 概念和理论 进程是计算机资源分配最小单位 进程三状态.同步.异步.阻塞.非阻塞 2. 进程的创建 实例化.自建类run,start,join,terminate,daemon等 ...
- python全栈开发day21-2 几个装饰器总结
1 @property 将一个方法伪装成属性 2.@propertty @f.setter 设置伪装成方法的属性 3.@propertty @f.deleter 删除一个伪装成方法的属性. class ...
- WP8 调用webservice 错误 The remote server returned an error: NotFound 解决
本人出错是由于本地的IIS不能被局域网其它机器访问导致的,如果你所用的本机IIS 也不可被其它机器访问,则可按照本文进行设置 具体操作时需要在防火墙设置中添加 入站规则 具体步骤如下: 1．控 ...
- BZOJ1059 [ZJOI2007]矩阵游戏 二分图匹配 匈牙利算法
欢迎访问~原文出处——博客园-zhouzhendong 去博客园看该题解 题目传送门 - BZOJ1059 题意概括 有一个n*n(n<=200)的01矩阵,问你是否可以通过交换整行和整列使得左 ...
- [OpenCV-Python] OpenCV 中的图像处理 部分 IV (四）
部分 IVOpenCV 中的图像处理 OpenCV-Python 中文教程(搬运)目录 21 OpenCV 中的轮廓 21.1 初识轮廓目标 • 理解什么是轮廓 • 学习找轮廓,绘制轮廓等 • 函数: ...