Flink入门 - API
final StreamExecutionEnvironment streamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment();
/*
* Filter
*/
DataStream<Long> input = streamExecutionEnvironment.generateSequence(-5, 5);
input.filter(new FilterFunction<Long>() {
@Override
public boolean filter(Long value) throws Exception {
// TODO Auto-generated method stub
return value >= 0;
}
}).print();
streamExecutionEnvironment.execute();
/*
* Connect
*/
DataStream<Long> someStream = streamExecutionEnvironment.generateSequence(0, 10);
DataStream<String> otherStream = streamExecutionEnvironment.fromElements(WordCountData.WORDS);
ConnectedStreams<Long, String> connectedStreams = someStream.connect(otherStream);
DataStream<String> result = connectedStreams.flatMap(new CoFlatMapFunction<Long, String, String>() {
@Override
public void flatMap1(Long value, Collector<String> out) throws Exception {
// TODO Auto-generated method stub
out.collect(value.toString());
}
@Override
public void flatMap2(String value, Collector<String> out) throws Exception {
// TODO Auto-generated method stub
Arrays.asList(value.split("\\W+")).stream().forEachOrdered(str -> out.collect(str));
}
});
result.print();
streamExecutionEnvironment.execute();
/*
* KeyBy
*/
DataStream<Tuple4<String, String, String, Integer>> input = streamExecutionEnvironment.fromElements(TRANSCRIPT);
KeyedStream<Tuple4<String, String, String, Integer>, Tuple> keyedStream = input.keyBy("f0");
keyedStream.print();
keyedStream.maxBy("f3").print();
streamExecutionEnvironment.execute();
public static final Tuple4[] TRANSCRIPT = new Tuple4[] {
Tuple4.of("class1","张三","语文",100),
Tuple4.of("class1","李四","语文",78),
Tuple4.of("class1","王五","语文",99),
Tuple4.of("class2","赵六","语文",81),
Tuple4.of("class2","钱七","语文",59),
Tuple4.of("class2","马二","语文",97)
};
/*
* Map
*/
DataStream<Long> input = streamExecutionEnvironment.generateSequence(0, 10);
DataStream<Long> plusOne = input.map(new MapFunction<Long, Long>() {
@Override
public Long map(Long value) throws Exception {
// TODO Auto-generated method stub
return value + 1;
}
});
plusOne.print();
streamExecutionEnvironment.execute();
/*
* Fold
*/
DataStream<Tuple4<String, String, String, Integer>> input = streamExecutionEnvironment.fromElements(TRANSCRIPT);
DataStream<String> result = input.keyBy(0).fold("Start", new FoldFunction<Tuple4<String, String, String, Integer>, String>() {
@Override
public String fold(String str, Tuple4<String, String, String, Integer> value) throws Exception {
// TODO Auto-generated method stub
return str + " = " + value.f1 + " ";
}
});
result.print();
streamExecutionEnvironment.execute();
public static final Tuple4[] TRANSCRIPT = new Tuple4[] {
Tuple4.of("class1","张三","语文",100),
Tuple4.of("class1","李四","语文",78),
Tuple4.of("class1","王五","语文",99),
Tuple4.of("class2","赵六","语文",81),
Tuple4.of("class2","钱七","语文",59),
Tuple4.of("class2","马二","语文",97)
};
/**
1> Start = 赵六
1> Start = 赵六 = 钱七
1> Start = 赵六 = 钱七 = 马二
2> Start = 张三
2> Start = 张三 = 李四
2> Start = 张三 = 李四 = 王五
*/
/*
* Reduce
*/
DataStream<Tuple4<String, String, String, Integer>> input = streamExecutionEnvironment.fromElements(TRANSCRIPT);
KeyedStream<Tuple4<String, String, String, Integer>, Tuple> keyedStream = input.keyBy(0);
keyedStream.reduce(new ReduceFunction<Tuple4<String, String, String, Integer>>() {
@Override
public Tuple4<String, String, String, Integer> reduce(Tuple4<String, String, String, Integer> value1,
Tuple4<String, String, String, Integer> value2) throws Exception {
// TODO Auto-generated method stub
value1.f3 += value2.f3;
return value1;
}
}).print();
streamExecutionEnvironment.execute();
/**
2> (class1,张三,语文,100)
2> (class1,张三,语文,178)
2> (class1,张三,语文,277)
1> (class2,赵六,语文,81)
1> (class2,赵六,语文,140)
1> (class2,赵六,语文,237)
*/
/*
* Project
*/
DataStream<Tuple4<String, String, String, Integer>> input = streamExecutionEnvironment.fromElements(TRANSCRIPT);
DataStream<Tuple2<String, Integer>> output = input.project(1, 3);
output.print();
streamExecutionEnvironment.execute();
/**
4> (张三,100)
4> (钱七,59)
2> (王五,99)
3> (赵六,81)
1> (李四,78)
1> (马二,97)
*/
/*
* SplitAndSelect
*/
DataStream<Long> input = streamExecutionEnvironment.generateSequence(0, 10);
SplitStream<Long> splitStream = input.split(new OutputSelector<Long>() {
@Override
public Iterable<String> select(Long value) {
// TODO Auto-generated method stub
List<String> output = new ArrayList<>();
if (value % 2 == 0) {
output.add(EVEN);
} else {
output.add(ODD);
}
return output;
}
});
// splitStream.print();
DataStream<Long> even = splitStream.select(EVEN);
DataStream<Long> odd = splitStream.select(ODD);
DataStream<Long> all = splitStream.select(EVEN, ODD);
odd.print();
streamExecutionEnvironment.execute();
/*
* FlatMap
*/
DataStream<String> input = streamExecutionEnvironment.fromElements(WordCountData.WORDS);
DataStream<String> wordStream = input.flatMap(new FlatMapFunction<String, String>() {
@Override
public void flatMap(String value, Collector<String> out) throws Exception {
// TODO Auto-generated method stub
Arrays.asList(value.toLowerCase().split("\\W+")).stream().filter(str -> str.length() > 0).forEach(str -> out.collect(str));
}
});
wordStream.print();
streamExecutionEnvironment.execute();
Flink入门 - API的更多相关文章
- Flink入门(五)——DataSet Api编程指南
Apache Flink Apache Flink 是一个兼顾高吞吐.低延迟.高性能的分布式处理框架.在实时计算崛起的今天,Flink正在飞速发展.由于性能的优势和兼顾批处理,流处理的特性,Flink ...
- Flink入门宝典(详细截图版)
本文基于java构建Flink1.9版本入门程序,需要Maven 3.0.4 和 Java 8 以上版本.需要安装Netcat进行简单调试. 这里简述安装过程,并使用IDEA进行开发一个简单流处理程序 ...
- Flink入门(二)——Flink架构介绍
1.基本组件栈 了解Spark的朋友会发现Flink的架构和Spark是非常类似的,在整个软件架构体系中,同样遵循着分层的架构设计理念,在降低系统耦合度的同时,也为上层用户构建Flink应用提供了丰富 ...
- Flink入门(四)——编程模型
flink是一款开源的大数据流式处理框架,他可以同时批处理和流处理,具有容错性.高吞吐.低延迟等优势,本文简述flink的编程模型. 数据集类型: 无穷数据集:无穷的持续集成的数据集合 有界数据集:有 ...
- 【翻译】Flink Table Api & SQL — SQL客户端Beta 版
本文翻译自官网:SQL Client Beta https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/sqlCl ...
- 记一次flink入门学习笔记
团队有几个系统数据量偏大,且每天以几万条的数量累增.有一个系统每天需要定时读取数据库,并进行相关的业务逻辑计算,从而获取最新的用户信息,定时任务的整个耗时需要4小时左右.由于定时任务是夜晚执行,目前看 ...
- 不一样的Flink入门教程
前言 微信搜[Java3y]关注这个朴实无华的男人,点赞关注是对我最大的支持! 文本已收录至我的GitHub:https://github.com/ZhongFuCheng3y/3y,有300多篇原创 ...
- Flink入门-第一篇:Flink基础概念以及竞品对比
Flink入门-第一篇:Flink基础概念以及竞品对比 Flink介绍 截止2021年10月Flink最新的稳定版本已经发展到1.14.0 Flink起源于一个名为Stratosphere的研究项目主 ...
- Flink入门使用
完全参考:Flink1.3QuickStart 启动本地运行 首先找一台安装了hadoop的linux. 将安装包解压,到bin目录启动local模式的脚本. tar -zxvf flink-1.3. ...
随机推荐
- SQLServer len 函数, 查字符串长度函数
declare @name char(1000) --注意:char(10)为10位,要是位数小了会让数据出错 set @name='s{sss}fc{fggh}dghdf{cccs}x' selec ...
- 【转载】 tf.ConfigProto和tf.GPUOptions用法总结
原文地址: https://blog.csdn.net/C_chuxin/article/details/84990176 -------------------------------------- ...
- 查看apache httpd server中加载了哪些模块
说明: 有的时候,需要查看当前apache中都加载了哪些模块,通过以下命令进行查看 [root@hadoop1 httpd-]# bin/apachectl -t -D DUMP_MODULES Lo ...
- C++ Set运用实例
C++ Set运用实例 #include <iostream> #include <set> #include <algorithm> #include <i ...
- bootstrap datetimepicker 添加清空按钮
<div class="ys-datetimepicker"> <input class="form-control" size=" ...
- sql 各种依赖关系解释
1.数据依赖 数据依赖指的是通过一个关系中属性间的相等与否体现出来的数据间的相互关系,其中最重要的是函数依赖和多值依赖. 2.函数依赖 设X,Y是关系R的两个属性集合,当任何时刻R中的任意两个元组中的 ...
- Dev系列控件的AJAX使用Demo
一.Dev Data Edit控件通用属性以及方法: 属性 1.GetEnabled():返回控件是否为可操作状态 2.GetText():返回控件的Text的值 3.SetEnabled():设置控 ...
- 微信小程序tabBar底部导航 不显示问题解析
2019年十月八号 转藏: 版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明. 本文链接:https://blog.csdn.net/wy_Blo ...
- Javascript / Nodejs call 和 apply
call: 改变了函数运行的作用域,即改变函数里面this的指向apply:同call,apply第二个参数是数组结构 例如: this.name = 'Ab'var obj = {name: 'BB ...
- Android 问题解决 HorizontalScrollView显示不全(转)
原链接:https://www.jianshu.com/p/003adbcaff9d Android 问题解决 HorizontalScrollView显示不全 <HorizontalScrol ...