RDD算子
RDD算子
- #常用Transformation(即转换,延迟加载)
- #通过并行化scala集合创建RDD
- val rdd1 = sc.parallelize(Array(1,2,3,4,5,6,7,8))
- #查看该rdd的分区数量
- rdd1.partitions.length
- val rdd1 = sc.parallelize(List(5,6,4,7,3,8,2,9,1,10))
- val rdd2 = sc.parallelize(List(5,6,4,7,3,8,2,9,1,10)).map(_*2).sortBy(x=>x,true)
- val rdd3 = rdd2.filter(_>10)
- val rdd2 = sc.parallelize(List(5,6,4,7,3,8,2,9,1,10)).map(_*2).sortBy(x=>x+"",true)
- val rdd2 = sc.parallelize(List(5,6,4,7,3,8,2,9,1,10)).map(_*2).sortBy(x=>x.toString,true)
- val rdd4 = sc.parallelize(Array("a b c", "d e f", "h i j"))
- rdd4.flatMap(_.split(' ')).collect
- val rdd5 = sc.parallelize(List(List("a b c", "a b b"),List("e f g", "a f g"), List("h i j", "a a b")))
- List("a b c", "a b b") =List("a","b",))
- rdd5.flatMap(_.flatMap(_.split(" "))).collect
- #union求并集,注意类型要一致
- val rdd6 = sc.parallelize(List(5,6,4,7))
- val rdd7 = sc.parallelize(List(1,2,3,4))
- val rdd8 = rdd6.union(rdd7)
- rdd8.distinct.sortBy(x=>x).collect
- #intersection求交集
- val rdd9 = rdd6.intersection(rdd7)
- val rdd1 = sc.parallelize(List(("tom", 1), ("jerry", 2), ("kitty", 3)))
- val rdd2 = sc.parallelize(List(("jerry", 9), ("tom", 8), ("shuke", 7), ("tom", 2)))
- #join(连接)
- val rdd3 = rdd1.join(rdd2)
- val rdd3 = rdd1.leftOuterJoin(rdd2)
- val rdd3 = rdd1.rightOuterJoin(rdd2)
- #groupByKey
- val rdd3 = rdd1 union rdd2
- rdd3.groupByKey
- //(tom,CompactBuffer(1, 8, 2))
- rdd3.groupByKey.map(x=>(x._1,x._2.sum))
- groupByKey.mapValues(_.sum).collect
- Array((tom,CompactBuffer(1, 8, 2)), (jerry,CompactBuffer(9, 2)), (shuke,CompactBuffer(7)), (kitty,CompactBuffer(3)))
- #WordCount
- sc.textFile("/root/words.txt").flatMap(x=>x.split(" ")).map((_,1)).reduceByKey(_+_).sortBy(_._2,false).collect
- sc.textFile("/root/words.txt").flatMap(x=>x.split(" ")).map((_,1)).groupByKey.map(t=>(t._1, t._2.sum)).collect
- #cogroup
- val rdd1 = sc.parallelize(List(("tom", 1), ("tom", 2), ("jerry", 3), ("kitty", 2)))
- val rdd2 = sc.parallelize(List(("jerry", 2), ("tom", 1), ("shuke", 2)))
- val rdd3 = rdd1.cogroup(rdd2)
- val rdd4 = rdd3.map(t=>(t._1, t._2._1.sum + t._2._2.sum))
- #cartesian笛卡尔积
- val rdd1 = sc.parallelize(List("tom", "jerry"))
- val rdd2 = sc.parallelize(List("tom", "kitty", "shuke"))
- val rdd3 = rdd1.cartesian(rdd2)
- ###################################################################################################
- #spark action
- val rdd1 = sc.parallelize(List(1,2,3,4,5), 2)
- #collect
- rdd1.collect
- #reduce
- val r = rdd1.reduce(_+_)
- #count
- rdd1.count
- #top
- rdd1.top(2)
- #take
- rdd1.take(2)
- #first(similer to take(1))
- rdd1.first
- #takeOrdered
- rdd1.takeOrdered(3)
spark RDD api
- http://homepage.cs.latrobe.edu.au/zhe/ZhenHeSparkRDDAPIExamples.html
- mapPartitionsWithIndex
- val func = (index: Int, iter: Iterator[(String)]) => {
- iter.map(x => "[partID:" + index + ", val: " + x + "]")
- }
- mapPartitionsWithIndex
- val func = (index: Int, iter: Iterator[Int]) => {
- iter.map(x => "[partID:" + index + ", val: " + x + "]")
- }
- val rdd1 = sc.parallelize(List(1,2,3,4,5,6,7,8,9), 2)
- rdd1.mapPartitionsWithIndex(func).collect
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- aggregate
- def func1(index: Int, iter: Iterator[(Int)]) : Iterator[String] = {
- iter.toList.map(x => "[partID:" + index + ", val: " + x + "]").iterator
- }
- val rdd1 = sc.parallelize(List(1,2,3,4,5,6,7,8,9), 2)
- rdd1.mapPartitionsWithIndex(func1).collect
- rdd1.aggregate(0)(math.max(_, _), _ + _)
- rdd1.aggregate(5)(math.max(_, _), _ + _)
- val rdd2 = sc.parallelize(List("a","b","c","d","e","f"),2)
- def func2(index: Int, iter: Iterator[(String)]) : Iterator[String] = {
- iter.toList.map(x => "[partID:" + index + ", val: " + x + "]").iterator
- }
- rdd2.aggregate("")(_ + _, _ + _)
- rdd2.aggregate("=")(_ + _, _ + _)
- val rdd3 = sc.parallelize(List("12","23","345","4567"),2)
- rdd3.aggregate("")((x,y) => math.max(x.length, y.length).toString, (x,y) => x + y)
- val rdd4 = sc.parallelize(List("12","23","345",""),2)
- rdd4.aggregate("")((x,y) => math.min(x.length, y.length).toString, (x,y) => x + y)
- val rdd5 = sc.parallelize(List("12","23","","345"),2)
- rdd5.aggregate("")((x,y) => math.min(x.length, y.length).toString, (x,y) => x + y)
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- aggregateByKey
- val pairRDD = sc.parallelize(List( ("cat",2), ("cat", 5), ("mouse", 4),("cat", 12), ("dog", 12), ("mouse", 2)), 2)
- def func2(index: Int, iter: Iterator[(String, Int)]) : Iterator[String] = {
- iter.map(x => "[partID:" + index + ", val: " + x + "]")
- }
- pairRDD.mapPartitionsWithIndex(func2).collect
- pairRDD.aggregateByKey(0)(math.max(_, _), _ + _).collect
- pairRDD.aggregateByKey(100)(math.max(_, _), _ + _).collect
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- checkpoint
- sc.setCheckpointDir("hdfs://node-1.edu360.cn:9000/ck")
- val rdd = sc.textFile("hdfs://node-1.edu360.cn:9000/wc").flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_)
- rdd.checkpoint
- rdd.isCheckpointed
- rdd.count
- rdd.isCheckpointed
- rdd.getCheckpointFile
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- coalesce, repartition
- val rdd1 = sc.parallelize(1 to 10, 10)
- val rdd2 = rdd1.coalesce(2, false)
- rdd2.partitions.length
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- collectAsMap
- val rdd = sc.parallelize(List(("a", 1), ("b", 2)))
- rdd.collectAsMap
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- combineByKey
- val rdd1 = sc.textFile("hdfs://node-1.edu360.cn:9000/wc").flatMap(_.split(" ")).map((_, 1))
- val rdd2 = rdd1.combineByKey(x => x, (a: Int, b: Int) => a + b, (m: Int, n: Int) => m + n)
- rdd2.collect
- val rdd3 = rdd1.combineByKey(x => x + 10, (a: Int, b: Int) => a + b, (m: Int, n: Int) => m + n)
- rdd3.collect
- val rdd4 = sc.parallelize(List("dog","cat","gnu","salmon","rabbit","turkey","wolf","bear","bee"), 3)
- val rdd5 = sc.parallelize(List(1,1,2,2,2,1,2,2,2), 3)
- val rdd6 = rdd5.zip(rdd4)
- val rdd7 = rdd6.combineByKey(List(_), (x: List[String], y: String) => x :+ y, (m: List[String], n: List[String]) => m ++ n)
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- countByKey
- val rdd1 = sc.parallelize(List(("a", 1), ("b", 2), ("b", 2), ("c", 2), ("c", 1)))
- rdd1.countByKey
- rdd1.countByValue
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- filterByRange
- val rdd1 = sc.parallelize(List(("e", 5), ("c", 3), ("d", 4), ("c", 2), ("a", 1)))
- val rdd2 = rdd1.filterByRange("b", "d")
- rdd2.colllect
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- flatMapValues
- val a = sc.parallelize(List(("a", "1 2"), ("b", "3 4")))
- rdd3.flatMapValues(_.split(" "))
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- foldByKey
- val rdd1 = sc.parallelize(List("dog", "wolf", "cat", "bear"), 2)
- val rdd2 = rdd1.map(x => (x.length, x))
- val rdd3 = rdd2.foldByKey("")(_+_)
- val rdd = sc.textFile("hdfs://node-1.edu360.cn:9000/wc").flatMap(_.split(" ")).map((_, 1))
- rdd.foldByKey(0)(_+_)
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- foreachPartition
- val rdd1 = sc.parallelize(List(1, 2, 3, 4, 5, 6, 7, 8, 9), 3)
- rdd1.foreachPartition(x => println(x.reduce(_ + _)))
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- keyBy
- val rdd1 = sc.parallelize(List("dog", "salmon", "salmon", "rat", "elephant"), 3)
- val rdd2 = rdd1.keyBy(_.length)
- rdd2.collect
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- keys values
- val rdd1 = sc.parallelize(List("dog", "tiger", "lion", "cat", "panther", "eagle"), 2)
- val rdd2 = rdd1.map(x => (x.length, x))
- rdd2.keys.collect
- rdd2.values.collect
- -------------------------------------------------------------------------------------------
- -------------------------------------------------------------------------------------------
- mapPartitions( it: Iterator => {it.map(x => x * 10)})
RDD算子的更多相关文章
- RDD 算子补充
一.RDD算子补充 1.mapPartitions mapPartitions的输入函数作用于每个分区, 也就是把每个分区中的内容作为整体来处理. (map是把每一行) mapPa ...
- RDD算子、RDD依赖关系
RDD:弹性分布式数据集, 是分布式内存的一个抽象概念 RDD:1.一个分区的集合, 2.是计算每个分区的函数 , 3.RDD之间有依赖关系 4.一个对于key-value的RDD的Partit ...
- spark教程(四)-SparkContext 和 RDD 算子
SparkContext SparkContext 是在 spark 库中定义的一个类,作为 spark 库的入口点: 它表示连接到 spark,在进行 spark 操作之前必须先创建一个 Spark ...
- Spark性能调优-RDD算子调优篇(深度好文,面试常问,建议收藏)
RDD算子调优 不废话,直接进入正题! 1. RDD复用 在对RDD进行算子时,要避免相同的算子和计算逻辑之下对RDD进行重复的计算,如下图所示: 对上图中的RDD计算架构进行修改,得到如下图所示的优 ...
- Spark中普通集合与RDD算子的sortBy()有什么区别
分别观察一下集合与算子的sortBy()的参数列表 普通集合的sortBy() RDD算子的sortBy() 结论:普通集合的sortBy就没有false参数,也就是说只能默认的升序排. 如果需要对普 ...
- Spark RDD算子介绍
Spark学习笔记总结 01. Spark基础 1. 介绍 Spark可以用于批处理.交互式查询(Spark SQL).实时流处理(Spark Streaming).机器学习(Spark MLlib) ...
- 大数据入门第二十二天——spark(二)RDD算子(2)与spark其它特性
一.JdbcRDD与关系型数据库交互 虽然略显鸡肋,但这里还是记录一下(点开JdbcRDD可以看到限制比较死,基本是鸡肋.但好在我们可以通过自定义的JdbcRDD来帮助我们完成与关系型数据库的交互.这 ...
- 大数据入门第二十二天——spark(二)RDD算子(1)
一.RDD概述 1.什么是RDD RDD(Resilient Distributed Dataset)叫做分布式数据集,是Spark中最基本的数据抽象,它代表一个不可变.可分区.里面的元素可并行计算的 ...
- RDD算子的使用
TransformationDemo.scala import org.apache.spark.{HashPartitioner, SparkConf, SparkContext} import s ...
随机推荐
- Pytorch划分数据集的方法
之前用过sklearn提供的划分数据集的函数,觉得超级方便.但是在使用TensorFlow和Pytorch的时候一直找不到类似的功能,之前搜索的关键字都是"pytorch split dat ...
- RNN和LSTM
一.RNN 全称为Recurrent Neural Network,意为循环神经网络,用于处理序列数据. 序列数据是指在不同时间点上收集到的数据,反映了某一事物.现象等随时间的变化状态或程度.即数据之 ...
- 【原创】大数据基础之Logstash(4)高可用
logstash高可用体现为不丢数据(前提为服务器短时间内不可用后可恢复比如重启服务器或重启进程),具体有两个方面: 进程重启(服务器重启) 事件消息处理失败 在logstash中对应的解决方案为: ...
- 关于Java中static关键字的用法以及从内存角度解析
在Java中,static可以用来修饰成员变量和成员方法. 修饰成员变量,称为静态成员方法 修饰静态方法,称为静态成员方法 搞清楚用法和区别之前,先搞清static声明的变量和普通非静态变量在内存的分 ...
- 详解MariaDB数据库的存储过程
1.什么是存储过程 很多时候,SQL语句都是针对一个或多个表的单条语句.但是也有时候有的查询语句需要进行多次联表查询才能完成,此时就需要用到存储过程了. 存储过程(Stored Procedure)是 ...
- vue学习(二)
- 实践笔记_J2EE_Server_Tomcat_tomcat域名绑定_1_单域名绑定
Tomcat域名绑定(1)单域名绑定 1. 测试环境说明 名称 版本 ...
- VisualVM远程监控Java
1.服务器启动jstatd服务设置: jstatd将使用RMISecurityPolicy,并且需要指定安全策略文件. 安全策略文件必须符合安全策略语法,通过命令 jstatd -J-Djava.se ...
- centos7设置httpd
1.httpd开机自启动 systemctl enable httpd.service 2.httpd开机不自启动 systemctl disable httpd.service 3.启动httpd ...
- Linux scp 命令卡住的原因
When transferring large files(for example mksysb images) using scp through a firewall, the scp conne ...