cache和persist

将RDD数据进行存储,persist(newLevel: StorageLevel)设置了存储级别,cache()和persist()是相同的,存储级别为MEMORY_ONLY。因为RDD的transformation是lazy的,只有action算子才会触发transformain真正的执行,如果一个rdd需要进行多次的action算子操作,最好能够使用cache或persist将rdd缓存至内存中,这样除第一次action会触发transformation操作,后面的action算子都不会再次触发transformation操作。

class StorageLevel private(
private var _useDisk: Boolean,
private var _useMemory: Boolean,
private var _useOffHeap: Boolean,
private var _deserialized: Boolean,
private var _replication: Int = 1) /*复制份数,默认为1*/
extends Externalizable val NONE = new StorageLevel(false, false, false, false)
val DISK_ONLY = new StorageLevel(true, false, false, false)
val DISK_ONLY_2 = new StorageLevel(true, false, false, false, 2)
val MEMORY_ONLY = new StorageLevel(false, true, false, true)
val MEMORY_ONLY_2 = new StorageLevel(false, true, false, true, 2)
val MEMORY_ONLY_SER = new StorageLevel(false, true, false, false)
val MEMORY_ONLY_SER_2 = new StorageLevel(false, true, false, false, 2)
val MEMORY_AND_DISK = new StorageLevel(true, true, false, true)
val MEMORY_AND_DISK_2 = new StorageLevel(true, true, false, true, 2)
val MEMORY_AND_DISK_SER = new StorageLevel(true, true, false, false)
val MEMORY_AND_DISK_SER_2 = new StorageLevel(true, true, false, false, 2)
val OFF_HEAP = new StorageLevel(true, true, true, false, 1) /** Persist this RDD with the default storage level (`MEMORY_ONLY`). */
def persist(): this.type = persist(StorageLevel.MEMORY_ONLY) /** Persist this RDD with the default storage level (`MEMORY_ONLY`). */
def cache(): this.type = persist()
/**
* Set this RDD's storage level to persist its values across operations after the first time
* it is computed. This can only be used to assign a new storage level if the RDD does not
* have a storage level set yet. Local checkpointing is an exception.
*/
def persist(newLevel: StorageLevel): this.type
/**
* Mark this RDD for persisting using the specified level.
*
* @param newLevel the target storage level
* @param allowOverride whether to override any existing level with the new one
*/
private def persist(newLevel: StorageLevel, allowOverride: Boolean): this.type

mapValues(func)

元素是key-value对的RDD的每一个元素的value经过func映射(key不变)构建一个新的RDD
/**
* Pass each value in the key-value pair RDD through a map function without changing the keys;
* this also retains the original RDD's partitioning.
*/
def mapValues[U](f: V => U): RDD[(K, U)]
val rdd = sc.parallelize(List((1,1),(1,2),(1,3),(2,1),(2,2),(2,3)),3)
val rdd1 = rdd.mapValues(x=>1L)
rdd1.foreachPartition(it=>{
while(it.hasNext){
println(it.next)
}
println("================")
}
)
scala> val rdd = sc.parallelize(List((1,1),(1,2),(1,3),(2,1),(2,2),(2,3)),3)
rdd: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[1] at parallelize at <console>:24 scala> val rdd1 = rdd.mapValues(x=>1L)
rdd1: org.apache.spark.rdd.RDD[(Int, Long)] = MapPartitionsRDD[2] at mapValues at <console>:26 scala> rdd1.foreachPartition(it=>{
| while(it.hasNext){
| println(it.next)
| }
| println("================")
| }
| )
(1,1)
(1,1)
================
(1,1)
(2,1)
================
(2,1)
(2,1)
================

以上就是将(1,1),(1,2),(1,3),(2,1),(2,2),(2,3)中的value重新赋值为1,变为(1,1),(1,1),(1,1),(2,1),(2,1),(2,1)。下面使用reduceByKey计算每一个key出现的次数。

scala> val rdd1 = rdd.mapValues(x=>1L).reduceByKey(_ + _)
rdd1: org.apache.spark.rdd.RDD[(Int, Long)] = ShuffledRDD[4] at reduceByKey at <console>:26 scala> rdd1.collect.toMap
res4: scala.collection.immutable.Map[Int,Long] = Map(1 -> 3, 2 -> 3)

其实以上操作就是action算子countByKey()的实现。

collect(f: PartialFunction[T, U])

/**
* Return an RDD that contains all matching values by applying `f`.
*/
def collect[U: ClassTag](f: PartialFunction[T, U]): RDD[U]

PartialFunction[T,U]是个什么东东呢?看一下PartialFunction的apply函数,即需要定义一个f(x)函数,入参类型为A,结果输出类型为B

/** Converts ordinary function to partial one
* @since 2.10
*/
def apply[A, B](f: A => B): PartialFunction[A, B] = { case x => f(x) }
val f : PartialFunction[Int,String] = {case 0 => "Sunday"
case 1 => "Monday"
case 2 => "Tuesday"
case 3 => "Wednesday"
case 4 => "Thursday"
case 5 => "Friday"
case 6 => "Saturday"
case _ => "Unknown"
} val rdd = sc.parallelize(0 to 9)
rdd.collect(f).collect res3: Array[String] = Array(Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Unknown, Unknown, Unknown)

glom()

将每个partition中的元素合并成一个数组
/**
* Return an RDD created by coalescing all elements within each partition into an array.
*/
def glom(): RDD[Array[T]]
scala> val rdd = sc.parallelize(1 to 9,3)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[9] at parallelize at <console>:24 scala> rdd.glom
res7: org.apache.spark.rdd.RDD[Array[Int]] = MapPartitionsRDD[10] at glom at <console>:27 scala> rdd.glom.collect
res8: Array[Array[Int]] = Array(Array(1, 2, 3), Array(4, 5, 6), Array(7, 8, 9))

subtract(other: RDD[T])

返回other中不存在的元素组成新的RDD,分区属性如果没有指定,则和原RDD保持一致。
/**
* Return an RDD with the elements from `this` that are not in `other`.
*
* Uses `this` partitioner/partition size, because even if `other` is huge, the resulting
* RDD will be &lt;= us.
*/
def subtract(other: RDD[T]): RDD[T]
val rdd1 = sc.parallelize(1 to 10,2)
val rdd2 = sc.parallelize(5 to 20,3)
val rdd = rdd1.subtract(rdd2)
rdd.collect
rdd.partitions.length
scala> val rdd1 = sc.parallelize(1 to 10,2)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:24 scala> val rdd2 = sc.parallelize(5 to 20,3)
rdd2: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[1] at parallelize at <console>:24 scala> val rdd = rdd1.subtract(rdd2)
rdd: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[5] at subtract at <console>:28 scala> rdd.collect
res0: Array[Int] = Array(2, 4, 1, 3) scala> rdd.partitions.length
res2: Int = 2

指定分区数量

/**
* Return an RDD with the elements from `this` that are not in `other`.
*/
def subtract(other: RDD[T], numPartitions: Int): RDD[T]
scala> val rdd = rdd1.subtract(rdd2,5)
rdd: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[9] at subtract at <console>:28 scala> rdd.partitions.length
res3: Int = 5

自定义分区属性partitioner

/**
* Return an RDD with the elements from `this` that are not in `other`.
*/
def subtract(
other: RDD[T],
p: Partitioner)(implicit ord: Ordering[T] = null): RDD[T]
class MyPartitioner(numParts:Int) extends org.apache.spark.Partitioner{
override def numPartitions: Int = numParts
override def getPartition(key: Any): Int = {
key.toString.toInt%numPartitions
}
} val rdd = rdd1.subtract(rdd2,new MyPartitioner(3)) rdd.foreachPartition(
x=>{
while(x.hasNext){
println(x.next)
}
println("============")
}
)
scala> class MyPartitioner(numParts:Int) extends org.apache.spark.Partitioner{
| override def numPartitions: Int = numParts
| override def getPartition(key: Any): Int = {
| key.toString.toInt%numPartitions
| }
| }
defined class MyPartitioner scala> val rdd = rdd1.subtract(rdd2,new MyPartitioner(3))
rdd: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[13] at subtract at <console>:29 scala> rdd.foreachPartition(
| x=>{
| while(x.hasNext){
| println(x.next)
| }
| println("============")
| }
| )
3
============
1
4
============
2
============

zip

两个RDD对应位置(按顺序)组成key-value对创建新的RDD,两个RDD的元素在每个分区中数量必须相同,partition数量必须相同。
/**
* Zips this RDD with another one, returning key-value pairs with the first element in each RDD,
* second element in each RDD, etc. Assumes that the two RDDs have the *same number of
* partitions* and the *same number of elements in each partition* (e.g. one was made through
* a map on the other).
*/
def zip[U: ClassTag](other: RDD[U]): RDD[(T, U)]
val rdd1 = sc.parallelize(1 to 5,2)
val rdd2 = sc.parallelize(List("one","two","three","four","five"),2)
val rdd = rdd1.zip(rdd2)
rdd.collect
scala> val rdd1 = sc.parallelize(1 to 5,2)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[14] at parallelize at <console>:24 scala> val rdd2 = sc.parallelize(List("one","two","three","four","five"),2)
rdd2: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[15] at parallelize at <console>:24 scala> val rdd = rdd1.zip(rdd2)
rdd: org.apache.spark.rdd.RDD[(Int, String)] = ZippedPartitionsRDD2[16] at zip at <console>:28 scala> rdd.collect
res5: Array[(Int, String)] = Array((1,one), (2,two), (3,three), (4,four), (5,five))

combineByKey

/**
* Simplified version of combineByKeyWithClassTag that hash-partitions the resulting RDD using the
* existing partitioner/parallelism level. This method is here for backward compatibility. It
* does not provide combiner classtag information to the shuffle.
*
* @see [[combineByKeyWithClassTag]]
*/
def combineByKey[C](
createCombiner: V => C,
mergeValue: (C, V) => C,
mergeCombiners: (C, C) => C): RDD[(K, C)]
var rdd = sc.parallelize(Array(("A",1),("A",2),("A",3),("B",1),("B",2),("C",1)))
rdd.combineByKey(
(v : Int) => v + "$",
(c : String, v : Int) => c + "@" + v,
(c1 : String, c2 : String) => c1 + "||" + c2
).collect
scala> var rdd = sc.parallelize(Array(("A",1),("A",2),("A",3),("B",1),("B",2),("C",1)))
rdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[29] at parallelize at <console>:24
scala> rdd.combineByKey(
| (v : Int) => v + "$",
| (c : String, v : Int) => c + "@" + v,
| (c1 : String, c2 : String) => c1 + "||" + c2
| ).collect
res20: Array[(String, String)] = Array((B,1$@2), (A,1$@2@3), (C,1$))

没看明白啊!没看明白啊!没看明白啊!没看明白啊!没看明白啊!

flatMapValues

与flatMap类似,只是value经过函数f映射后得到1个或多个元素与key组成新的key-value,然后创建新的RDD。
/**
* Pass each value in the key-value pair RDD through a flatMap function without changing the
* keys; this also retains the original RDD's partitioning.
*/
def flatMapValues[U](f: V => TraversableOnce[U]): RDD[(K, U)]
var rdd = sc.parallelize(Array(("A",1),("A",2),("A",3),("B",1),("B",2),("C",1)))
rdd.flatMapValues(x => { x to 3}).collect
scala> var rdd = sc.parallelize(Array(("A",1),("A",2),("A",3),("B",1),("B",2),("C",1)))
rdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[32] at parallelize at <console>:24 scala> rdd.flatMapValues(x => { x to 3})
res24: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[33] at flatMapValues at <console>:27 scala> rdd.flatMapValues(x => { x to 3}).collect
res25: Array[(String, Int)] = Array((A,1), (A,2), (A,3), (A,2), (A,3), (A,3), (B,1), (B,2), (B,3), (B,2), (B,3), (C,1), (C,2), (C,3))

foldByKey

对每一个key的value进行聚合运算,其中zeroValue会与每一个key组成一个key-value对参与运算。
/**
* Merge the values for each key using an associative function and a neutral "zero value" which
* may be added to the result an arbitrary number of times, and must not change the result
* (e.g., Nil for list concatenation, 0 for addition, or 1 for multiplication.).
*/
def foldByKey(zeroValue: V)(func: (V, V) => V): RDD[(K, V)]
var rdd = sc.parallelize(Array(("A",1),("A",2),("A",3),("B",1),("B",2),("C",1)))
rdd.foldByKey(0)(_+_)
rdd.foldByKey(0)(_+_).collect
rdd.foldByKey(1)(_+_)
rdd.foldByKey(1)(_+_).collect
scala> var rdd = sc.parallelize(Array(("A",1),("A",2),("A",3),("B",1),("B",2),("C",1)))
rdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[35] at parallelize at <console>:24 scala> rdd.foldByKey(0)(_+_)
res26: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[36] at foldByKey at <console>:27 scala> rdd.foldByKey(0)(_+_).collect
res27: Array[(String, Int)] = Array((B,3), (A,6), (C,1)) scala> rdd.foldByKey(1)(_+_)
res28: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[38] at foldByKey at <console>:27 scala> rdd.foldByKey(1)(_+_).collect
res29: Array[(String, Int)] = Array((B,4), (A,7), (C,2))

keys

返回key-value的所有key.
/**
* Return an RDD with the keys of each tuple.
*/
def keys: RDD[K] = self.map(_._1)
scala> var rdd = sc.parallelize(Array(("A",1),("A",2),("A",3),("B",1),("B",2),("C",1)))
rdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[42] at parallelize at <console>:24 scala> rdd.keys.collect
res32: Array[String] = Array(A, A, A, B, B, C)

values

返回key-value的所有value.
/**
* Return an RDD with the values of each tuple.
*/
def values: RDD[V] = self.map(_._2)
scala> var rdd = sc.parallelize(Array(("A",1),("A",2),("A",3),("B",1),("B",2),("C",1)))
rdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[44] at parallelize at <console>:24 scala> rdd.values
res33: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[45] at values at <console>:27 scala> rdd.values.collect
res34: Array[Int] = Array(1, 2, 3, 1, 2, 1)

Spark RDD Transformation 简单用例(三)的更多相关文章

  1. Spark RDD Transformation 简单用例(二)

    aggregateByKey(zeroValue)(seqOp, combOp, [numTasks]) aggregateByKey(zeroValue)(seqOp, combOp, [numTa ...

  2. Spark RDD Transformation 简单用例(一)

    map(func) /** * Return a new RDD by applying a function to all elements of this RDD. */ def map[U: C ...

  3. Spark RDD Action 简单用例(一)

    collectAsMap(): Map[K, V] 返回key-value对,key是唯一的,如果rdd元素中同一个key对应多个value,则只会保留一个./** * Return the key- ...

  4. Spark RDD Action 简单用例(二)

    foreach(f: T => Unit) 对RDD的所有元素应用f函数进行处理,f无返回值./** * Applies a function f to all elements of this ...

  5. spark RDD transformation与action函数整理

    1.创建RDD val lines = sc.parallelize(List("pandas","i like pandas")) 2.加载本地文件到RDD ...

  6. spark rdd Transformation和Action 剖析

    1.看到 这篇总结的这么好, 就悄悄的转过来,供学习 wordcount.toDebugString查看RDD的继承链条 所以广义的讲,对任何函数进行某一项操作都可以认为是一个算子,甚至包括求幂次,开 ...

  7. PHP 下基于 php-amqp 扩展的 RabbitMQ 简单用例 (三) -- Header Exchange

    此模式下,消息的routing key 和队列的 routing key 会被完全忽略,而是在交换机推送消息和队列绑定交换机时, 分别为消息和队列设置 headers 属性, 通过匹配消息和队列的 h ...

  8. Spark RDD概念学习系列之RDD的依赖关系(宽依赖和窄依赖)(三)

    RDD的依赖关系?   RDD和它依赖的parent RDD(s)的关系有两种不同的类型,即窄依赖(narrow dependency)和宽依赖(wide dependency). 1)窄依赖指的是每 ...

  9. Apache Spark 2.2.0 中文文档 - Spark RDD(Resilient Distributed Datasets)论文 | ApacheCN

    Spark RDD(Resilient Distributed Datasets)论文 概要 1: 介绍 2: Resilient Distributed Datasets(RDDs) 2.1 RDD ...

随机推荐

  1. CDH:cdh5环境搭建

    安装环境三台centos7 vmw: cdh- 192.168.0.141 [主节点] cdh- 192.168.0.142 [从节点] cdh- 192.168.0.143 [从节点] 1)[各节点 ...

  2. 郑晔谈 Java 开发:新工具、新框架、新思维【转载】【整理】

    原文地址 导语:"我很惊讶地发现,现在许多程序员讨论的内容几乎和我十多年前刚开始做 Java 时几乎完全一样.要知道,我们生存的这个行业号称是变化飞快的.其实,这十几年时间,在开发领域已经有 ...

  3. 免费网络视频监控软件cmsclient

    http://www.brickcom.com/products/DetailView.php?modelname=CMS-Client&series=CMS#product-support ...

  4. JavaScript深入系列15篇

    JavaScirpt深入之从原型到原型链 构造函数创建对象 我们先使用构造函数创建一个对象: function Person() { } var person = new Person(); pers ...

  5. Winform开发框架之通用Windows摄像头调用拍照--SNF快速开发平台3.3-Spring.Net.Framework

    今天做了一个windows系统下调用摄像头.进行开启.关闭.拍照.设置等等功能演示. 进行源码贡献,欢迎大家下载使用 一.DEMO效果如下: 二.DEMO演示代码如下: using SNF.Utili ...

  6. 大数高精度计算库gmp简介

    1.编译安装,我用的ubuntu18.04 $sudo apt-get install m4 //默认没安装,gmp用这个 $tar -jvxf gmp-.tar.bz2 //解压 $cd gmp- ...

  7. Atitit 视图参数解决方案 oracle版和mysql版本 attilax总结.docx

    Atitit 视图参数解决方案 oracle版和mysql版本 attilax总结.docx 1.1. Package机制1 1.2. 全局变量机制1 1.3. 临时表模式,oracle mysql都 ...

  8. C#读取CSV

    public class CSVFileHelper { /// <summary> /// 将DataTable中数据写入到CSV文件中 /// </summary> /// ...

  9. 扩展layui中的自带字体图标

    项目中,虽然layui的字体图标库中已经有了1000多个图标了,但是也有时候不能满足我们自定义图标的需求,所以需要进行自定义,具体步骤如下: 1.在iconfont上找到自己喜欢的图标,也可以上传ui ...

  10. java框架篇---hibernate(多对多)映射关系

    以学生和老师为例的来讲解多对多映射. 实体类: Student package cn.itcast.g_hbm_manyToMany; import java.util.HashSet; import ...