RDD、DataFrame、Dataset三者三者之间转换
转化: RDD、DataFrame、Dataset三者有许多共性,有各自适用的场景常常需要在三者之间转换 DataFrame/Dataset转RDD: 这个转换很简单 val rdd1=testDF.rdd
val rdd2=testDS.rdd RDD转DataFrame: import spark.implicits._
val testDF = rdd.map {line=>
(line._1,line._2)
}.toDF("col1","col2") 一般用元组把一行的数据写在一起,然后在toDF中指定字段名 RDD转Dataset:
import spark.implicits._
case class Coltest(col1:String,col2:Int)extends Serializable //定义字段名和类型
val testDS = rdd.map {line=>
Coltest(line._1,line._2)
}.toDS 可以注意到,定义每一行的类型(case class)时,已经给出了字段名和类型,后面只要往case class里面添加值即可 Dataset转DataFrame: 这个也很简单,因为只是把case class封装成Row import spark.implicits._
val testDF = testDS.toDF DataFrame转Dataset: import spark.implicits._
case class Coltest(col1:String,col2:Int)extends Serializable //定义字段名和类型
val testDS = testDF.as[Coltest] 这种方法就是在给出每一列的类型后,使用as方法,转成Dataset,这在数据类型是DataFrame又需要针对各个字段处理时极为方便
特别注意: 在使用一些特殊的操作时,一定要加上 import spark.implicits._ 不然toDF、toDS无法使用
package dataframe
import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
//
// Explore interoperability between DataFrame and Dataset. Note that Dataset
// is covered in much greater detail in the 'dataset' directory.
//
object DatasetConversion {
case class Cust(id: Integer, name: String, sales: Double, discount: Double, state: String)
case class StateSales(state: String, sales: Double)
def main(args: Array[String]) {
val spark =
SparkSession.builder()
.appName("DataFrame-DatasetConversion")
.master("local[4]")
.getOrCreate()
import spark.implicits._
// create a sequence of case class objects
// (we defined the case class above)
val custs = Seq(
Cust(1, "Widget Co", 120000.00, 0.00, "AZ"),
Cust(2, "Acme Widgets", 410500.00, 500.00, "CA"),
Cust(3, "Widgetry", 410500.00, 200.00, "CA"),
Cust(4, "Widgets R Us", 410500.00, 0.0, "CA"),
Cust(5, "Ye Olde Widgete", 500.00, 0.0, "MA")
)
// Create the DataFrame without passing through an RDD
val customerDF : DataFrame = spark.createDataFrame(custs)
//
// println("*** DataFrame schema")
//
// customerDF.printSchema()
//
// println("*** DataFrame contents")
//
// customerDF.show()
// +---+---------------+--------+--------+-----+
//| id| name| sales|discount|state|
//+---+---------------+--------+--------+-----+
//| 1| Widget Co|120000.0| 0.0| AZ|
//| 2| Acme Widgets|410500.0| 500.0| CA|
//| 3| Widgetry|410500.0| 200.0| CA|
//| 4| Widgets R Us|410500.0| 0.0| CA|
//| 5|Ye Olde Widgete| 500.0| 0.0| MA|
//+---+---------------+--------+--------+-----+
//
// println("*** Select and filter the DataFrame")
//
val smallerDF =
customerDF.select("sales", "state").filter($"state".equalTo("CA"))
//
// smallerDF.show()
//
// +--------+-----+
//| sales|state|
//+--------+-----+
//|410500.0| CA|
//|410500.0| CA|
//|410500.0| CA|
//+--------+-----+
///////////////////////////////////////////////////////////////////////////////////
// Convert it to a Dataset by specifying the type of the rows -- use a case
// class because we have one and it's most convenient to work with. Notice
// you have to choose a case class that matches the remaining columns.
// BUT also notice that the columns keep their order from the DataFrame --
// later you'll see a Dataset[StateSales] of the same type where the
// columns have the opposite order, because of the way it was created.
val customerDS : Dataset[StateSales] = smallerDF.as[StateSales]
//
// println("*** Dataset schema")
//
// customerDS.printSchema()
//
// println("*** Dataset contents")
//
// customerDS.show()
// Select and other operations can be performed directly on a Dataset too,
// but be careful to read the documentation for Dataset -- there are
// "typed transformations", which produce a Dataset, and
// "untyped transformations", which produce a DataFrame. In particular,
// you need to project using a TypedColumn to gate a Dataset.
// val verySmallDS : Dataset[Double] = customerDS.select($"sales".as[Double])
//
// println("*** Dataset after projecting one column")
//
// verySmallDS.show()
//
//+--------+
//| sales|
//+--------+
//|410500.0|
//|410500.0|
//|410500.0|
//+--------+
// If you select multiple columns on a Dataset you end up with a Dataset
// of tuple type, but the columns keep their names.
val tupleDS : Dataset[(String, Double)] =
customerDS.select($"state".as[String], $"sales".as[Double])
//
// println("*** Dataset after projecting two columns -- tuple version")
//
// tupleDS.show()
//
//+-----+--------+
//|state| sales|
//+-----+--------+
//| CA|410500.0|
//| CA|410500.0|
//| CA|410500.0|
//+-----+--------+
// You can also cast back to a Dataset of a case class. Notice this time
// the columns have the opposite order than the last Dataset[StateSales]
// val betterDS: Dataset[StateSales] = tupleDS.as[StateSales]
//
// println("*** Dataset after projecting two columns -- case class version")
//
// betterDS.show()
//
//+-----+--------+
//|state| sales|
//+-----+--------+
//| CA|410500.0|
//| CA|410500.0|
//| CA|410500.0|
//+-----+--------+
// Converting back to a DataFrame without making other changes is really easy
// val backToDataFrame : DataFrame = tupleDS.toDF()
//
// println("*** This time as a DataFrame")
//
// backToDataFrame.show()
//
//+-----+--------+
//|state| sales|
//+-----+--------+
//| CA|410500.0|
//| CA|410500.0|
//| CA|410500.0|
//+-----+--------+
//
// // While converting back to a DataFrame you can rename the columns
val renamedDataFrame : DataFrame = tupleDS.toDF("MyState", "MySales")
println("*** Again as a DataFrame but with renamed columns")
renamedDataFrame.show()
// +-------+--------+
//|MyState| MySales|
//+-------+--------+
//| CA|410500.0|
//| CA|410500.0|
//| CA|410500.0|
//+-------+--------+
}
}
RDD、DataFrame、Dataset三者三者之间转换的更多相关文章
- APACHE SPARK 2.0 API IMPROVEMENTS: RDD, DATAFRAME, DATASET AND SQL
What’s New, What’s Changed and How to get Started. Are you ready for Apache Spark 2.0? If you are ju ...
- spark的数据结构 RDD——DataFrame——DataSet区别
转载自:http://blog.csdn.net/wo334499/article/details/51689549 RDD 优点: 编译时类型安全 编译时就能检查出类型错误 面向对象的编程风格 直接 ...
- sparkSQL中RDD——DataFrame——DataSet的区别
spark中RDD.DataFrame.DataSet都是spark的数据集合抽象,RDD针对的是一个个对象,但是DF与DS中针对的是一个个Row RDD 优点: 编译时类型安全 编译时就能检查出类型 ...
- RDD, DataFrame or Dataset
总结: 1.RDD是一个Java对象的集合.RDD的优点是更面向对象,代码更容易理解.但在需要在集群中传输数据时需要为每个对象保留数据及结构信息,这会导致数据的冗余,同时这会导致大量的GC. 2.Da ...
- spark rdd df dataset
RDD.DataFrame.DataSet的区别和联系 共性: 1)都是spark中得弹性分布式数据集,轻量级 2)都是惰性机制,延迟计算 3)根据内存情况,自动缓存,加快计算速度 4)都有parti ...
- byte[] 、Bitmap与Drawbale 三者直接的转换
经常遇到这种类似头疼的问题 byte[] .Bitmap与Drawbale 三者直接的转换 1.byte[] ->Bitmap Bitmap Bitmap = BitmapFactory.dec ...
- Spark入门之DataFrame/DataSet
目录 Part I. Gentle Overview of Big Data and Spark Overview 1.基本架构 2.基本概念 3.例子(可跳过) Spark工具箱 1.Dataset ...
- C#中对象,字符串,dataTable、DataReader、DataSet,对象集合转换成Json字符串方法。
C#中对象,字符串,dataTable.DataReader.DataSet,对象集合转换成Json字符串方法. public class ConvertJson { #region 私有方法 /// ...
- 关于不同进制数之间转换的数学推导【Written By KillerLegend】
关于不同进制数之间转换的数学推导 涉及范围:正整数范围内二进制(Binary),八进制(Octonary),十进制(Decimal),十六进制(hexadecimal)之间的转换 数的进制有多种,比如 ...
随机推荐
- Sleep 等待连接攻击
Sleep The thread is waiting for the client to send a new statement to it. https://dev.mysql.com/doc/ ...
- php之二叉树
二叉树的特点: ①.每个节点最多有两个子树,所以二叉树中不存在度大于2的节点.注意不是只有两个子树,最多有两个子树,没有子树或者只有一颗子树都是可以的. ②左子树和右子树是有顺序的. ③即使树中只有一 ...
- .2 Git 分支 - 分支的新建与合并
分支的新建与合并 https://git-scm.com/book/zh/v1/Git-%E5%88%86%E6%94%AF-%E5%88%86%E6%94%AF%E7%9A%84%E6%96%B0% ...
- Beanstalkd 基本概念和使用
1:什么是 Beanstalkd ? Beanstalkd 一个高性能.轻量级的分布式内存队列系统 简单来说,就是一个队列,相比于 数据库/redis 队列相比. 更专业.能完成的功能更多.就这么理解 ...
- Gson使用技巧
1. CharMatcher String serviceUrl = CharMatcher.is('/').trimTrailingFrom(ConfigHelper.metaServiceUrl( ...
- 写出简洁的Python代码: 使用Exceptions(转)
add by zhj: 非常好的文章,异常在Python的核心代码中使用的非常广泛,超出一般人的想象,比如迭代器中,当我们用for遍历一个可迭代对象时, Python是如何判断遍历结束的呢?是使用的S ...
- pl/sql中文乱码
增加系统变量变量名:NLS_LANG变量值:SIMPLIFIED CHINESE_CHINA.ZHS16GBK
- 10.7-uC/OS-III内部任务(定时器任务 OS_TmrTask())
{这节所说的定时器都是软件定时器} 1.uC/OS-III为用户提供了定时器任务,相应代码在OS_TMR.C中.定时器任务是可选的,通过将OS_CFG.H中的OS_CFG_TMR_EN设置为1使能.当 ...
- SpringBoot-热部署Devtools
热部署 什么是热部署 所谓的热部署:比如项目的热部署,就是在应用程序在不停止的情况下,实现新的部署 项目演示案例 @RestController @Slf4j public class IndexCo ...
- webmin小结
centos7安装webmin https://www.cnblogs.com/andy9468/p/10537201.html webmin重置密码 重置webmin账户root的密码为例: htt ...