APACHE SPARK 2.0 API IMPROVEMENTS: RDD, DATAFRAME, DATASET AND SQL
What’s New, What’s Changed and How to get Started.
Are you ready for Apache Spark 2.0?
If you are just getting started with Apache Spark, the 2.0 release is the one to start with as the APIs have just gone through a major overhaul to improve ease-of-use.
If you are using an older version and want to learn what has changed then this article will give you the low down on why you should upgrade and what the impact to your code will be.
What’s new with Apache Spark 2.0?
Let’s start with the good news, and there’s plenty.
- There are really only two programmatic APIs now; RDD and Dataset. For backwards compatibility, DataFrame still exists but is just a synonym for a Dataset.
- Spark SQL has been improved to support a wider range of queries, including correlated subqueries. This was largely led by an effort to run TPC-DS benchmarks in Spark.
- Performance is once again significantly improved thanks to advanced “whole stage code generation” when compiling query plans
CSV support is now built-in and based on the DataBricks spark-csv project, making it a breeze to create Datasets from CSV data with little coding.
Spark 2.0 is a major release, and there are some breaking changes that mean you may need to rewrite some of your code. Hereare some things we ran into when updating our apache-spark-examples.
- For Scala users, SparkSession replaces SparkContext and SQLContext as the top-level context, but still provides access to SQLContext and SQLContext for backwards compatibility
- DataFrame is now a synonym for Dataset[Row] and you can use these two types interchangeably, although we recommend using the latter.
- Performing a map() operation on a Dataset now returns a Dataset rather than an RDD, reducing the need to keep switching between the two APIs, and improving performance.
- Some Java functional interfaces, such as FlatMapFunction, have been updated to return Iterator<T>rather than Iterable<T>.
Get help upgrading to Apache Spark 2.0 or making the transition from Java to Scala. Contact Us!
RDD vs. Dataset 2.0
Both the RDD API and the Dataset API represent data sets of a specific class. For instance, you can create an RDD[Person] as well as a Dataset[Person] so both can provide compile-time type-safety. Both can also be used with the generic Row structure provided in Spark for cases where classes might not exist that represent the data being manipulated, such as when reading CSV files.
RDDs can be used with any Java or Scala class and operate by manipulating those objects directly with all of the associated costs of object creation, serialization and garbage collection.
Datasets are limited to classes that implement the Scala Product trait, such as case classes. There is a very good reason for this limitation. Datasets store data in an optimized binary format, often in off-heap memory, to avoid the costs of deserialization and garbage collection. Even though it feels like you are coding against regular objects, Spark is really generating its own optimized byte-code for accessing the data directly.
RDD
1
2
3
|
// raw object manipulation
val rdd: RDD[Person] = …
val rdd2: RDD[String] = rdd.map(person => person.lastName)
|
Dataset
1
2
3
|
// optimized direct access to off-heap memory without deserializing objects
val ds: Dataset[Person] = …
val ds2: Dataset[String] = ds.map(person => person.lastName)
|
Getting Started with Scala
Here are some code samples to help you get started fast with Apache Spark 2.0 and Scala.
Creating SparkSession
SparkSession is now the starting point for a Spark driver program, instead of creating a SparkContext and a SQLContext.
1
2
3
4
5
6
7
8
|
val spark = SparkSession.builder
.master("local[*]")
.appName("Example")
.getOrCreate()
// accessing legacy SparkContext and SQLContext
spark.sparkContext
spark.sqlContext
|
Creating a Dataset from a collection
SparkSession provides a createDataset method that accepts a collection.
1
|
var ds: Dataset[String] = spark.createDataset(List("one","two","three"))
|
Converting an RDD to a Dataset
SparkSession provides a createDataset method for converting an RDD to a Dataset. This only works if you import spark.implicits_ (where spark is the name of the SparkSession variable).
1
2
3
4
5
|
// always import implicits so that Spark can infer types when creating Datasets
import spark.implicits._
val rdd: RDD[Person] = ??? // assume this exists
val dataset: Dataset[Person] = spark.createDataset[Person](rdd)
|
Converting a DataFrame to a Dataset
A DataFrame (which is really a Dataset[Row]) can be converted to a Dataset of a specific class by performing a map() operation.
1
2
3
4
5
6
7
8
|
// read a text file into a DataFrame a.k.a. Dataset[Row]
var df: Dataset[Row] = spark.read.text("people.txt")
// use map() to convert to a Dataset of a specific class
var ds: Dataset[Person] = spark.read.text("people.txt")
.map(row => parsePerson(row))
def parsePerson(row: Row) : Person = ??? // fill in parsing logic here
|
Reading a CSV directly as a Dataset
The built-in CSV support makes it easy to read a CSV and return a Dataset of a specific case class. This only works if the CSV contains a header row and the field names match the case class.
1
2
3
4
|
val ds: Dataset[Person] = spark.read
.option("header","true")
.csv("people.csv")
.as[Person]
|
Getting Started with Java
Here are some code samples to help you get started fast with Spark 2.0 and Java.
Creating SparkSession
1
2
3
4
5
6
7
|
SparkSession spark = SparkSession.builder()
.master("local[*]")
.appName("Example")
.getOrCreate();
// Java still requires of the JavaSparkContext
JavaSparkContext sc = new JavaSparkContext(spark.sparkContext());
|
Creating a Dataset from a collection
SparkSession provides a createDataset method that accepts a collection.
1
2
3
4
|
Dataset<Person> ds = spark.createDataset(
Collections.singletonList(new Person(1, "Joe", "Bloggs")),
Encoders.bean(Person.class)
);
|
Converting an RDD to a Dataset
SparkSession provides a createDataset method for converting an RDD to a Dataset.
1
2
3
4
|
Dataset<Person> ds = spark.createDataset(
javaRDD.rdd(), // convert a JavaRDD to an RDD
Encoders.bean(Person.class)
);
|
Converting a DataFrame to a Dataset
A DataFrame (which is really a Dataset[Row]) can be converted to a Dataset of a specific class by performing a map() operation.
1
2
3
4
5
6
7
8
|
Dataset<Person> ds = df.map(new MapFunction<Row, Person>() {
@Override
public Person call(Row value) throws Exception {
return new Person(Integer.parseInt(value.getString(0)),
value.getString(1),
value.getString(2));
}
}, Encoders.bean(Person.class));
|
Reading a CSV directly as a Dataset
The built-in CSV support makes it easy to read a CSV and return a Dataset of a specific case class. This only works if the CSV contains a header row and the field names match the case class.
1
2
3
4
|
Dataset<Person> ds = spark.read()
.option("header", "true")
.csv("testdata/people.csv")
.as(Encoders.bean(Person.class));
|
Spark+Scala beats Spark+Java
Using Apache Spark with Java is harder than using Apache Spark with Scala and we spent significantly longer upgrading our Java examples than we did with our Scala examples, including running into some confusing runtime errors that were hard to track down (for example, we hit a runtime error with Spark’s code generation because one of our Java classes was not declared as public).
Also, we weren’t always able to use concise lambda functions even though we are using Java 8, and had to revert to anonymous inner classes with verbose (and confusing) syntax.
Conclusion
Spark 2.0 represents a significant milestone in the evolution of this open source project and provides cleaner APIs and improved performance compared to the 1.6 release.
The Scala API is a joy to code with, but the Java API can often be frustrating. It’s worth biting the bullet and switching to Scala.
Full source code for a number of examples is available from our github repo here.
Get help upgrading to Spark 2.0 or making the transition from Java to Scala. Contact Us!
APACHE SPARK 2.0 API IMPROVEMENTS: RDD, DATAFRAME, DATASET AND SQL的更多相关文章
- Apache Spark 2.0三种API的传说:RDD、DataFrame和Dataset
Apache Spark吸引广大社区开发者的一个重要原因是:Apache Spark提供极其简单.易用的APIs,支持跨多种语言(比如:Scala.Java.Python和R)来操作大数据. 本文主要 ...
- Apache Spark 3.0 预览版正式发布,多项重大功能发布
2019年11月08日 数砖的 Xingbo Jiang 大佬给社区发了一封邮件,宣布 Apache Spark 3.0 预览版正式发布,这个版本主要是为了对即将发布的 Apache Spark 3. ...
- Apache Spark 3.0 将内置支持 GPU 调度
如今大数据和机器学习已经有了很大的结合,在机器学习里面,因为计算迭代的时间可能会很长,开发人员一般会选择使用 GPU.FPGA 或 TPU 来加速计算.在 Apache Hadoop 3.1 版本里面 ...
- spark的数据结构 RDD——DataFrame——DataSet区别
转载自:http://blog.csdn.net/wo334499/article/details/51689549 RDD 优点: 编译时类型安全 编译时就能检查出类型错误 面向对象的编程风格 直接 ...
- Spark注册UDF函数,用于DataFrame DSL or SQL
import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ object Test2 { def ...
- sparkSQL中RDD——DataFrame——DataSet的区别
spark中RDD.DataFrame.DataSet都是spark的数据集合抽象,RDD针对的是一个个对象,但是DF与DS中针对的是一个个Row RDD 优点: 编译时类型安全 编译时就能检查出类型 ...
- There Are Now 3 Apache Spark APIs. Here’s How to Choose the Right One
See Apache Spark 2.0 API Improvements: RDD, DataFrame, DataSet and SQL here. Apache Spark is evolvin ...
- RDD, DataFrame or Dataset
总结: 1.RDD是一个Java对象的集合.RDD的优点是更面向对象,代码更容易理解.但在需要在集群中传输数据时需要为每个对象保留数据及结构信息,这会导致数据的冗余,同时这会导致大量的GC. 2.Da ...
- 且谈 Apache Spark 的 API 三剑客:RDD、DataFrame 和 Dataset
作者:Jules S. Damji 译者:足下 本文翻译自 A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets ,翻译已 ...
随机推荐
- PL/SQL基础语法入门
先前安装了PL/SQL软件 PL/SQL全称为Procedural Language/SQL. PL/SQL也是一种程序语言,叫做过程化SQL语言,是Oracle数据库对SQL语句的扩展 打PL/SQ ...
- [六] 函数式接口的复合方法示例 predicate 谓词逻辑运算 Function接口 组合运算 比较器 逆序 比较链
复合的方法 有些函数式接口提供了允许复合的方法 也就是可以将Lambda表达式复合成为一个更加复杂的方法 之前的章节中有说到: 接口中的compose, andThen, and, or, negat ...
- DAL分页
using System;using System.Collections.Generic;using LModel.DTO;using Newtonsoft.Json;using System.Da ...
- PHP Composer 依赖管理的用法
1:下载 1.1:方法一: 通过PHP来安装 cd G:\web\es6 curl -sS https://getcomposer.org/installer | php #这个命令会下载compos ...
- 如何搭建一个VUE项目
搭建环境 搭建node环境 下载 1.进入node.js官方网站下载页,点击下图中框出位置,进行下载即可,当前版本为8.9.4,下载网址为:https://nodejs.org/zh-cn/downl ...
- 开源项目商业分析实例(1) - MonicaHQ
本来写一篇开源商业模式的稿子,因为有四大主题,这个稿子有点大,导致现在半个月过去了,都还没有憋出来. 今天想想还是采用MVP(minimum viable product,最小化可行产品)模式吧. ...
- 从.Net到Java学习第一篇——开篇
以前我常说,公司用什么技术我就学什么.可是对于java,我曾经一度以为“学java是不可能的,这辈子不可能学java的.”结果,一遇到公司转java,我就不得不跑路了,于是乎,回头一看N家公司交过社保 ...
- MySQL 常用指令小结
l 创建数据库:CREATE DATABASE table_name; l 删除数据库:DROP DATABASE table_name; l 展示数据库:SHOW DATABASE; l 选 ...
- Android 轮播图Banner切换图片的效果
Android XBanner使用详解 2018年03月14日 08:19:59 AND_Devil 阅读数:910 版权声明:本文为博主原创文章,未经博主允许不得转载. https://www. ...
- ASP.NET Core 入门教程 7、ASP.NET Core MVC 分部视图入门
一.前言 1.本教程主要内容 ASP.NET Core MVC (Razor)分部视图简介 ASP.NET Core MVC (Razor)分部视图基础教程 ASP.NET Core MVC (Raz ...