What’s New, What’s Changed and How to get Started.

Are you ready for Apache Spark 2.0?

If you are just getting started with Apache Spark, the 2.0 release is the one to start with as the APIs have just gone through a major overhaul to improve ease-of-use.

If you are using an older version and want to learn what has changed then this article will give you the low down on why you should upgrade and what the impact to your code will be.

What’s new with Apache Spark 2.0?

Let’s start with the good news, and there’s plenty.

  • There are really only two programmatic APIs now; RDD and Dataset. For backwards compatibility, DataFrame still exists but is just a synonym for a Dataset.
  • Spark SQL has been improved to support a wider range of queries, including correlated subqueries. This was largely led by an effort to run TPC-DS benchmarks in Spark.
  • Performance is once again significantly improved thanks to advanced “whole stage code generation” when compiling query plans

CSV support is now built-in and based on the DataBricks spark-csv project, making it a breeze to create Datasets from CSV data with little coding.

Spark 2.0 is a major release, and there are some breaking changes that mean you may need to rewrite some of your code. Hereare some things we ran into when updating our apache-spark-examples.

  • For Scala users, SparkSession replaces SparkContext and SQLContext as the top-level context, but still provides access to SQLContext and SQLContext for backwards compatibility
  • DataFrame is now a synonym for Dataset[Row] and you can use these two types interchangeably,  although we recommend using the latter.
  • Performing a map() operation on a Dataset now returns a Dataset rather than an RDD, reducing the need to keep switching between the two APIs, and improving performance.
  • Some Java functional interfaces, such as FlatMapFunction, have been updated to return Iterator<T>rather than Iterable<T>.

Get help upgrading to Apache Spark 2.0 or making the transition from Java to Scala. Contact Us!

RDD vs. Dataset 2.0

Both the RDD API and the Dataset API represent data sets of a specific class. For instance, you can create an RDD[Person] as well as a Dataset[Person] so both can provide compile-time type-safety. Both can also be used with the generic Row structure provided in Spark for cases where classes might not exist that represent the data being manipulated, such as when reading CSV files.

RDDs can be used with any Java or Scala class and operate by manipulating those objects directly with all of the associated costs of object creation, serialization and garbage collection.

Datasets are limited to classes that implement the Scala Product trait, such as case classes. There is a very good reason for this limitation. Datasets store data in an optimized binary format, often in off-heap memory, to avoid the costs of deserialization and garbage collection. Even though it feels like you are coding against regular objects, Spark is really generating its own optimized byte-code for accessing the data directly.

RDD

 
1
2
3
// raw object manipulation
val rdd: RDD[Person] = …
val rdd2: RDD[String] = rdd.map(person => person.lastName)

Dataset

 
1
2
3
// optimized direct access to off-heap memory without deserializing objects
val ds: Dataset[Person] = …
val ds2: Dataset[String] = ds.map(person => person.lastName)

Getting Started with Scala

Here are some code samples to help you get started fast with Apache Spark 2.0 and Scala.

Creating SparkSession

SparkSession is now the starting point for a Spark driver program, instead of creating a SparkContext and a SQLContext.

 
1
2
3
4
5
6
7
8
val spark = SparkSession.builder
      .master("local[*]")
      .appName("Example")
      .getOrCreate()
 
// accessing legacy SparkContext and SQLContext
spark.sparkContext
spark.sqlContext

Creating a Dataset from a collection

SparkSession provides a createDataset method that accepts a collection.

 
1
var ds: Dataset[String] = spark.createDataset(List("one","two","three"))

Converting an RDD to a Dataset

SparkSession provides a createDataset method for converting an RDD to a Dataset. This only works if you import spark.implicits_ (where spark is the name of the SparkSession variable).

 
1
2
3
4
5
// always import implicits so that Spark can infer types when creating Datasets
import spark.implicits._
 
val rdd: RDD[Person] = ??? // assume this exists
val dataset: Dataset[Person] = spark.createDataset[Person](rdd)

Converting a DataFrame to a Dataset

A DataFrame (which is really a Dataset[Row]) can be converted to a Dataset of a specific class by performing a map() operation.

 
1
2
3
4
5
6
7
8
// read a text file into a DataFrame a.k.a. Dataset[Row]
var df: Dataset[Row] = spark.read.text("people.txt")
 
// use map() to convert to a Dataset of a specific class
var ds: Dataset[Person] = spark.read.text("people.txt")
      .map(row => parsePerson(row))
 
def parsePerson(row: Row) : Person = ??? // fill in parsing logic here

Reading a CSV directly as a Dataset

The built-in CSV support makes it easy to read a CSV and return a Dataset of a specific case class. This only works if the CSV contains a header row and the field names match the case class.

 
1
2
3
4
val ds: Dataset[Person] = spark.read
    .option("header","true")
    .csv("people.csv")
    .as[Person]

Getting Started with Java

Here are some code samples to help you get started fast with Spark 2.0 and Java.

Creating SparkSession

 
1
2
3
4
5
6
7
SparkSession spark = SparkSession.builder()
  .master("local[*]")
  .appName("Example")
  .getOrCreate();
 
  // Java still requires of the JavaSparkContext
  JavaSparkContext sc = new JavaSparkContext(spark.sparkContext());

Creating a Dataset from a collection

SparkSession provides a createDataset method that accepts a collection.

 
1
2
3
4
Dataset<Person> ds = spark.createDataset(
    Collections.singletonList(new Person(1, "Joe", "Bloggs")),
    Encoders.bean(Person.class)
);

Converting an RDD to a Dataset

SparkSession provides a createDataset method for converting an RDD to a Dataset.

 
1
2
3
4
Dataset<Person> ds = spark.createDataset(
  javaRDD.rdd(), // convert a JavaRDD to an RDD
  Encoders.bean(Person.class)
);

Converting a DataFrame to a Dataset

A DataFrame (which is really a Dataset[Row]) can be converted to a Dataset of a specific class by performing a map() operation.

 
1
2
3
4
5
6
7
8
Dataset<Person> ds = df.map(new MapFunction<Row, Person>() {
  @Override
  public Person call(Row value) throws Exception {
    return new Person(Integer.parseInt(value.getString(0)),
                      value.getString(1),
                      value.getString(2));
  }
}, Encoders.bean(Person.class));

Reading a CSV directly as a Dataset

The built-in CSV support makes it easy to read a CSV and return a Dataset of a specific case class. This only works if the CSV contains a header row and the field names match the case class.

 
1
2
3
4
Dataset<Person> ds = spark.read()
  .option("header", "true")
  .csv("testdata/people.csv")
  .as(Encoders.bean(Person.class));

Spark+Scala beats Spark+Java

Using Apache Spark with Java is harder than using Apache Spark with Scala and we spent significantly longer upgrading our Java examples than we did with our Scala examples, including running into some confusing runtime errors that were hard to track down (for example, we hit a runtime error with Spark’s code generation because one of our Java classes was not declared as public).

Also, we weren’t always able to use concise lambda functions even though we are using Java 8, and had to revert to anonymous inner classes with verbose (and confusing) syntax.

Conclusion

Spark 2.0 represents a significant milestone in the evolution of this open source project and provides cleaner APIs and improved performance compared to the 1.6 release.

The Scala API is a joy to code with, but the Java API can often be frustrating. It’s worth biting the bullet and switching to Scala.

Full source code for a number of examples is available from our github repo here.

Get help upgrading to Spark 2.0 or making the transition from Java to Scala. Contact Us!

APACHE SPARK 2.0 API IMPROVEMENTS: RDD, DATAFRAME, DATASET AND SQL的更多相关文章

  1. Apache Spark 2.0三种API的传说:RDD、DataFrame和Dataset

    Apache Spark吸引广大社区开发者的一个重要原因是:Apache Spark提供极其简单.易用的APIs,支持跨多种语言(比如:Scala.Java.Python和R)来操作大数据. 本文主要 ...

  2. Apache Spark 3.0 预览版正式发布,多项重大功能发布

    2019年11月08日 数砖的 Xingbo Jiang 大佬给社区发了一封邮件,宣布 Apache Spark 3.0 预览版正式发布,这个版本主要是为了对即将发布的 Apache Spark 3. ...

  3. Apache Spark 3.0 将内置支持 GPU 调度

    如今大数据和机器学习已经有了很大的结合,在机器学习里面,因为计算迭代的时间可能会很长,开发人员一般会选择使用 GPU.FPGA 或 TPU 来加速计算.在 Apache Hadoop 3.1 版本里面 ...

  4. spark的数据结构 RDD——DataFrame——DataSet区别

    转载自:http://blog.csdn.net/wo334499/article/details/51689549 RDD 优点: 编译时类型安全 编译时就能检查出类型错误 面向对象的编程风格 直接 ...

  5. Spark注册UDF函数,用于DataFrame DSL or SQL

    import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ object Test2 { def ...

  6. sparkSQL中RDD——DataFrame——DataSet的区别

    spark中RDD.DataFrame.DataSet都是spark的数据集合抽象,RDD针对的是一个个对象,但是DF与DS中针对的是一个个Row RDD 优点: 编译时类型安全 编译时就能检查出类型 ...

  7. There Are Now 3 Apache Spark APIs. Here’s How to Choose the Right One

    See Apache Spark 2.0 API Improvements: RDD, DataFrame, DataSet and SQL here. Apache Spark is evolvin ...

  8. RDD, DataFrame or Dataset

    总结: 1.RDD是一个Java对象的集合.RDD的优点是更面向对象,代码更容易理解.但在需要在集群中传输数据时需要为每个对象保留数据及结构信息,这会导致数据的冗余,同时这会导致大量的GC. 2.Da ...

  9. 且谈 Apache Spark 的 API 三剑客:RDD、DataFrame 和 Dataset

    作者:Jules S. Damji 译者:足下 本文翻译自 A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets ,翻译已 ...

随机推荐

  1. Jenkins 无法捕获构建脚本错误问题

    Jenkins 版本 2.121.1 编写构建脚本执行,发现脚本执行出错,不会中断构建过程,导致最后展现的构建结果是错误的. 原因:构建脚本头部加入 #!/bin/bash ,jenkins会将脚本放 ...

  2. Spring中bean实例化的三种方式

    之前我已经有好几篇博客介绍Spring框架了,不过当时我们都是使用注解来完成注入的,具体小伙伴可以参考这几篇博客(Spring&SpringMVC框架案例).那么今天我想来说说如何通过xml配 ...

  3. 带着萌新看springboot源码03

    上一节讲到了快速新建一个springboot应用,以及springboot的自动配置类起作用的时机,并且一起看了一个自动配置类的源码. 这一节我们来粗略看看当用户在浏览器输入一个url,怎么样返回一个 ...

  4. AVL树和平衡二叉树 平衡因子 右旋转LL 左旋转RR LR RL

    前言 今天要介绍几种高级数据结构AVL树,介绍之前AVL,会先说明平衡二叉树,并将树的学习路线进行总结,并介绍维持平衡的方法:右旋转.左旋转. 一.树学习路线 1.路线总结 总结了一下树的学习路线,如 ...

  5. PE知识复习之PE的各种头属性解析

    PE知识复习之PE的各种头属性解析 一丶DOS头结构体 typedef struct _IMAGE_DOS_HEADER { // DOS .EXE header WORD e_magic; // M ...

  6. javascript入门篇(一)

    未定义:undefined 布尔类型值:true,false 判断类型:typeof() 绝对值:Math.abs(-1) 声明常量:const   声明变量:var 小数两值互换如果出现问题,可以通 ...

  7. java continue break 关键字 详解 区别 用法 标记 标签 使用 示例 联系

    本文关键词: java continue break 关键字 详解 区别  用法 标记  标签 使用 示例 联系   跳出循环 带标签的continue和break 嵌套循环  深入continue ...

  8. springmvc 项目完整示例08 前台页面以及知识点总结

    至此已经基本测试成功了,我们稍作完善,让它成为一个更加完整的项目 我们现在重新规划下逻辑 两个页面 一个登录页面 一个欢迎页面 登陆页面输入账号密码,登陆成功的话,跳转登陆成功 欢迎页面 并且,更新用 ...

  9. vue webpack打包背景图片

    vue的背景图 和 img标签图大于10KB都不会转成base64处理,可以设置limit(不推荐),所以要设置一个公共路径,解决办法如下

  10. C# WebService创建与使用

    WebService现在虽然有更好的方式去替代它,但是它简单,开发成本低.很简单,对着本文一看就会 1.WebService的创建开发 2.WebService的发布测试 3.WebService的调 ...