Spark 2.0
Apache Spark 2.0: Faster, Easier, and Smarter
http://blog.madhukaraphatak.com/categories/spark-two/
https://amplab.cs.berkeley.edu/technical-preview-of-apache-spark-2-0-easier-faster-and-smarter/
Dataset - New Abstraction of Spark
For long, RDD was the standard abstraction of Spark.
But from Spark 2.0, Dataset will become the new abstraction layer for spark. Though RDD API will be available, it will become low level API, used mostly for runtime and library development. All user land code will be written against the Dataset abstraction and it’s subset Dataframe API.
2.0中,最关键的是在RDD这个low level抽象层上,又加了一组DataSet的high level的抽象层,让用户可以跟方便的开发
From Definition, ” A Dataset is a strongly typed collection of domain-specific objects that can be transformed in parallel using functional or relational operations. Each dataset also has an untyped view called a DataFrame, which is a Dataset of Row. “
which sounds similar to RDD definition
” RDD represents an immutable,partitioned collection of elements that can be operated on in parallel “
Dataset is a superset of Dataframe API which is released in Spark 1.3.
Dataframe是一种特殊化的Dataset,Dataframe = Dataset[row]
SparkSession - New entry point of Spark
In earlier versions of spark, spark context was entry point for Spark. As RDD was main API, it was created and manipulated using context API’s. For every other API,we needed to use different contexts.For streaming, we needed StreamingContext, for SQL sqlContext and for hive HiveContext. But as DataSet and Dataframe API’s are becoming new standard API’s we need an entry point build for them. So in Spark 2.0, we have a new entry point for DataSet and Dataframe API’s called as Spark Session.
因为有了新的抽象层,所以需要加新的入口,SparkSession
就像RDD对应于SparkContext
更强大的SQL支持
On the SQL side, we have significantly expanded the SQL capabilities of Spark, with the introduction of a new ANSI SQL parser and support for subqueries.
Spark 2.0 can run all the 99 TPC-DS queries, which require many of the SQL:2003 features.
Tungsten 2.0
Spark 2.0 ships with the second generation Tungsten engine.
This engine builds upon ideas from modern compilers and MPP databases and applies them to data processing.
The main idea is to emit optimized bytecode at runtime that collapses the entire query into a single function, eliminating virtual function calls and leveraging CPU registers for intermediate data. We call this technique “whole-stage code generation.”
在内存管理和基于CPU的性能优化上,faster
Structured Streaming
Spark 2.0’s Structured Streaming APIs is a novel way to approach streaming.
It stems from the realization that the simplest way to compute answers on streams of data is to not having to reason about the fact that it is a stream.
意思就是说,不要意识到流,流就是个无限的DataFrames
这个是典型的spark的思路,batch是一切的根本,和Flink截然相反
Spark 2.0的更多相关文章
- Spark 1.0.0 横空出世 Spark on Yarn 部署(Hadoop 2.4)
就在昨天,北京时间5月30日20点多.Spark 1.0.0最终公布了:Spark 1.0.0 released 依据官网描写叙述,Spark 1.0.0支持SQL编写:Spark SQL Progr ...
- APACHE SPARK 2.0 API IMPROVEMENTS: RDD, DATAFRAME, DATASET AND SQL
What’s New, What’s Changed and How to get Started. Are you ready for Apache Spark 2.0? If you are ju ...
- Apache Spark 3.0 将内置支持 GPU 调度
如今大数据和机器学习已经有了很大的结合,在机器学习里面,因为计算迭代的时间可能会很长,开发人员一般会选择使用 GPU.FPGA 或 TPU 来加速计算.在 Apache Hadoop 3.1 版本里面 ...
- spark 2.0.0集群安装与hive on spark配置
1. 环境准备: JDK1.8 hive 2.3.4 hadoop 2.7.3 hbase 1.3.3 scala 2.11.12 mysql5.7 2. 下载spark2.0.0 cd /home/ ...
- Spark 2.0 PCA主成份分析
PCA在Spark2.0中用法比较简单,只需要设置: .setInputCol(“features”)//保证输入是特征值向量 .setOutputCol(“pcaFeatures”)//输出 .se ...
- Apache Spark 2.0三种API的传说:RDD、DataFrame和Dataset
Apache Spark吸引广大社区开发者的一个重要原因是:Apache Spark提供极其简单.易用的APIs,支持跨多种语言(比如:Scala.Java.Python和R)来操作大数据. 本文主要 ...
- Spark 2.0 DataFrame map操作中Unable to find encoder for type stored in a Dataset.问题的分析与解决
转载:http://blog.csdn.net/sparkexpert/article/details/52871000 随着新版本的spark已经逐渐稳定,最近拟将原有框架升级到spark 2.0. ...
- Spark 2.0.0 SPARK-SQL returns NPE Error
com.esotericsoftware.kryo.KryoException: java.lang.NullPointerExceptionSerialization trace:underlyin ...
- Apache Spark 3.0 预览版正式发布,多项重大功能发布
2019年11月08日 数砖的 Xingbo Jiang 大佬给社区发了一封邮件,宣布 Apache Spark 3.0 预览版正式发布,这个版本主要是为了对即将发布的 Apache Spark 3. ...
随机推荐
- BZOJ 1072: [SCOI2007]排列perm 状态压缩DP
1072: [SCOI2007]排列perm Description 给一个数字串s和正整数d, 统计s有多少种不同的排列能被d整除(可以有前导0).例如123434有90种排列能被2整除,其中末位为 ...
- Android WebView访问SSL证书网页(onReceivedSslError)
Android WebView访问https SSL证书网页,如淘宝,需要在onReceivedSslError添加SSL支持 webview.setWebViewClient(new WebView ...
- 关于post和get传递参数的区别
今天一朋友给我一段代码,说使用request.querystrin得不到传过来的值,我们一起找到很长时间,终于给找到了,错误的原因是他将form中的method参数写成了get了,所以使用reques ...
- mvc-2事件监听
现代浏览器都支持的事件 click dbclick mouseover mousemove mouseout focus blur change(表单输入框特有) submit(表单特有) addEv ...
- 使用INFORMATION_SCHEMA.Columns查询数据表结构
使用情形1:写数据字典(Word文档)时, 需要获取表结构,以加快书写速度,当然,前提是没有文档导出工具. 使用情形2:生成实体时, 前提也是没有代码生成工具. 脚本 SELECT A.COLUMN_ ...
- BZOJ1409 : Password
$f[n]\bmod q=p^{Fib[n]}\bmod q=p^{Fib[n]\bmod\varphi(q)}\bmod q$ 首先线性筛预处理出所有素数,然后对于每次询问,求出$\varphi(q ...
- CC150 - 11.3
Question: Given a sorted array of n integers that has been rotated an unknown number of times, write ...
- js-sdk探索之微信网页分享
微信是一个很不错的传播平台,最近公司需要做一个新年贺卡,使用html5制作一个很小的动画,然后发送给客户,不需要和后台有任何的联系,一个很简单的功能,需要利用微信的分享功能,毕竟微信分享的带小图片.简 ...
- 安装win7 ubuntu双系统
http://www.cnblogs.com/shengansong/archive/2011/10/23/2221716.html http://www.cnblogs.com/shenganson ...
- 使用ProxychainsMac下安装及配置
下面几种解决方式 一.先在VPS用composer把Laravel给拖回来,本地你就别想用Composer正常下载Laravel回来了 二.使用全局代理,暂时木有折腾过在终端下怎么折腾,GUI可以使用 ...