chapter 1 introduction to the analysis with spark

the conponents of Sparks

  spark core(contains the basic  functionality of sparks. spark Core  is also the  home to the APIs that defines the RDDs),

  spark sql(structured data ) is the package  for working with the structured data.it allow query  data via SQL as well as Apache hive , and it support many sources of data ,including Hive tables ,Parquet And jason.also allow developers to intermix SQL queries with the programatic data manipulation supported By the RDDs in Python ,java And Scala .

  spark streaming(real-time),enables processing the live of streaming of data.

  MLib(machine learning)

  GraphX(graph processing )is a library for manipulating the graph .

A Brief History of Spark

  spark is a open source project that has beed And is maintained By a thriving And diverse community  of developer .

chapter 2 downloading spark And getting started

  walking through the process of downloding And running the sprak on local mode on single computer .

  you don't needmaster Scala,java orPython.Spark itself is written in Scala, and runs on the Java Virtual Machine (JVM). To run Spark
on either your laptop or a cluster, all you need is an installation of Java 6 or newer. If you wish to use the Python API you will also need a Python interpreter (version 2.6 or newer).Spark does not yet work with Python 3.

downloading spark,select the "pre-build for Hadoop 2.4 And later".

tips:

widows user May run into issues installing .you can use the ziptool untar the .tar file Note :instatll spark in a directionalry with no space (e.g. c:\spark).

after you untar you will get a new directionaru with the same name but without the final .tar suffix .

damn it:

Most of this book includes code in all of Spark’s languages, but interactive shells are
available only in Python and Scala. Because a shell is very useful for learning the API, we recommend using one of these languages for these examples even if you are a Java
developer. The API is similar in every language.

change the directionaty to the spark,type bin\pyspark,you will see the logo.

Introduction  to Core spark concepts

Driver program

  |----your application

  |----distributed datasets that you defined

  usually weapply many operations on thedatasets.

  ***in the preceding example ,the Driver program was the spark she'll intself ,And you can type in the operation that you wanted.

  ***Driver program access the spark through a SparkContext object,which representing the  connection  to  a  computing cluster. what's  more  the  sparkcontext is automatically created for you called as  sc,in the  pyspark ,you can print  the infomation  about this  Object  By  typing "sc".well i think you will  know  the SparkContext have 3 kinds in java ,Python And Scala respectively.

SparkContext  have  many operations ,such as count(),first() and so on. Driver program typically manage a number of nodes called executors. when you call any operation on a cluster different machines might count in different ranges of the file.beacuse we run the spark shell locally,it execute all works on a single machine.

Passing Funtions to Spark

  look at the following example in python:

 lines = sc.textFile(""README.md);
pythonLines = lines.filter(lambda line : "Python" in line);
pythoLines.first();

if you are unfamilat with the lambda sytax. it's a shorthand way to define function inline in Python or Scala, then pass the function's name to the Spark. you can do like this :

def hasPython(line): # this function judge that wheter every line contain the "Python" in a file

    return "Python" in line.

pythonLines = lines.filter(hasPython)

of course you can write in java.but they are defined as classes, implementing interface Funtion:

JavaRDD<String> pythonLines = filter(new Funtion<String, Bollean>()
{
Boolean call(String line)
{
return lines.contains(line);
}
});

nowadays java8 have supported lambda.

Spark qutomatically takes your functions (e.g. lines.contains("Python"))and ships it to executors nodes. Thus, you can write code in a single driver program and automatically have parts of it run on mutiple nodes.

 Standalone Applications

Apart from running Spark interactively, Spark can be linked to standalone applications in either Java, Python or Scala. The main difference from using it in the shell is that you need to initialize your SparkContext.After that ,the functions is same.Remember , if you using it in the shell, the SparkContext is created automatically for you called "sc", you can use it direactly.

The proces linked to Spark varies from languages. In Java and Scala , you give your aplliation Maven dependency on the spark-core artifact.Maven is a popular package managerment tool for java-based languages let you link to libaries in public repositories.you can use Maven itself build your projet , or use other tools that can talk to the Maven repositories, including Scala's sbt od Gradle. Popular IDE like Eclipse also allow you to directly add a Maven dependency to a  project.

In Python,you simply write application as Python scripts, but you must run them using the bin\spark-submit script included in Spark. The spark-submit script include the dependency for us in Python,what's more it sets up the enrivonment for Spark's Python API to function. Simply run your scripts like this:

bin\spak-submit my_script.py

(Note that you will have to use backslashes instead of forward slashes on Window)s

读learning spark lighting chapter1~chapter2的更多相关文章

  1. 【原】Learning Spark (Python版) 学习笔记(一)----RDD 基本概念与命令

    <Learning Spark>这本书算是Spark入门的必读书了,中文版是<Spark快速大数据分析>,不过豆瓣书评很有意思的是,英文原版评分7.4,评论都说入门而已深入不足 ...

  2. 【原】Learning Spark (Python版) 学习笔记(四)----Spark Sreaming与MLlib机器学习

    本来这篇是准备5.15更的,但是上周一直在忙签证和工作的事,没时间就推迟了,现在终于有时间来写写Learning Spark最后一部分内容了. 第10-11 章主要讲的是Spark Streaming ...

  3. 【原】Learning Spark (Python版) 学习笔记(三)----工作原理、调优与Spark SQL

    周末的任务是更新Learning Spark系列第三篇,以为自己写不完了,但为了改正拖延症,还是得完成给自己定的任务啊 = =.这三章主要讲Spark的运行过程(本地+集群),性能调优以及Spark ...

  4. Learning Spark: Lightning-Fast Big Data Analysis 中文翻译

    Learning Spark: Lightning-Fast Big Data Analysis 中文翻译行为纯属个人对于Spark的兴趣,仅供学习. 如果我的翻译行为侵犯您的版权,请您告知,我将停止 ...

  5. Learning Spark (Python版) 学习笔记(一)----RDD 基本概念与命令

    <Learning Spark>这本书算是Spark入门的必读书了,中文版是<Spark快速大数据分析>,不过豆瓣书评很有意思的是,英文原版评分7.4,评论都说入门而已深入不足 ...

  6. 【原】Learning Spark (Python版) 学习笔记(二)----键值对、数据读取与保存、共享特性

    本来应该上周更新的,结果碰上五一,懒癌发作,就推迟了 = =.以后还是要按时完成任务.废话不多说,第四章-第六章主要讲了三个内容:键值对.数据读取与保存与Spark的两个共享特性(累加器和广播变量). ...

  7. Learning Spark中文版--第三章--RDD编程(1)

       本章介绍了Spark用于数据处理的核心抽象概念,具有弹性的分布式数据集(RDD).一个RDD仅仅是一个分布式的元素集合.在Spark中,所有工作都表示为创建新的RDDs.转换现有的RDD,或者调 ...

  8. Learning Spark 第四章——键值对处理

    本章主要介绍Spark如何处理键值对.K-V RDDs通常用于聚集操作,使用相同的key聚集或者对不同的RDD进行聚集.部分情况下,需要将spark中的数据记录转换为键值对然后进行聚集处理.我们也会对 ...

  9. 线性回归的Spark实现 [Linear Regression / Machine Learning / Spark]

    1- 问题提出 2- 线性回归 3- 理论推导 4- Python/Spark实现 # -*- coding: utf-8 -*- from pyspark import SparkContext t ...

随机推荐

  1. JWebFileTrans: 一款可以从网络上下载文件的小程序(一)

    一 摘要 JWebFileTrans是一款基于socket的网络文件传输小程序,目前支持从HTTP站点下载文件,后续会增加ftp站点下载.断点续传.多线程下载等功能.其代码已开源到github上面,下 ...

  2. javascript设计模式与开发实践阅读笔记(11)—— 模板方法模式

    模板方法模式: 由两部分结构组成,第一部分是抽象父类,第二部分是具体的实现子类.通常在抽象父类中封装了子类的算法框架,包括实现一些公共方法以及封装子类中所有方法的执行顺序.子类通过继承这个抽象类,也继 ...

  3. css 样式重置

    html, body, div, span, applet, object, iframe, h1, h2, h3, h4, h5, h6, p, blockquote, pre, a, abbr, ...

  4. 使用JAVA开发微信公众平台(一)——环境搭建与开发接入

    一. 初始微信公众平台 微信公众平台,即我们平时所说的"公众号",曾用名"官方平台"."媒体平台",但最终命名为"公众平台&quo ...

  5. Webappbuilder开发快速预览

    Webappbuilder开发快速预览 by 李远祥 Webappbuilder for ArcGIS 是由ArcGIS JavaScripit API和dojo创建的,它允许通过创建自己的widge ...

  6. js里的神奇双引号的长度

    "和"这两个引号(注意不是和字,是两侧的两个引号),你看出什么区别了么? 一个是复制进来的,另外一个是写上的,应该是半角英文了? "".length " ...

  7. 最新升级的火狐38.0.6识别ajax调用返回的""空值可能有异常。

    自已在调用一段ajax开发中,返回的是空值 string result = string.Empty;return result; 但在页面进行$.ajax调用 时 输出alert(result);应 ...

  8. tableView的编辑

    首先记住声明编辑样式的属性  UITableViewCellEditingStyle 和四个步骤 第一步:让tableView处于编辑状态 [self.rootView.tabView setEdit ...

  9. 构建自动化前端样式回归测试——BackstopJS篇

    在使用scss和less开发的时候,遇到过一件很有趣的事,因为网站需要支持响应式,就开了一个响应式样式框架,简单的几百行scss代码,居然生成了近100KB的css代码,因此决定重构这个样式库.而重构 ...

  10. 关于Ansi_Nulls、Quoted_Identifier、Ansi_Padding的用法

    --QUOTED_IDENTIFIER  语法SET QUOTED_IDENTIFIER { ON | OFF } 注释当 SET QUOTED_IDENTIFIER 为 ON 时,标识符可以由双引号 ...