Installing Apache Spark on Ubuntu 16.04
Santosh Srinivas
on 07 Nov 2016, tagged onApache Spark, Analytics, Data Minin
I've finally got to a long pending to-do-item to play with Apache Spark.
The following installation steps worked for me on Ubuntu 16.04.
- Download the latest pre-built version from http://spark.apache.org/downloads.html
The below options worked for me:
- Unzip and move Spark
cd ~/Downloads/
tar xzvf spark-2.0.1-bin-hadoop2.7.tgz
mv spark-2.0.1-bin-hadoop2.7/ spark
sudo mv spark/ /usr/lib/
- Install SBT
As mentioned at sbt - Download
echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
sudo apt-get update
sudo apt-get install sbt
- Make sure Java is installed
If not, install java
sudo apt-add-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
- Configure Spark
cd /usr/lib/spark/conf/
cp spark-env.sh.template spark-env.sh
vi spark-env.sh
Add the following lines
JAVA_HOME=/usr/lib/jvm/java-8-oracle
SPARK_WORKER_MEMORY=4g
- Configure IPv6
Basically, disable IPv6 using sudo vi /etc/sysctl.conf
and add below lines
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
- Configure .bashrc
I modified .bashrc in Sublime Text using subl ~/.bashrc
and added the following lines
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export SBT_HOME=/usr/share/sbt-launcher-packaging/bin/sbt-launch.jar
export SPARK_HOME=/usr/lib/spark
export PATH=$PATH:$JAVA_HOME/bin
export PATH=$PATH:$SBT_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin
- Configure fish (Optional - But I love the fish shell)
Modify config.fish
using subl ~/.config/fish/config.fish
and add the following lines
#Credit: http://fishshell.com/docs/current/tutorial.html#tut_startup
set -x PATH $PATH /usr/lib/spark
set -x PATH $PATH /usr/lib/spark/bin
set -x PATH $PATH /usr/lib/spark/sbin
- Test Spark (Should work both in fish and bash)
Run pyspark
(this is available in /usr/lib/spark/bin/
) and test out.
For example ....
>>> a = 5
>>> b = 3
>>> a+b
8
>>> print(“Welcome to Spark”)
Welcome to Spark
## type Ctrl-d to exit
Try also, the built in run-example using run-example org.apache.spark.examples.SparkPi
That's it! You are ready to rock on using Apache Spark!
Next, I plan to checkout analysis using R as mentioned inhttp://www.milanor.net/blog/wp-content/uploads/2016/11/interactiveDataAnalysiswithSparkR_v5.pdf
Installing Apache Spark on Ubuntu 16.04的更多相关文章
- Install and Configure Apache Kafka on Ubuntu 16.04
https://devops.profitbricks.com/tutorials/install-and-configure-apache-kafka-on-ubuntu-1604-1/ by hi ...
- Install LAMP Stack On Ubuntu 16.04
原文:http://www.unixmen.com/how-to-install-lamp-stack-on-ubuntu-16-04/ LAMP is a combination of operat ...
- Ubuntu 16.04 LAMP server tutorial with Apache 2.4, PHP 7 and MariaDB (instead of MySQL)
https://www.howtoforge.com/tutorial/install-apache-with-php-and-mysql-on-ubuntu-16-04-lamp/ This tut ...
- digitalocean --- How To Install Apache Tomcat 8 on Ubuntu 16.04
https://www.digitalocean.com/community/tutorials/how-to-install-apache-tomcat-8-on-ubuntu-16-04 Intr ...
- 安装Hadoop及Spark(Ubuntu 16.04)
安装Hadoop及Spark(Ubuntu 16.04) 安装JDK 下载jdk(以jdk-8u91-linux-x64.tar.gz为例) 新建文件夹 sudo mkdir /usr/lib/jvm ...
- 解决Ubuntu 16.04 上Android Studio2.3上面运行APP时提示DELETE_FAILED_INTERNAL_ERROR Error while Installing APKs的问题
本人工作环境:Ubuntu 16.04 LTS + Android Studio 2.3 AVD启动之后,运行APP,报错提示: DELETE_FAILED_INTERNAL_ERROR Error ...
- Installing Moses on Ubuntu 16.04
Installing Moses on Ubuntu 16.04 The process of installation To install requirements sudo apt-get in ...
- Installing Hyperledger Fabric v1.1 on Ubuntu 16.04 — Part I
There is an entire library of Blockchain APIs which you can select according to the needs that suffi ...
- 如何在Ubuntu 16.04上安装Apache Web服务器
转载自:https://www.howtoing.com/how-to-install-the-apache-web-server-on-ubuntu-16-04 介绍 Apache HTTP服务器是 ...
随机推荐
- 【LOJ】#2306. 「NOI2017」蔬菜
题解 从后往前递推 如果我们知道了第i天的最优方案和第i天选择的蔬菜,加入第i天选择的蔬菜数量为S,我们只需要减去最小的S - (i - 1) * M 个蔬菜即可 所以我们只要求出最后一天的蔬菜选择 ...
- python日常总结
1. post请求中是否可以在url中携带请求体信息? 可以.Get请求时,请求体放在URL中; POST请求,请求体既可以是Form表单中的数据 也可以在请求的URL地址中放请求体信息. 如: &l ...
- Ionic Js十六:滚动条
ion-scroll ion-scroll 用于创建一个可滚动的容器. <ion-scroll [delegate-handle=""] [direction="& ...
- ACM训练计划建议(转)
ACM训练计划建议 From:freecode# Date:2015/5/20 前言: 老师要我们整理一份训练计划给下一届的学弟学妹们,整理出来了,费了不少笔墨,就也将它放到博客园上供大家参考. 菜 ...
- Oracle数据库DDL,DML,视图,PLSQL编程
动手敲~~~ --创建一个表空间--beijing create tablespace beijing datafile 'c:\beijing.dbf' size 100m autoextend o ...
- (五)静态断言(下),static_assert
二.静态断言与static_assert 通过上一篇,我们可以看到,断言assert宏只有在程序运行的时候才能起作用.而#error值在编译器预处理时才能起作用. 有时候,我们希望在编译时候能做一些断 ...
- NetStandard类库实现Log4Net集成
前面都是Log4Net集成到NetCore项目中,集成到NetStandard类库还是第一次,所以记录一下 小提示:NetStandard要想同时被NetCore和NetFramework调用,需要在 ...
- ELK收集openstack日志
1.安装jdk 每个openstack服务器需要安装jdk,我安装的版本jdk-7u71-linux-x64.rpm 2.安装.配置Elastic Search install https://dow ...
- 顺序线性表之大整数求和C++实现
顺序线性表之大整数求和 大整数求和伪代码 1.初始化进位标志 flag=0: 2.求大整数 A 和 B 的长度: int aLength = a.GetLength(); int bLength = ...
- [BZOJ1815&BZOJ1488]有色图/图的同构(Polya定理)
由于有很多本质相同的重复置换,我们先枚举各种长度的点循环分别有多少个,这个暴搜的复杂度不大,n=53时也只有3e5左右.对于每种搜索方案可以轻易求出它所代表的置换具体有多少个. 但我们搜索的是点置换组 ...