1.下载 Spark安装之前的准备 文件的解压与改名 tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz rm -rf spark-2.2.0-bin-hadoop2.7.tgz 为了我后面方便配置spark,在这里我把文件夹的名字给改了 mv spark-2.2.0-bin-hadoop2.7 spark-2.2.0----------------------------------------------------------------------------…

1.配置ssh,使集群服务器之间的通讯,不再每次都输入密码进行认证. 2. [root@hc--uatbeta2 hadoop]# start-all.shStarting namenodes on [hc--uatbeta2.novalocal]ERROR: Attempting to operate on hdfs namenode as rootERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.Sta…

As a non-Java developer, I am quit stuck in Java environment setting because I am not familiar with Java. I write it down as following, I think it is the best way to write it in /etc/profile: export JAVA_HOME=/usr/local/jdk1.8.0_111 export PATH=$JAVA…

Documentation: https://keras.io/ 1. 利用anaconda 管理python库是明智的选择. conda update conda conda update anaconda conda update --all conda install mingw libpython pip install --upgrade --no-deps theano pip install keras 2. 测试theano python执行: import theano the…

目前正在学习Hadoop和spark之类的东西,一个月把Hadoop的基础东西过了一遍,但是感觉好动都没跟上老师的课程,哪位前辈了解这方面的东西希望给指点迷津.接下来我们还要学习spark和nosql,每天基本都是12点以后才能入睡,早上8点准时开工.很累很累,但任然想继续努力把它做好. 在Hadoop中,现在主要学了hdfs,hive,sqoop,pig等内容,现在spark正在进行spark/sql和spark/hive. 求前辈指点.…

如何改变 Visual Studio 的默认环境设置: 1. 工具栏 Tools --> Import and Export Settings... 2. 选择 Reset All Settings,单击 “Next” 3. 选择保存设置的选项,单击“Next” 4. 选择想要设定的编程环境,点击“Finish”. 参考链接:http://stackoverflow.com/questions/12084958/changing-default-enviromnent-setting-when-…

WLS(Windows Subsystem for Linux) Base WLS Installation Guide Initializing a newly installed distro WSL Doc More 更改默认源至国内镜像 $ sudo cp /etc/apt/sources.list /etc/apt/sources.list.backup # 备份配置文件 $ sudo vim /etc/apt/sources.list # 使用vim打开配置文件 # 在vim打开的配…

java 7 jdk http://www.ifunmac.com/2013/04/mac-jdk-7/ http://blog.sina.com.cn/s/blog_6dce99b101016744.html eclipse http://jingyan.baidu.com/article/7f41ecece8ef5b593c095c71.html…

BigDL: Distributed Deep Learning on Apache Spark What is BigDL? BigDL is a distributed deep learning library for Apache Spark; with BigDL, users can write their deep learning applications as standard Spark programs, which can directly run on top of e…

Spark Streaming 编程指南 Overview A Quick Example Basic Concepts Linking Initializing StreamingContext Discretized Streams (DStreams) Input DStreams and Receivers Transformations on DStreams Output Operations on DStreams DataFrame and SQL Operations MLli…