Centos 6.5 x64环境下 spark 1.6 maven 编译-- 已验证
Centos 6.5 x64
jdk 1.7
scala 2.10
maven 3.3.3
cd spark-1.6
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
mvn -Dhadoop.version=2.7.1 -Phadoop-2.6 -DskipTests clean package
http://spark.apache.org/docs/latest/building-spark.html
关于部署包的生成:
编译源码后,加以配置就可以运行spark了,但是此时的目录过于庞大,有800M左右,部署起来不方便,所以需要生成部署包!
源码根目录下的make-distribution.sh可以生成部署包
注意 从spark1.1.0,开始支持maven的配置参数。该命令已经带有maven的编译过程,所以不需要先编译再打包!!
make-distribution.sh参数:
- –hadoop VERSION:打包时所用的Hadoop版本号,不加此参数时hadoop版本为1.0.4。
- –with-yarn:是否支持Hadoop YARN,不加参数时为不支持yarn。
- –with-hive:是否在Spark SQL 中支持hive,不加此参数时为不支持hive。
- –skip-java-test:是否在编译的过程中略过java测试,不加此参数时为略过。
- –with-tachyon:是否支持内存文件系统Tachyon,不加此参数时不支持tachyon。
- –tgz:在根目录下生成 spark-$VERSION-bin.tgz,不加此参数时不生成tgz文件,只生成/dist目录。
- –name NAME:和–tgz结合可以生成spark-$VERSION-bin-$NAME.tgz的部署包,不加此参数时NAME为hadoop的版本号。
./make-distribution.sh –hadoop 2.2.0 –with-yarn –tgz
./make-distribution.sh –hadoop 2.2.0 –with-yarn –with-hive –tgz
./make-distribution.sh –hadoop 2.2.0 –with-yarn –with-tachyon –tgz
生成在部署包位于根目录下,文件名类似于spark-1.0.0-bin-2.2.0.tgz。
可能出现的问题总结:
1.spark源码中的pom.xml中指定了maven的版本,所以要提交安装相对应的版本
2.内存不够
- [ERROR] PermGen space -> [Help ]
- [ERROR]
- [ERROR] To see the full stack trace of the errors,re-run Maven with the -e switch.
- [ERROR] Re-run Maven using the -X switch to enable full debug logging.
- [ERROR]
- [ERROR] For more information about the errors and possible solutions,
- please read the following articles:
- [ERROR] [Help ]http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
上面这个错误是因为编译的时候内存不够导致的,可以在编译的时候加大内存。
export MAVEN_OPTS=
"-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
3.mqtt-client-0.4.0无法下载
- [error] Server access Error: sun.security.validator.ValidatorException:
- PKIX path building failed:
- sun.security.provider.certpath.SunCertPathBuilderException:
- unable to find valid certification path to requested target
- url=https://repo.eclipse.org/content/repositories/paho-releases/
- org/eclipse/paho/mqtt-client/0.4./mqtt-client-0.4..pom
- [warn] module not found: org.eclipse.paho#mqtt-client;0.4.
这个错误是Maven无法下载mqtt-client-0.4.0.pom,因为他用到的是HTTPS协议。解决这个错误可以通过添加对repo.eclipse.org的信任从而解决这个问题。那如何解决呢?下载InstallCert.java类:
- /**
- * http://blogs.sun.com/andreas/resource/InstallCert.java
- * Use:
- * java InstallCert hostname
- * Example:
- *% java InstallCert ecc.fedora.redhat.com
- */
- import javax.net.ssl.*;
- import java.io.*;
- import java.security.KeyStore;
- import java.security.MessageDigest;
- import java.security.cert.CertificateException;
- import java.security.cert.X509Certificate;
- /**
- * Class used to add the server's certificate to the KeyStore
- * with your trusted certificates.
- */
- public class InstallCert {
- public static void main(String[] args) throws Exception {
- String host;
- int port;
- char[] passphrase;
- if ((args.length == 1) || (args.length == 2)) {
- String[] c = args[0].split(":");
- host = c[0];
- port = (c.length == 1) ? 443 : Integer.parseInt(c[1]);
- String p = (args.length == 1) ? "changeit" : args[1];
- passphrase = p.toCharArray();
- } else {
- System.out.println("Usage: java InstallCert <host>[:port"+
- "[passphrase]");
- return;
- }
- File file = new File("jssecacerts");
- if (file.isFile() == false) {
- char SEP = File.separatorChar;
- File dir = new File(System.getProperty("java.home") + SEP
- + "lib" + SEP + "security");
- file = new File(dir, "jssecacerts");
- if (file.isFile() == false) {
- file = new File(dir, "cacerts");
- }
- }
- System.out.println("Loading KeyStore " + file + "...");
- InputStream in = new FileInputStream(file);
- KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
- ks.load(in, passphrase);
- in.close();
- SSLContext context = SSLContext.getInstance("TLS");
- TrustManagerFactory tmf =
- TrustManagerFactory.getInstance(
- TrustManagerFactory.getDefaultAlgorithm());
- tmf.init(ks);
- X509TrustManager defaultTrustManager =
- (X509TrustManager) tmf.getTrustManagers()[0];
- SavingTrustManager tm = new SavingTrustManager(defaultTrustManager);
- context.init(null, new TrustManager[]{tm}, null);
- SSLSocketFactory factory = context.getSocketFactory();
- System.out.println("Opening connection to " + host + ":" + port + "...");
- SSLSocket socket = (SSLSocket) factory.createSocket(host, port);
- socket.setSoTimeout(10000);
- try {
- System.out.println("Starting SSL handshake...");
- socket.startHandshake();
- socket.close();
- System.out.println();
- System.out.println("No errors, certificate is already trusted");
- } catch (SSLException e) {
- System.out.println();
- e.printStackTrace(System.out);
- }
- X509Certificate[] chain = tm.chain;
- if (chain == null) {
- System.out.println("Could not obtain server certificate chain");
- return;
- }
- BufferedReader reader =
- new BufferedReader(new InputStreamReader(System.in));
- System.out.println();
- System.out.println("Server sent " + chain.length + " certificate(s):");
- System.out.println();
- MessageDigest sha1 = MessageDigest.getInstance("SHA1");
- MessageDigest md5 = MessageDigest.getInstance("MD5");
- for (int i = 0; i < chain.length; i++) {
- X509Certificate cert = chain[i];
- System.out.println
- (" " + (i + 1) + " Subject " + cert.getSubjectDN());
- System.out.println(" Issuer " + cert.getIssuerDN());
- sha1.update(cert.getEncoded());
- System.out.println(" sha1 " + toHexString(sha1.digest()));
- md5.update(cert.getEncoded());
- System.out.println(" md5 " + toHexString(md5.digest()));
- System.out.println();
- }
- System.out.println("Enter certificate to add to trusted keystore or"+
- " 'q' to quit: [1]");
- String line = reader.readLine().trim();
- int k;
- try {
- k = (line.length() == 0) ? 0 : Integer.parseInt(line) - 1;
- } catch (NumberFormatException e) {
- System.out.println("KeyStore not changed");
- return;
- }
- X509Certificate cert = chain[k];
- String alias = host + "-" + (k + 1);
- ks.setCertificateEntry(alias, cert);
- OutputStream out = new FileOutputStream("jssecacerts");
- ks.store(out, passphrase);
- out.close();
- System.out.println();
- System.out.println(cert);
- System.out.println();
- System.out.println
- ("Added certificate to keystore 'jssecacerts' using alias '"
- + alias + "'");
- }
- private static final char[] HEXDIGITS = "0123456789abcdef".toCharArray();
- private static String toHexString(byte[] bytes) {
- StringBuilder sb = new StringBuilder(bytes.length * 3);
- for (int b : bytes) {
- b &= 0xff;
- sb.append(HEXDIGITS[b >> 4]);
- sb.append(HEXDIGITS[b & 15]);
- sb.append(' ');
- }
- return sb.toString();
- }
- private static class SavingTrustManager implements X509TrustManager {
- private final X509TrustManager tm;
- private X509Certificate[] chain;
- SavingTrustManager(X509TrustManager tm) {
- this.tm = tm;
- }
- public X509Certificate[] getAcceptedIssuers() {
- throw new UnsupportedOperationException();
- }
- public void checkClientTrusted(X509Certificate[] chain, String authType)
- throws CertificateException {
- throw new UnsupportedOperationException();
- }
- public void checkServerTrusted(X509Certificate[] chain, String authType)
- throws CertificateException {
- this.chain = chain;
- tm.checkServerTrusted(chain, authType);
- }
- }
- }
编译上面的类:
$ javac InstallCert.java
$ java InstallCert repo.eclipse.org
Centos 6.5 x64环境下 spark 1.6 maven 编译-- 已验证的更多相关文章
- Centos 6.5 X64 环境下编译 hadoop 2.6.0 --已验证
Centos 6.5 x64 hadoop 2.6.0 jdk 1.7 protobuf-2.5.0 maven-3.0.5 set environment export JAVA_HOME=/hom ...
- Win8 x64环境下VS2010 C#工程运行报错:没有注册类 (异常来自 HRESULT:0x80040154
来源:http://blog.sina.com.cn/s/blog_7095482001019c2v.html 问题描述: 在Win8 x64环境下,VS2010的C#工程中引用了COM组件(Acti ...
- X86和X64环境下的基本类型所占用的字节大小
同样的程序代码,使用Visual Studio 进行编译,当目标平台分别为x86或x64环境时,其编译结果是不同的.在x86环境下,指针都是4个字节的:而在x64环境下,指针都是8字节的.测试代码如下 ...
- Windows 7 x64环境下JDK8安装过程
Windows 7 x64环境下JDK8安装过程 下载地址:http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads ...
- RedHat 6.7 Enterprise x64环境下使用RHCS部署Oracle 11g R2双机双实例HA
环境 软硬件环境 硬件环境: 浪潮英信服务器NF570M3两台,华为OceanStor 18500存储一台,以太网交换机两台,光纤交换机两台. 软件环境: 操作系统:Redhat Enterpris ...
- RedHat 6.7 Enterprise x64环境下使用RHCS部署Oracle 11g R2双机HA
环境 软硬件环境 硬件环境: 浪潮英信服务器NF570M3两台,华为OceanStor 18500存储一台,以太网交换机两台,光纤交换机两台. 软件环境: 操作系统:Redhat Enterprise ...
- Centos环境下部署游戏服务器-编译
游戏服务器是在windows环境开发的,相关跨平台的东西在这里不谈了,只谈如何将Visual Studio 工程转换到Linux下编译.这里涉及到的软件分别为:Centos版本为6.4,Visual ...
- kerberos环境下spark消费kafka写入到Hbase
一.准备环境: 创建Kafka Topic和HBase表 1. 在kerberos环境下创建Kafka Topic 1.1 因为kafka默认使用的协议为PLAINTEXT,在kerberos环境下需 ...
- 多线程环境下非安全Dictionary引起的“已添加了具有相同键的项”问题
问题: 代码是在多线程环境下,做了简单的Key是否存的判断, 测试代码如下: public class Program { static Dictionary<string, Logger> ...
随机推荐
- getElementsByClassName简单实现
function getElementsByClassName(node, className) { var aClassReg = className.split(' ').map(function ...
- 我们平时是怎么写html和css的?
文章的起因,我只是为了回复一个帖子,http://bbs.csdn.net/topics/390908928?page=1 结果,一扯就根本停不下来.索性,一捅为快,反正是周末. 拿到效果图时,有这么 ...
- 【Make a H5 game】JS for beginner——FROM U2B
https://www.youtube.com/watch?v=F2Dc-JlwgN4&feature=iv&src_vid=WfL4LNUL3R0&annotation_id ...
- c++文件输入输出流fstream,对输入>>和输出<<重载
1. fstream 继承自iostream --> 要包含头文件#include<fstream> 2. 建立文件流对象 3. 打开文件夹 4. 测试是否打开成功 5. 进行读写操 ...
- Repeater、地址栏传值、Response--2016年12月30日
Repeater Repeater支持以下5种模板 ● ItemTemplate : 对每一个数据项进行格式设置 [Formats each item from the data sou ...
- Oracle客户端工具出现“Cannot access NLS data files or invalid environment specified”错误的解决办法
Oracle客户端工具出现"Cannot access NLS data files or invalid environment specified"错误的解决办法 方法一:参考 ...
- JQuery实现无刷新下拉加载图片
最近做的一个项目需要做页面无刷新下拉加载图片,调研了一番,大多都采用检测滚动条达到底部,然后利用ajax加载下一页数据对页面数据进行添加,根据这一逻辑,自己写了一个,具体代码如下: JQu ...
- (三)Netty源码学习笔记之boss线程处理流程
尊重原创,转载注明出处,原文地址:http://www.cnblogs.com/cishengchongyan/p/6160194.html 本文我们将先从NioEventLoop开始来学习服务端的 ...
- Python之Web前端jQuery扩展
Python之Web前端: 一. jQuery表单验证 二. jQuery扩展 三. 滚动菜单 一. jQuery表单验证: 任何可以交互的站点都有输入表单,只要有可能,就应该对用户输入的数据进行验证 ...
- Reverse Core 第二部分 - 14&15章 - 运行时压缩&调试UPX压缩的notepad
@date: 2016/11/29 @author: dlive 0x00 前言 周六周日两天在打HCTF2016线上赛,没时间看书,打完比赛接着看~~ 0x01 运行时压缩 对比upx压缩前后的no ...