问题导读:
1.编译源码前需要安装哪些软件?
2.安装之后该如何设置环境变量?
3.为什么不要使用JDK1.8?
4.mvn package -Pdist -DskipTests -Dtar的作用是什么?

1. 前言
Hadoop-2.4.0的源码目录下有个BUILDING.txt文件,它介绍了如何在Linux和Windows下编译源代码,本文基本是遵照BUILDING.txt指示来操作的,这里再做一下简单的提炼。
第一次编译要求能够访问互联网,Hadoop的编译依赖非常多的东西,一定要保证机器可访问互联网,否则难逐一解决所有的编译问题,但第一次之后的编译则不用再下载了。
如不能上网可以参考:虚拟机三种网络模式该如何上网指导
2. 安装依赖
在编译Hadoop 2.4.0源码之前,需要将下列几个依赖的东西安装好:
1) JDK 1.6或更新版本(本文使用JDK1.7,请不要安装JDK1.8版本,JDK1.8和Hadoop 2.4.0不匹配,编译Hadoop 2.4.0源码时会报很多错误)
2) Maven 3.0或更新版本
3) ProtocolBuffer 2.5.0
4) CMake 2.6或更新版本
5) Findbugs 1.3.9,可选的(本文编译时未安装)

在安装好之后,还需要设置一下环境变量,可以修改/etc/profile,也可以是修改~/.profile,增加如下内容:
export JAVA_HOME=/root/jdk
export CLASSPATH=$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH

export CMAKE_HOME=/root/cmake
export PATH=$CMAKE_HOME/bin:$PATH

export PROTOC_HOME=/root/protobuf
export PATH=$PROTOC_HOME/bin:$PATH

export MAVEN_HOME=/root/maven
export PATH=$MAVEN_HOME/bin:$PATH

本文以root用户在/root目录下进行安装,但实际可以选择非root用户及非/root目录进行安装。

2.1. 安装ProtocolBuffer
标准的automake编译安装方式:
1) cd /root
2) tar xzf protobuf-2.5.0.tar.gz
3) cd protobuf-2.5.0
4) ./conigure --prefix=/root/protobuf
5) make
6) make install

2.2. 安装CMake
1) cd /root
2) tar xzf cmake-2.8.12.2.tar.gz
3) cd cmake-2.8.12.2
4) ./bootstrap --prefix=/root/cmake
5) make
6) make install

2.3. 安装JDK
1) cd /root
2) tar xzf jdk-7u55-linux-x64.gz
3) cd jdk1.7.0_55
4) ln -s jdk1.7.0_55 jdk

2.4. 安装Maven
1) cd /root
2) tar xzf apache-maven-3.0.5-bin.tar.gz
3) ln -s apache-maven-3.0.5 maven

3. 编译Hadoop源代码
完成上述准备工作后,即可通过执行命令:mvn package -Pdist -DskipTests -Dtar,启动对Hadoop源代码的编译。请注意一定不要使用JDK1.8。

编译成功后,jar文件会放在target子目录下,可以在Hadoop源码目录下借用find命令搜索各个target子目录。
编译成功后,会生成Hadoop二进制安装包hadoop-2.4.0.tar.gz,放在源代码的hadoop-dist/target子目录下:
main:
     [exec] $ tar cf hadoop-2.4.0.tar hadoop-2.4.0
     [exec] $ gzip -f hadoop-2.4.0.tar
     [exec]
     [exec] Hadoop dist tar available at: /root/hadoop-2.4.0-src/hadoop-dist/target/hadoop-2.4.0.tar.gz
     [exec]
[INFO] Executed tasks
[INFO]
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /root/hadoop-2.4.0-src/hadoop-dist/target/hadoop-dist-2.4.0-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [4.647s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [5.352s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [7.239s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.424s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [2.918s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [6.261s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [5.321s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [5.953s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [3.783s]
[INFO] Apache Hadoop Common .............................. SUCCESS [1:54.010s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [9.721s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.048s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [4:15.270s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [6:18.553s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [16.237s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [6.543s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.036s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.051s]
[INFO] hadoop-yarn-api ................................... SUCCESS [1:35.227s]
[INFO] hadoop-yarn-common ................................ SUCCESS [43.216s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.055s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [16.476s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [19.942s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [4.926s]
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [9.804s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [23.320s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [1.208s]
[INFO] hadoop-yarn-client ................................ SUCCESS [9.177s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.113s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [4.106s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [3.265s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.056s]
[INFO] hadoop-yarn-project ............................... SUCCESS [5.552s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.096s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [37.231s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [27.135s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [4.886s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [17.876s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [14.140s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [11.305s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [3.083s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [9.855s]
[INFO] hadoop-mapreduce .................................. SUCCESS [5.110s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [7.778s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [12.973s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [3.265s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [11.060s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [7.412s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [4.221s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [4.771s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [0.032s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [8.030s]
[INFO] Apache Hadoop Client .............................. SUCCESS [7.730s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.158s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [7.485s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [6.912s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.029s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [40.425s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21:57.892s
[INFO] Finished at: Mon Apr 21 14:33:22 CST 2014
[INFO] Final Memory: 88M/243M
[INFO] ------------------------------------------------------------------------

附1:编译环境
整个过程是在阿里云64位主机上进行的,2.30GHz单核1G内存:
[root@AY140408105805619186Z hadoop-2.4.0-src]# uname -a
Linux AY140408105805619186Z 2.6.18-308.el5 #1 SMP Tue Feb 21 20:06:06 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

[root@AY140408105805619186Z ~]# cat /etc/redhat-release
CentOS release 5.8 (Final)

附2:版本信息
名称
版本
包名
说明
Maven
3.0.5
apache-maven-3.0.5-bin.tar.gz
使用3.2.1可能会有问题
CMake
2.8.12.2
cmake-2.8.12.2.tar.gz

JDK
1.7.0
jdk-7u55-linux-x64.gz
不能使用JDK1.8.0
Protocol Buffers
2.5.0
protobuf-2.5.0.tar.gz

Hadoop
2.4.0
hadoop-2.4.0-src.tar.gz

附3:常见错误
1) unexpected end tag: </ul>
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar
(module-javadocs) on project hadoop-annotations: MavenReportException:
Error while creating archive:
[ERROR] Exit code: 1 -
/root/hadoop-2.4.0-src/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceStability.java:27:
error: unexpected end tag: </ul>
[ERROR] * </ul>
[ERROR] ^
[ERROR]
[ERROR] Command line was: /root/jdk1.8.0/jre/../bin/javadoc @options @packages
原因是InterfaceStability.java中的注释问题:

解决办法,将JDK换成1.7版本,使用JDK1.8编译就会遇到上述问题,将</ul>行删除可以解决问题,但后续还会遇到类似的问题,所以不要使用JDK1.8编译Hadoop 2.4.0。

在Linux上编译Hadoop-2.4.0实践与总结的更多相关文章

  1. mac OS X Yosemite 上编译hadoop 2.6.0/2.7.0及TEZ 0.5.2/0.7.0 注意事项

    1.jdk 1.7问题 hadoop 2.7.0必须要求jdk 1.7.0,而oracle官网已经声明,jdk 1.7 以后不准备再提供更新了,所以趁现在还能下载,赶紧去down一个mac版吧 htt ...

  2. 64位CentOS上编译 Hadoop 2.2.0

    下载了Hadoop预编译好的二进制包,hadoop-2.2.0.tar.gz,启动起来后.总是出现这样的警告: WARN util.NativeCodeLoader: Unable to load n ...

  3. CentOS 64位上编译 Hadoop 2.6.0

    Hadoop不提供64位编译好的版本号,仅仅能用源代码自行编译64位版本号. 学习一项技术从安装開始.学习hadoop要从编译開始. 1.操作系统编译环境 yum install cmake lzo- ...

  4. 在Linux上编译Hadoop-2.4.0

    目录 目录 1 1. 前言 1 2. 安装依赖 1 2.1. 安装ProtocolBuffer 2 2.2. 安装CMake 2 2.3. 安装JDK 2 2.4. 安装Maven 3 3. 编译Ha ...

  5. Linux上安装Hadoop集群(CentOS7+hadoop-2.8.0)--------hadoop环境的搭建

    Linux上安装Hadoop集群(CentOS7+hadoop-2.8.0)------https://blog.csdn.net/pucao_cug/article/details/71698903 ...

  6. 在docker容器中编译hadoop 3.1.0

    在docker容器中编译hadoop 3.1.0 优点:docker安装好之后可以一键部署编译环境,不用担心各种库不兼容等问题,编译失败率低. Hadoop 3.1.0 的源代码目录下有一个 `sta ...

  7. 尝试在Linux上编译KestrelHttpServer

    Kestrel是目前在非Windows平台上运行ASP.NET 5应用程序的唯一可用Web服务器,但微软似乎将它冷落在一边,源代码更新很慢. 今天试着在Linux上编译Kestrel的源代码,遇到了很 ...

  8. 使用Windows上Eclipse远程调试Linux上的Hadoop

    一.设置Eclipse运行用户     如果以与Hadoop运行用户名(比如grid)不同的用户运行Eclipse,则无法对Hadoop运行用户所属的文件进行管理,运行Map/Reduce程序也会报& ...

  9. Linux上编译安装PHP

    这篇文章主要介绍了关于Linux上编译安装PHP,有着一定的参考价值,现在分享给大家,有需要的朋友可以参考一下 之前在服务器上编译安装了PHP运行环境,但是安装完过了一段时间就差不多忘记了,只是零零星 ...

随机推荐

  1. 【转载】阿里云Windows服务器快速部署PHP运行环境

    PHP也是一种常用的网站脚本语言,时常用于网站应用程序的编写,PHP运行环境包含几个要素:PHP.Apache服务器.MySql数据库,此3个要素是Php网站运行的必要条件.在阿里云的Windows服 ...

  2. Python多线程的简单实现(生产者消费者模型)

    __author__ = "JentZhang" import time, threading, queue q = queue.Queue(maxsize=) # 声明队列 de ...

  3. vuex 、store、state (转载)

    vuex  文档 https://vuex.vuejs.org/zh/guide/state.html

  4. ECharts中color : function的用法(转)

    ECharts图表实战经验1:如何设置图表同序列不同数据点的独立颜色值   最近有不少朋友在追问这样一个问题:我单序列的柱状图,我想让每一个根柱子的颜色都不一样,应该如何做? 针对这个问题,其实我只想 ...

  5. c# 后台分页 jqgrid

    /// <summary> /// 获取设备数量 /// </summary> /// <param name="Organid">单位ID&l ...

  6. window 服务器的Tomcat 控制台日志保存到日志文件.

    在Linux系统中,Tomcat 启动后默认将很多信息都写入到 catalina.out 文件中,我们可以通过tail  -f  catalina.out 来跟踪Tomcat 和相关应用运行的情况. ...

  7. Vue项目build打包部署到Tomcat后,刷新报404错误解决方案

    问题描述: 一.更新依赖,并打包项目 cd /root/.jenkins/workspace/v-test;npm installcd /root/.jenkins/workspace/v-test; ...

  8. Sql 判断函数是否存在、sql判断表是否存在、sql判断存储过程是否存在、sql判断视图是否存在

    --数据库是否存在 IF exists(SELECT * FROM master..sysdatabases WHERE name=N'库名') PRINT 'exists' ELSE PRINT ' ...

  9. CSS3 - @keyframes

    语法 @keyframes animationname { keyframes-selector {css-styles;} } 值 描述 animationname 必需.定义动画的名称. keyf ...

  10. POJ 2942Knights of the Round Table(tarjan求点双+二分图染色)

    Time Limit: 7000MS   Memory Limit: 65536K Total Submissions: 13954   Accepted: 4673 Description Bein ...