编译安装hadoop2.6.3

一、安装环境

1.1  JAVA 

安装java1.7

下载jdk1.7:

[root@node1~]# wget http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.tar.gz?AuthParam=1452765180_64b65bb908cae46ab9a9e492c842d7c7

设置JAVA环境变量：

PATH=$PATH:$HOME/bin:/usr/local/mongodb-linux-x86_64-3.2.0/bin

JAVA_HOME=/usr/local/jdk1.7.0_79

CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASS

PATH=$JAVA_HOME/bin:$PATH

 

1.2 Maven

下载安装maven:

[root@node1~]# wget http://mirrors.cnnic.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

然后解压，到/usr/local/，配置path

PATH=$JAVA_HOME/bin:/usr/local/apache-maven-3.3.9/bin:$PATH

 

1.3 Findbugs

可选安装，略

 

1.4 protobuf

参考：http://blog.csdn.net/huguoping830623/article/details/45482725

安装：./configure --prefix=/usr/local/protobuf2.5/  

make && make install

安装中可能报错：

configure: error: C++ preprocessor "/lib/cpp" fails sanity check

原因是需要gcc相关包:

　　# yum install glibc-headers
　　# yum install gcc-c++

安装后把/usr/local/protobuf2.5/bin目录加入PATH

 

1.5 其它包

yum -y install  lzo-devel  zlib-devel  gcc autoconf automake libtool openssl-devel fuse-devel cmake 

 

* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac

* Zlib devel (if compiling native code)

* openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )

* Jansson C XML parsing library ( if compiling libwebhdfs )   

./configure 

make && make install

* Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs )

    [root@node1 ~]# wgethttps://github.com/libfuse/libfuse/releases/download/fuse_2_9_4/fuse-2.8.6.tar.gz

    tar -zxvf fuse-2.8.6.tar.gz

   ./configure
   make -j8
   make install

* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

 

二、编译安装

1.下载hadoop源文件

[root@node1~]# wget http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.6.3/hadoop-2.6.3-src.tar.gz

然后解压

 

2.编译hadoop文件

进入hadoop源文件的解压目录，执行：

$ mvn package -Pdist,native -DskipTests -Dtar

要保证能联网，因此要从网上下载依赖包

编译过程要很久。编译完成后放在 hadoop-2.6.3-src/hadoop-dist/target下。

[root@node1 target]# ls -l

total 530484

drwxr-xr-x. 2 root root      4096 Jan 19 13:13 antrun

-rw-r--r--. 1 root root      1867 Jan 19 13:13 dist-layout-stitching.sh

-rw-r--r--. 1 root root       640 Jan 19 13:13 dist-tar-stitching.sh

drwxr-xr-x. 9 root root      4096 Jan 19 13:13 hadoop-2.6.3        hadoop编译后的解压文件

-rw-r--r--. 1 root root 180792661 Jan 19 13:13 hadoop-2.6.3.tar.gz hadoop安装文件

-rw-r--r--. 1 root root      2778 Jan 19 13:13 hadoop-dist-2.6.3.jar

-rw-r--r--. 1 root root 362386511 Jan 19 13:13 hadoop-dist-2.6.3-javadoc.jar

drwxr-xr-x. 2 root root      4096 Jan 19 13:13 javadoc-bundle-options

drwxr-xr-x. 2 root root      4096 Jan 19 13:13 maven-archiver

drwxr-xr-x. 2 root root      4096 Jan 19 13:13 test-dir

复制到安装目录：

[root@node1 target]# cp -r hadoop-2.6.3 /usr/local/

[root@node1 target]# cd /usr/local/hadoop-2.6.3

编辑hadoop-env.sh文件

[root@node1 hadoop-2.6.3]# vi etc/hadoop/hadoop-env.sh  

修改:

export JAVA_HOME=JAVA_HOME=/usr/local/jdk1.7.0_79

添加：

export HADOOP_PREFIX=/usr/local/hadoop-2.6.3

然后测试：

 

 

[root@node1 hadoop-2.6.3]# bin/hadoop version

Hadoop 2.6.3

Subversion Unknown -r Unknown

Compiled by root on 2016-01-19T04:56Z

Compiled with protoc 2.5.0

From source with checksum 722f77f825e326e13a86ff62b34ada

This command was run using /usr/local/hadoop-2.6.3/share/hadoop/common/hadoop-common-2.6.3.jar

测试成功！

 

三、测试

  $ mkdir input
  $ cp etc/hadoop/*.xml input
  $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.3.jar grep input output 'dfs[a-z.]+'
  $ cat output/*

 

如果报INFO metrics.MetricsUtil: Unable to obtain hostName:node1错

修改/etc/hosts文件，加入本机的别名node1

127.0.0.1 localhost localhost.localdomain localhost修改为

127.0.0.1 localhost.localdomain node1

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

来自为知笔记(Wiz)