hadoop官网

我以Hadoop 2.7.3为例. hadoop 2.7.3 官网  . 用的操作系统是64bit Ubuntu14.04. 其中我们还可以学习 Apache Maven Project , 它是用来编译 hadoop 源代码的. 还要了解 Apache Log4j , 它用来调试 hadoop 源代码.

1. Hadoop环境准备

1.1 准备学习环境

1.1.1 基础软件下载 (hadoop 2.7.3 最好用jdk1.7版本,否则mvn源码时会出现各种错误; 而jdk1.7对应Eclipse Mars (4.5))

  Hadoop采用的开发语言主要是Java,因而搭建Hadoop环境所需的最基础软件首先应该包括Java基础开发包JDK和Java项目管理工具Maven, 考虑到源代码阅读和调试的便利性,本文采用功能强大的集成开发环境Eclipse. 搭建Hadoop阅读环境需要的各种软件以及下载方式如表1所示.

表1 搭建Hadoop阅读环境所需的软件

软件 下载网址 推荐版本
JDK http://www.oracle.com/technetwork/java/javase/downloads/index.html 1.6以上
Maven http://maven.apache.org/download.cgi 3.0.2以上
Eclipse https://www.eclipse.org/downloads/ Galileo(3.5)以上

1.1.2 如何准备Linux环境( 本文以64bit Ubuntu为例)

  jdk1.7官网下载   Eclipse Mars (4.5)官网下载  . 安装jdk以及eclipse参见我的博客  Ubuntu14.04安装Maven可以在命令行sudo apt-get -y install maven, 可以用 mvn -version查看mvn版本, 我的是Apache Maven 3.0.5.

安装, 配置Maven及Eclipse

(1)安装与配置Maven

  首先解压下载包,比如解压到/usr/lib/apache-maven-3.0.5目录下,然后修改/etc/profile文件,在里面添加以下内容:

#set Maven environment
export MAVEN_HOME=/usr/lib/apache-maven-3.0.5
export PATH=$PATH$:$MAVEN_HOME/bin

  输入以下命令使配置生效:

source /etc/profile

  同Windows下的验证方式一样,重启终端,执行 mvn --version 命令,若输出以下内容,则说明安装成功:

Apache Maven 3.0.5
.......

(2) 安装Eclipse

  同Windows环境下安装方式一样,直接解压即可使用.

最终安装完成的目录结构为(给了个模板,并不一定非要这样,比如sudo apt-get -y install maven是直接安装的,不用配置):

1.2 获取Hadoop源码

    当前比较流行的Hadoop源代码版本有两个:Apache  Hadoop 和 Cloudera Distributed Hadoop ( 简称 CDH ). Apache Hadoop 是由雅虎, Cloudera, Facebook等公司组成的 Hadoop社区共同研发的, 它属于最原始的开源版本, 在该版本基础上,很多公司进行了封装和优化, 推出自己的开源版本,其中,最有名的一个是Cloudera公司发布的CDH版本. 考虑到Apache Hadoop是最原始的版本,且使用最为广泛,因而本文选用了Apache Hadoop版本作为分析对象.

  hadoop 2.7.3 官网下载页面  .包括源码(source)和二进制(binary)文件, 其中二进制(binary)文件文件对应hadoop-2.7.3.tar.gz,解压后是hadoop-2.7.3, 是已经编译好的,可以直接进行部署. 而源码(source)文件对应hadoop-2.7.3-src.tar.gz,解压后是hadoop-2.7.3-src,还没编译好,需要先进行编译(用mvn命令),然后才能安装部署. 我们分析阅读以及修改都是针对源码(source)文件hadoop-2.7.3-src.tar.gz.

1.3 搭建Hadoop源码阅读环境

1.3.1 创建Hadoop工程

  本节将介绍如何创建一个Hadoop源代码工程以方便阅读源代码. 参考我的博客 Eclipse中导入Hadoop源代码工程 .

1.3.2 Hadoop 源代码阅读技巧

1.4 Hadoop源代码组织结构

  参见我的博客 Hadoop 源代码组织结构

1.5 Hadoop初体验

1.5.1 搭建Hadoop环境

  参见我的博客 hadoop2 Ubuntu 下安装部署

1.5.2 Hadoop Shell 介绍

  参见我的博客 Hadoop Shell 介绍

1.6 编译及调试Hadoop源代码

    读者在阅读源代码的过程中, 可能需要修改部分源代码或者使用调试工具以便跟踪某些变量值变化过程,此时要用到 Hadoop 源代码编译和调试方法. 其中调试方法主要介绍使用 Eclipse 远程调试工具和打印调试日志两种.

  Hadoop 天生支持 Linux 而对其他操作系统( 如 Windows ) 很不友好, 我们鼓励直接在 Linux 平台下编译和调试 Hadoop 源代码.

1.6.1 编译 Hadoop 源代码

  参见 hadoop-2.7.3 文件下的 BUILDING.txt 文件: ( 编译的时候要先把需要的环境准备好 ; 编译用到 Maven , 是根据 pom.xml 来进行编译的, 所以只要该文件夹下有 pom.xml 就可mvn编译 )

Build instructions for Hadoop

----------------------------------------------------------------------------------
Requirements: * Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )
* Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs )
* Internet connection for first build (to fetch all Maven and Hadoop dependencies) ----------------------------------------------------------------------------------
Installing required packages for clean install of Ubuntu 14.04 LTS Desktop: * Oracle JDK 1.7 (preferred)
$ sudo apt-get purge openjdk*
$ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java7-installer
* Maven
$ sudo apt-get -y install maven
* Native libraries
$ sudo apt-get -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev
* ProtocolBuffer 2.5.0 (required)
$ sudo apt-get -y install libprotobuf-dev protobuf-compiler Optional packages: * Snappy compression
$ sudo apt-get install snappy libsnappy-dev
* Bzip2
$ sudo apt-get install bzip2 libbz2-dev
* Jansson (C Library for JSON)
$ sudo apt-get install libjansson-dev
* Linux FUSE
$ sudo apt-get install fuse libfuse-dev ----------------------------------------------------------------------------------
Maven main modules: hadoop (Main Hadoop project)
- hadoop-project (Parent POM for all Hadoop Maven modules. )
(All plugins & dependencies versions are defined here.)
- hadoop-project-dist (Parent POM for modules that generate distributions.)
- hadoop-annotations (Generates the Hadoop doclet used to generated the Javadocs)
- hadoop-assemblies (Maven assemblies used by the different modules)
- hadoop-common-project (Hadoop Common)
- hadoop-hdfs-project (Hadoop HDFS)
- hadoop-mapreduce-project (Hadoop MapReduce)
- hadoop-tools (Hadoop tools like Streaming, Distcp, etc.)
- hadoop-dist (Hadoop distribution assembler) ----------------------------------------------------------------------------------
Where to run Maven from? It can be run from any module. The only catch is that if not run from utrunk
all modules that are not part of the build run must be installed in the local
Maven cache or available in a Maven repository. ----------------------------------------------------------------------------------
Maven build goals: * Clean : mvn clean
* Compile : mvn compile [-Pnative]
* Run tests : mvn test [-Pnative]
* Create JAR : mvn package
* Run findbugs : mvn compile findbugs:findbugs
* Run checkstyle : mvn compile checkstyle:checkstyle
* Install JAR in M2 cache : mvn install
* Deploy JAR to Maven repo : mvn deploy
* Run clover : mvn test -Pclover [-DcloverLicenseLocation=${user.name}/.clover.license]
* Run Rat : mvn apache-rat:check
* Build javadocs : mvn javadoc:javadoc
* Build distribution : mvn package [-Pdist][-Pdocs][-Psrc][-Pnative][-Dtar]
* Change Hadoop version : mvn versions:set -DnewVersion=NEWVERSION Build options: * Use -Pnative to compile/bundle native code
* Use -Pdocs to generate & bundle the documentation in the distribution (using -Pdist)
* Use -Psrc to create a project source TAR.GZ
* Use -Dtar to create a TAR with the distribution (using -Pdist) Snappy build options: Snappy is a compression library that can be utilized by the native code.
It is currently an optional component, meaning that Hadoop can be built with
or without this dependency. * Use -Drequire.snappy to fail the build if libsnappy.so is not found.
If this option is not specified and the snappy library is missing,
we silently build a version of libhadoop.so that cannot make use of snappy.
This option is recommended if you plan on making use of snappy and want
to get more repeatable builds. * Use -Dsnappy.prefix to specify a nonstandard location for the libsnappy
header files and library files. You do not need this option if you have
installed snappy using a package manager.
* Use -Dsnappy.lib to specify a nonstandard location for the libsnappy library
files. Similarly to snappy.prefix, you do not need this option if you have
installed snappy using a package manager.
* Use -Dbundle.snappy to copy the contents of the snappy.lib directory into
the final tar file. This option requires that -Dsnappy.lib is also given,
and it ignores the -Dsnappy.prefix option. OpenSSL build options: OpenSSL includes a crypto library that can be utilized by the native code.
It is currently an optional component, meaning that Hadoop can be built with
or without this dependency. * Use -Drequire.openssl to fail the build if libcrypto.so is not found.
If this option is not specified and the openssl library is missing,
we silently build a version of libhadoop.so that cannot make use of
openssl. This option is recommended if you plan on making use of openssl
and want to get more repeatable builds.
* Use -Dopenssl.prefix to specify a nonstandard location for the libcrypto
header files and library files. You do not need this option if you have
installed openssl using a package manager.
* Use -Dopenssl.lib to specify a nonstandard location for the libcrypto library
files. Similarly to openssl.prefix, you do not need this option if you have
installed openssl using a package manager.
* Use -Dbundle.openssl to copy the contents of the openssl.lib directory into
the final tar file. This option requires that -Dopenssl.lib is also given,
and it ignores the -Dopenssl.prefix option. Tests options: * Use -DskipTests to skip tests when running the following Maven goals:
'package', 'install', 'deploy' or 'verify'
* -Dtest=<TESTCLASSNAME>,<TESTCLASSNAME#METHODNAME>,....
* -Dtest.exclude=<TESTCLASSNAME>
* -Dtest.exclude.pattern=**/<TESTCLASSNAME1>.java,**/<TESTCLASSNAME2>.java ----------------------------------------------------------------------------------
Building components separately If you are building a submodule directory, all the hadoop dependencies this
submodule has will be resolved as all other 3rd party dependencies. This is,
from the Maven cache or from a Maven repository (if not available in the cache
or the SNAPSHOT 'timed out').
An alternative is to run 'mvn install -DskipTests' from Hadoop source top
level once; and then work from the submodule. Keep in mind that SNAPSHOTs
time out after a while, using the Maven '-nsu' will stop Maven from trying
to update SNAPSHOTs from external repos. ----------------------------------------------------------------------------------
Protocol Buffer compiler The version of Protocol Buffer compiler, protoc, must match the version of the
protobuf JAR. If you have multiple versions of protoc in your system, you can set in your
build shell the HADOOP_PROTOC_PATH environment variable to point to the one you
want to use for the Hadoop build. If you don't define this environment variable,
protoc is looked up in the PATH.
----------------------------------------------------------------------------------
Importing projects to eclipse When you import the project to eclipse, install hadoop-maven-plugins at first. $ cd hadoop-maven-plugins
$ mvn install Then, generate eclipse project files. $ mvn eclipse:eclipse -DskipTests At last, import to eclipse by specifying the root directory of the project via
[File] > [Import] > [Existing Projects into Workspace]. ----------------------------------------------------------------------------------
Building distributions: Create binary distribution without native code and without documentation: $ mvn package -Pdist -DskipTests -Dtar Create binary distribution with native code and with documentation: $ mvn package -Pdist,native,docs -DskipTests -Dtar Create source distribution: $ mvn package -Psrc -DskipTests Create source and binary distributions with native code and documentation: $ mvn package -Pdist,native,docs,src -DskipTests -Dtar Create a local staging version of the website (in /tmp/hadoop-site) $ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site ----------------------------------------------------------------------------------
Installing Hadoop Look for these HTML files after you build the document by the above commands. * Single Node Setup:
hadoop-project-dist/hadoop-common/SingleCluster.html * Cluster Setup:
hadoop-project-dist/hadoop-common/ClusterSetup.html ---------------------------------------------------------------------------------- Handling out of memory errors in builds ---------------------------------------------------------------------------------- If the build process fails with an out of memory error, you should be able to fix
it by increasing the memory used by maven -which can be done via the environment
variable MAVEN_OPTS. Here is an example setting to allocate between 256 and 512 MB of heap space to
Maven export MAVEN_OPTS="-Xms256m -Xmx512m" ---------------------------------------------------------------------------------- Building on Windows ----------------------------------------------------------------------------------
Requirements: * Windows System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
tools must be present on your PATH. Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/download/win. If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express. It does not support compiling for 64-bit,
which is problematic if running a 64-bit system. The Windows SDK 7.1 is free to
download here: http://www.microsoft.com/en-us/download/details.aspx?id=8279 The Windows SDK 8.1 is available to download at: http://msdn.microsoft.com/en-us/windows/bg162891.aspx Cygwin is neither required nor supported. ----------------------------------------------------------------------------------
Building: Keep the source code tree in a short path to avoid running into problems related
to Windows maximum path length limitation. (For example, C:\hdc). Run builds from a Windows SDK Command Prompt. (Start, All Programs,
Microsoft Windows SDK v7.1, Windows SDK 7.1 Command Prompt.) JAVA_HOME must be set, and the path must not contain spaces. If the full path
would contain spaces, then use the Windows short path instead. You must set the Platform environment variable to either x64 or Win32 depending
on whether you're running a 64-bit or 32-bit system. Note that this is
case-sensitive. It must be "Platform", not "PLATFORM" or "platform".
Environment variables on Windows are usually case-insensitive, but Maven treats
them as case-sensitive. Failure to set this environment variable correctly will
cause msbuild to fail while building the native code in hadoop-common. set Platform=x64 (when building on a 64-bit system)
set Platform=Win32 (when building on a 32-bit system) Several tests require that the user must have the Create Symbolic Links
privilege. All Maven goals are the same as described above with the exception that
native code is built by enabling the 'native-win' Maven profile. -Pnative-win
is enabled by default when building on Windows since the native components
are required (not optional) on Windows. If native code bindings for zlib are required, then the zlib headers must be
deployed on the build machine. Set the ZLIB_HOME environment variable to the
directory containing the headers. set ZLIB_HOME=C:\zlib-1.2.7 At runtime, zlib1.dll must be accessible on the PATH. Hadoop has been tested
with zlib 1.2.7, built using Visual Studio 2010 out of contrib\vstudio\vc10 in
the zlib 1.2.7 source tree. http://www.zlib.net/ ----------------------------------------------------------------------------------
Building distributions: * Build distribution with native code : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar]

BUILDING.txt

# 编译 Hadoop 源代码的一些命令  ( 可以在 Hadoop 源代码文件的任何含有 pom.xml 的文件夹下进行 mvn , 表示你对该文件及其以下的文件进行编译. 如果相对整个源代码进行编译,则要在 hadoop-2.7.3-src 目录下进行编译 )
# 第一步
$ su root # 我添加的, 切换 root 权限
# 第二步
# 可以根据情况使用以下的命令 , 我们以第一个为例
$ mvn package -Pdist -DskipTests -Dtar
# 以上如果显示 BUILD SUCCESS, 就表示编译成功 (第一次编译很慢, 要下载很多东西 )
----------------------------------------------------------------------------------
Building distributions: Create binary distribution without native code and without documentation: $ mvn package -Pdist -DskipTests -Dtar Create binary distribution with native code and with documentation: $ mvn package -Pdist,native,docs -DskipTests -Dtar Create source distribution: $ mvn package -Psrc -DskipTests Create source and binary distributions with native code and documentation: $ mvn package -Pdist,native,docs,src -DskipTests -Dtar Create a local staging version of the website (in /tmp/hadoop-site) $ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site ----------------------------------------------------------------------------------

  还可以参考我的博客 Hadoop 修改源码以及将修改后的源码应用到部署好的Hadoop中   以及 Eclipse中导入Hadoop源代码工程  
1.6.2 调试 Hadoop 源代码

  参见我的博客 调试 Hadoop 源代码

第1部分总结, 搭建一个高效的源代码学习环境是深入学习 Hadoop 的良好开端, 在笔者看来, 一个高效的 Hadoop 学习环境至少应该包括源代码阅读环境, Hadoop 使用环境和源代码编译调试环境.

2. Hadoop自带测试用例

  自带测试用例, 如 Wordcount, TestDFSIO 等等.

3. Hadoop源码分析

  参见我的博客 Hadoop 三大调度器源码分析及编写自己的调度器

  YARN(MapReduce 2)运行MapReduce的过程-源码分析

4. Hadoop开发

4.1 开发Hadoop应用程序

有两种方式,分别是4.1.1和4.1.2

4.1.1 Linux下使用Eclipse开发Hadoop应用程序使用Eclipse编译运行MapReduce程序 Hadoop2.6.0_Ubuntu/CentOS  这种需要在本机上已经部署好hadoop环境,可以直接在Eclipse开发Hadoop应用程序并运行. 需要在Eclipse引入一个jar包. hadoop 2.7.3 要引入hadoop-eclipse-plugin-2.7.3.jar .参考github上给的hadoop2x-eclipse-plugin,  可以下载别人弄好的, 也可以自己制作1  自己制作2  自己制作3.

  参见我的博客 hadoop 2.7.3 (hadoop2.x)使用ant制作eclipse插件hadoop-eclipse-plugin-2.7.3.jar

4.1.2 参考我的博客 Eclipse编写hadoop程序并打成jar包上传到hadoop集群运行,  这种方式不需要本机部署好hadoop环境, 只需要编写是引入hadoop-2.7.3-src中的一些jar包即可, 然后打包上传给集群.

4.2 修改Hadoop源码

  参见我的博客 Hadoop 修改源码以及将修改后的源码应用到部署好的Hadoop中  以及 调试 Hadoop 源代码

hadoop推荐的更多相关文章

  1. 开始hadoop

    hadoop介绍 分布式存储系统HDFS(Hadoop Distributed File System),提供了高可靠性.高扩展性和高吞吐率的数据存储服务: 资源管理系统YARN(Yet Anothe ...

  2. 【hadoop】 hadoop 单机伪分布式安装

    准备: 虚拟机(CentOS 6.9) JDK1.8 hadoop2.8.0 一.JDK安装及配置 rpm -ivh jdkxxxx 安装 配置环境变量 vim /etc/profile export ...

  3. 【甘道夫】基于Mahout0.9+CDH5.2执行分布式ItemCF推荐算法

    环境: hadoop-2.5.0-cdh5.2.0 mahout-0.9-cdh5.2.0 引言 尽管Mahout已经宣布不再继续基于Mapreduce开发,迁移到Spark.可是实际面临的情况是公司 ...

  4. hadoop之hdfs命令详解

    本篇主要对hadoop命令和hdfs命令进行阐述,yarn命令会在之后的文章中体现 hadoop fs命令可以用于其他文件系统,不止是hdfs文件系统内,也就是说该命令的使用范围更广可以用于HDFS. ...

  5. fluem读取文件并写入到hadoop的hdfs

    接上一章,本章介绍使用 crontab 像指定文件定时写入,使用fluem 读取并写入到hadoop的hdfs 前提准备已安装好fluem ,和hadoop(推荐单机即可毕竟做实验) 一.进入终端执行 ...

  6. Java 进程占用 VIRT 虚拟内存超高的问题研究

    1. 现象 最近发现线上机器 java 8 进程的 VIRT 虚拟内存使用达到了 50G+,如下图所示: 2. 不管用的 -Xmx 首先第一想到的当然使用 java 的 -Xmx 去限制堆的使用.但是 ...

  7. 为什么linux下多线程程序如此消耗虚拟内存【转】

    转自:http://blog.csdn.net/chen19870707/article/details/43202679 权声明:本文为博主原创文章,未经博主允许不得转载.   目录(?)[-] 探 ...

  8. [推荐]Hadoop+HBase+Zookeeper集群的配置

    [推荐]Hadoop+HBase+Zookeeper集群的配置 Hadoop+HBase+Zookeeper集群的配置  http://wenku.baidu.com/view/991258e881c ...

  9. 想从事分布式系统,计算,hadoop等方面,需要哪些基础,推荐哪些书籍?--转自知乎

    作者:廖君链接:https://www.zhihu.com/question/19868791/answer/88873783来源:知乎 分布式系统(Distributed System)资料 < ...

随机推荐

  1. Vue生命周期方法。

  2. appium第一个安卓自动化工程

    转自:https://university.utest.com/how-to-set-up-your-first-android-automation-project-with-appium/ App ...

  3. gradle配置远程仓库(以及使用本地maven仓库)

    allprojects{ repositories { mavenLocal() def REPOSITORY_URL = 'http://maven.aliyun.com/nexus/content ...

  4. Hadoop如何计算map数和reduce数

    阅读本文可以带着下面问题: 1.map和reduce的数量过多会导致什么情况? 2.Reduce可以通过什么设置来增加任务个数? 3.一个task的map数量由谁来决定? 4.一个task的reduc ...

  5. java zip压缩优化版 解决压缩后文件一直被占用无法删除

    最近进行zip操作,从网上找到一个处理方法,但是经过试验存在一些bug,主要是文件流的申明存在问题,导致jvm一直占用文件而不释放,特意把自己修改的发出来,已备记录 import java.io.Bu ...

  6. jws webservice code

    1.服务器端建立 1.1.创建接口 [java] view plaincopy @WebService  public interface IWebService {      int add(int ...

  7. $.post 使用案例

    $.post( aplnCommon.topUrl + 'ajaxLogin/ajaxLogin.action', { 'userLoginId' : userName, 'pwd' : userPw ...

  8. update外联表,用另一个表数据更新本表数据

    update s set s.classbid = lc.itemidfrom    dbo.Lv_servers as s INNER JOIN dbo.Lv_LineChannel as lc O ...

  9. div 下 的img水平居中

    设置text-align:center; 这个div必须要设置宽度: 如:{text-align:center; width:100%;}

  10. centreon问题总结

    1.SNMP TABLE ERROR : Requested table is empty or does not exist 这是SNMP的服务端查询客服端失败,失败的原理是权限不足 解决办法: v ...