Hadoop 全分布模式平台搭建

现将博客搬家至CSDN，博主改去CSDN玩玩~ 传送门：http://blog.csdn.net/sinat_28177969/article/details/54138163

Ps：主要答疑区在本帖最下方，疑点会标注出来。个人在配置过程中遇到的困难都会此列举。

实验介绍：

　　本次实验主要介绍了Hadoop平台的两个核心工具，HDFS和Mapreduce，结合这两个核心在Linux下搭建基于YARN集群的全分布模式的Hadoop架构。

　　实验案例，基于Hadoop平台下的Wordcount分词统计的试验

实验需求：

　　1、PC机，局域网服务，Linux系统　　

背景介绍：

　　Hadoop实现了一个分布式文件系统，简称HDFS。

　　HDFS有高容错性的特点，并且设计用来部署在普PC机上，而且它提供高吞吐量来访问应用程序的数据，适合那些有着超大数据集的应用程序。

　　HDFS放宽了POSIX的要求，可以以流的形式访问文件系统中的数据。

　　Hadoop的框架最核心的设计就是：

　　　　HDFS和MapReduce。

　　　　HDFS为海量的数据提供了存储，则MapReduce为海量的数据提供了计算。

　　开发者在熟练掌握了hadoop的使用后轻松地在Hadoop上开发和运行处理海量数据的应用程序

　　NameNode

　　NameNode 是一个通常在 HDFS 实例中的单独机器上运行的软件。

　　它负责管理文件系统名称空间和控制外部客户机的访问。NameNode 决定是否将文件映射到 DataNode 上的复制块上。

　　对于最常见的 3 个复制块，第一个复制块存储在同一机架的不同节点上，最后一个复制块存储在不同机架的某个节点上。

　　实际的 I/O事务并没有经过 NameNode，只有表示 DataNode 和块的文件映射的元数据经过 NameNode。

　　当外部客户机发送请求要求创建文件时，NameNode 会以块标识和该块的第一个副本的 DataNode IP 地址作为响应，这个 NameNode 还会通知其他将要接收该块的副本的 DataNode。

　　NameNode 在一个称为 FsImage 的文件中存储所有关于文件系统名称空间的信息。

　　这个文件和一个包含所有事务的记录文件（这里是 EditLog）将存储在 NameNode 的本地文件系统上。FsImage 和 EditLog 文件也需要复制副本，以防文件损坏或 NameNode 系统丢失。

　　NameNode本身不可避免地具有SPOF单点失效的风险，主备模式并不能解决这个问题，通过Hadoop Non-stop namenode才能实现100% uptime可用时间。

　　DataNode

　　DataNode 也是一个通常在 HDFS实例中的单独机器上运行的软件。

　　Hadoop 集群包含一个 NameNode 和大量 DataNode。

　　DataNode 通常以机架的形式组织，机架通过一个交换机将所有系统连接起来。

　　Hadoop 的一个假设是：机架内部节点之间的传输速度快于机架间节点的传输速度。

　　DataNode 响应来自 HDFS 客户机的读写请求。它们还响应来自 NameNode 的创建、删除和复制块的命令。

　　NameNode 依赖来自每个 DataNode 的定期心跳（heartbeat）消息。每条消息都包含一个块报告，NameNode 可以根据这个报告验证块映射和其他文件系统元数据。

　　如果 DataNode 不能发送心跳消息，NameNode 将采取修复措施，重新复制在该节点上丢失的块。

实验步骤及结果：

1.搭建平台（全分布式hadoop + eclipse Neon.1 + JDK1.8）

　　集群搭建：

　　主机两台（可拓展）：

　　（1）两个主机系统均为Ubuntu 16.04 LTS

　　　　详情：

　　　　　　master 192.168.:103.26（虚拟机）

　　　　　　slave2 192.168.103.22（物理机）

　　　　　　注：

　　　　　　　　（1）slave1是在同学的笔记本上，因为他的笔记本总是飘忽不定，所以这次博客上就先不写他的ip地址

　　　　　　　　（2）master是虚拟机的理由就是第一次尝试怕配错环境，导致崩溃，所以用了VMware为master，方便拯救平台

　　（2）hadoop平台版本都为最新稳定版2.7.3(解压及安装hadoop)　　　　　　

　　　　　　下载地址：Hadoop官网 http://hadoop.apache.org/releases.html

　　　　　　步骤1：点开网页以后，点击红色箭头所指的链接

　　　　　　步骤2：点开后如下图

　　　　　　步骤3：选择一个链接下载（个人推荐最后一个 tsinghua.edu.cn 清华大学链接源比较好）

　　　　　　步骤4：下载完后打开文件管理器，选择Downloads文件夹（如果修改主要文件夹名字为中文的，应选择“下载”）

　　　　　　步骤5：解压到指定路径

　　　　　　步骤5.1：在当前文件夹下右键 - 在终端打开键入su root命令

　　　　　　步骤5.2：输入root用户密码后，如下图所示

　　　　　　步骤5.3：键入解压命令

sudo tar zxvf hadoop-2.7.3.tar.gz -C /usr/local/hadoop

　　　　　　　　（注意：如果提示hadoop文件夹不存在的，可以在root用户下用cd命令到 /usr/local路径下键入 sudo mkdir /hadoop 创建夹）

　　　　　　步骤5.4：解压后如下图所示

　　　　　　　　（注意：路径满足如图所示即可，或自行定义）

　　　　至此hadoop前期下载准备工作已经完成。接下准备java环境的配置

　　（3）JDK版本为java8-oracle（配置java环境）

　　　　　（环境：系统稳定联网状态下）

　　　　　步骤1：打开终端键入命令(root用户模式可以不用加sudo前缀)

sudo add-apt-repository ppa:webupd8team/java

　　　　　　步骤2：出现一段文字后按回车继续

　　　　　　步骤3：继续键入命令

sudo apt-get update

　　　　　　步骤4：待系统加载完所有下载源

　　　　　　步骤5：键入安装命令

sudo apt-get install oracle-java8-installer

　　　　　　步骤6：等待下载结束（过程稍微有点漫长）

　　　　　　这个版本的java默认安装在 /usr/lib/jvm文件夹下

　　　　　　安装结束后配置环境变量

　　　　　　终端输入：

sudo gedit /etc/profile

　　　　　　步骤7：配置完后，按 ctrl + s 保存

　　　　　　步骤8：在终端中输入

sudo source /etc/profile

　　　　　　使配置的环境变量生效

　　　　　　步骤9：和在Windows下配置一样，在终端测试java和javac命令是否生效，在linux下可以多测试下jps命令看java进程号

　　　　　　至此java环境变量配置完毕

　　（4）SSH免密配置

　　　　SSH 是目前较可靠，专为远程登录会话和其他网络服务提供安全性的协议。利用 SSH 协议可以有效防止远程管理过程中的信息泄露问题。SSH最初是UNIX系统上的一个程序，后来又迅速扩展到其他操作平台。

　　　　SSH在正确使用时可弥补网络中的漏洞。SSH客户端适用于多种平台。

　　　　　　Ubuntu Linux下配置免密登录主要依靠 ssh localhost的命令

　　　　　　！！注意,如果改过 /etc/hosts 下的内容需要重新配置（下图是我的例子）

　　　　由于后期为了避免hadoop的一些端口和IP错误，所以我把localhost的名字改了，顺带把 /etc/hostname 的名字也改了。

　　　　改了上述的 hosts 和 hostname的名字后，记得重启电脑或者虚拟机

　　　　192.168.91.45是我虚拟的IP的地址名字叫master 相当于没有改变配置文件之前的 127.0.0.1 localhost

　　　　所以配置ssh免密的时候是键入 ssh master 而不是 ssh localhost

　　　　话不多说！

　　　　步骤0：SSH需要安装OpenSSH-server(如果已经安装则无需理会)

sudo apt-get install openssh-server

　　　　步骤1：在非root用户模式下打开终端键入ssh localhost（或者是定义的用户名）

　　　　步骤2：提示输入密码，输入你的ssh密码（自己记得住就好）

　　　　步骤3：输入完以后，测试一下ssh localhost（或是自定义名字），输入密码后是否如下图弹出一些信息

　　　　步骤4：如果下午所示后，则创建ssh成功

　　　　步骤5：创建免密登录（不需要关闭终端），键入如下命令

ssh-keygen -t rsa

　　　　步骤6：一直按回车直至出现RSA窗口即可

　　　　步骤7：键入命令

sudo cp .ssh/id_rsa.pub .ssh/authorized_keys

　　　　步骤8：验证免密登录，输入ssh localhost（或者自定义的名字），是否还需要输入密码登录

　　　　root用户下：

　　　　步骤1：进入root用户模式（用户模式下在终端键盘入：su root，输入root密码即可）

　　　　步骤2：进入ssh配置文件

gedit /etc/ssh/sshd_config

　　　　步骤3：把PermitRootLogin的字段改成 yes（原来的好像是Prohibit xxxx的），有点忘记了。总之改成yes就可以了

　　　　步骤4：保存退出终端

　　　　步骤5：打开新的终端键入命令

sudo service ssh restart

　　　　重启ssh服务之后，打开终端

　　　　步骤6：进入root用户模式下，键入 ssh localhost（或是你的自定义名字）

　　　　步骤7：输入自定义ssh密码后，与用户模式下的类似

　　　　步骤8：键入 ssh-keygen -t rsa 创建RSA密钥

　　　　步骤9：一直回车直至出现RSA密钥图，（如果提示Overwrite 输入 y 即可）

　　　　步骤10：键入配置免密的命令

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

　　　　步骤11：完成后，在root用户模式输入 ssh localhost（或自定义名字）后，如下图所示即可。

　　　　至此，root用户和普通用户模式的ssh免密配置完成。

　　（4.1）SSH免密配置(节点篇)

　　　　　　需求：如果每个节点都需要下载安装hadoop ，则大量耗费人力物力。

　　　　　　解决：所以需要一个SSH来远程发送hadoop包分发给每个节点。

　　　　　　接下来来讲解master打通每个节点的连接方式（单节点和多节点一样，只要配置好就可以进行连接）

　　　　　　步骤1：在hosts文件中配置好各子节点的ip地址以及名称（如下图）

　　　　　　步骤2：编辑好hosts文件保存并关闭，（root用户模式下）打开终端输入

ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave*

(星号代表子节点号码，或者把slave*换成自定义的名称)

　　　　　　步骤3：提示输入，子节点的登录密码，输入完成后，等待命令完成

　　　　　　步骤4：在终端中输入 ssh slave*(或者自定义名字)，如下图：

　　　　　步骤5：ssh打通master和子节点的通道，可以通过scp命令传输数据了。

　　　　　至此，完成对于子节点的ssh免密访问配置。

　　（5）hadoop平台版本都为最新稳定版2.7.3(解压及安装hadoop)

　　　　hadoop配置下主要注意配置文件路径的问题

　　　　主要包括：hadoop根目录下 /etc/hadoop 里面的xml配置文件

　　　　　　例：hadoop-env.sh , hdfs-site.xml, mapred-site.xml , core-site.xml , yarn-site.xml

　　　　　　注：mapred-site.xml需要复制出来到本路径，原本是mapred-site.xml.template 需要用 cp 命令复制并改名字

　　　　　　　　或者可以通过 gedit 命令创建一个新的mapred-site.xml，把模板内的内容复制过去，然后再进行配置

　　　　　　配置文件1：hadoop-env.sh（配置环境变量，让hadoop识别）

　　　　　　配置文件2：core-site.xml

<configuration>

    <property>

        <name>fs.default.name</name>

        <value>hdfs://master:8020</value>

    </property>

    <property>

        <name>io.file.buffer.size</name>

        <value>131072</value>

    </property>

    <property>

        <name>hadoop.tmp.dir</name>

        <value>/home/hadoop/tmp</value>

    </property>

    <property>

        <name>hadoop.proxyuser.hadoop.hosts</name>

        <value>*</value>

    </property>

    <property>

        <name>hadoop.proxyuser.hadoop.group</name>

        <value>*</value>

    </property>

    <property>

        <name>dfs.permissions</name>

        <value>false</value>

    </property>

</configuration>

　　　　　　配置文件3：hdfs-site.xml

<configuration>

    <property>

        <name>dfs.namenode.secondary.http-address</name>

        <value>master:9000</value>

    </property>

    <property>

        <name>dfs.namenode.name.dir</name>

        <value>file:/home/hadoop/dfs/name</value>

    </property>

    <property>

        <name>dfs.datanode.data.dir</name>

        <value>file:/home/hadoop/dfs/data</value>

    </property>

    <property>

        <name>dfs.replication</name>

        <value>3</value>

    </property>

    <property>

        <name>dfs.webhdfs.enabled</name>

        <value>true</value>

    </property>

</configuration>

　　　　　　配置文件4： mapred-site.xml

<configuration>

    <property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

    </property>

    <property>

        <name>mapred.job.tracker</name>

        <value>master:9001</value>

    </property>

    <property>

        <name>mapred.job.tracker.http.address</name>

        <value>master:50030</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.address</name>

        <value>master:10020</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.webapp.address</name>

        <value>master:19888</value>

    </property>

</configuration>

　　　　　　配置文件5：yarn-site.xml

<configuration>

    <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

    <property>

        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

        <value>org.apache.hadoop.mapred.ShuffleHandler</value>

    </property>

    <property>

        <name>yarn.resourcemanager.address</name>

        <value>master:8032</value>

    </property>

    <property>

        <name>yarn.resourcemanager.scheduler.address</name>

        <value>master:8030</value>

    </property>

    <property>

        <name>yarn.resourcemanager.resource-tracker.address</name>

        <value>master:8031</value>

    </property>

    <property>

        <name>yarn.resourcemanager.admin.address</name>

        <value>master:8033</value>

    </property>

    <property>

        <name>yarn.resourcemanager.webapp.address</name>

        <value>master:8088</value>

    </property>

</configuration>

　　　　　　！！！注：如果是master节点（即服务器）需要添加多一个slaves文件指定slave

　　　　　　配置文件6：slaves（选）

slave2 192.168.90.33

　　　　最后步骤：以上配置文件配置完毕后打开终端窗口，输入

hadoop namenode -format

　　　　出现如下结果，没有JAVA报错即可

　　　　初始化hadoop namenode节点成功！

　　　　打开终端利用 cd 命令进入hadoop启动命令文件下

cd /usr/local/hadoop/hadoop-2.7.3/sbin

　　　　键入如下命令启动hadoop（root用户模式下）　　　　

./start-all.sh

　　　　关闭hadoop则键入命令关闭

./stop-all.sh

　　　　输入jps在master节点测试，如果如上图所示则测试成功

　　　　在ssh slave2 节点输入jps测试

　　　　通过hadoop 自带命令

hadoop dfsadmin -report

　　　　如上图所示输出Live Datanodes，说明有存活节点，死节点为空。

　　　　证明集群配置成功！

　　（6）集群安装hadoop（完成Master节点的hadoop安装以及SSH的搭建）　　　　

　　　　构建好master与各个slave之间的ssh通信，如下图所示

　　　　步骤1：测试ssh命令与各节点间的通信

　　　　步骤2：确认本机的hadoop安装地址

　　　　步骤3：

scp –r /usr/local/hadoop/ root@slaver2:/usr/local/hadoop

　　　　把master上的hadoop分发给slave2节点（其他节点依次类推，只要搭好ssh就可以传输）。

　　　　传输过程有点久，耐心等候。

　　　　步骤4：在slave节点上配置环境变量

HADOOP_HOME=/usr/local/Hadoop  PATH=$PATH:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin

　　　　步骤5：在master启动hadoop进行测试

Namenode界面 50070端口

hadoop管理界面 8088端口

hadoop SecondaryNamenode 管理界面端口9000

　　（7）hadoop Wordcount测试（完成eclipse和eclipse hadoop插件安装）

　　　　步骤0：安装eclipse和eclipse hadoop插件

　　　　　　步骤0.1：安装eclipse

　　　　下载后，解压到自定义路径，解压后如所示

　　　　　　在此给出eclipse hadoop插件下载（http://pan.baidu.com/s/1mi6UP5I）

　　　　　　下载后，把jar放到eclipse根目录的dropins的目录

　　　　　　在根目录进入终端，进入root用户模式，输入

./eclipse

　　　　　　进入eclipse界面，完成安装。

　　　　步骤1：启动hadoop完成上述集群测试

　　　　步骤2：通过终端把测试数据 test.txt上传到hdfs中（test.txt为hadoop跟目录下的NOTICE.txt）

　　　　　　步骤2.1：在hdfs目录下创建input文件夹

hadoop fs -mkdir /input

hadoop fs -put test.txt /input

　　　　如图所示，则上传成功。

　　　　如果权限不对的话可以修改权限

hadoop fs -chmod -R  /input/test.txt

　　　　步骤3：打开eclipse，并完成mapreduce的wordcount代码，完成eclipse hadoop的配置

　　　步骤4：确保左上角的DFS Location能够显示hdfs中的文件目录

WordCount代码：

package org.apache.hadoop.examples;

import java.io.IOException;

import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

    public static class TokenizerMapper

    extends Mapper<Object, Text, Text, IntWritable>{

        private final static IntWritable one = new IntWritable();

        private Text word = new Text();

        public void map(Object key, Text value, Context context) throws IOException, InterruptedException {

            StringTokenizer itr = new StringTokenizer(value.toString());

            while (itr.hasMoreTokens())

            {

                word.set(itr.nextToken());

                context.write(word, one);

            }

        }

    }

    public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable>

    {

        private IntWritable result = new IntWritable();

        public void reduce(Text key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException

        {

            int sum = ;

            for (IntWritable val : values)

            {

                sum += val.get();

            }

            result.set(sum);

            context.write(key, result);

        }

    }

    public static void main(String[] args) throws Exception

    {

        Configuration conf = new Configuration();

        Job job = Job.getInstance(conf, "word count");

        job.setJarByClass(WordCount.class);

        job.setMapperClass(TokenizerMapper.class);

        job.setCombinerClass(IntSumReducer.class);

        job.setReducerClass(IntSumReducer.class);

        job.setOutputKeyClass(Text.class);

        job.setOutputValueClass(IntWritable.class);

        FileInputFormat.addInputPath(job, new Path(args[]));

        FileOutputFormat.setOutputPath(job, new Path(args[]));

        System.exit(job.waitForCompletion(true) ?  : );

    }

}

log4j日志文件：

log4j.rootLogger=debug, stdout, R

#log4j.rootLogger=stdout, R

log4j.appender.stdout=org.apache.log4j.ConsoleAppender

log4j.appender.stdout.layout=org.apache.log4j.PatternLayout

#log4j.appender.stdout.layout.ConversionPattern=%5p - %m%n

log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - %m%n

log4j.appender.R=org.apache.log4j.RollingFileAppender

log4j.appender.R.File=log4j.log

log4j.appender.R.MaxFileSize=100KB

log4j.appender.R.MaxBackupIndex=1

log4j.appender.R.layout=org.apache.log4j.PatternLayout

#log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n

log4j.appender.R.layout.ConversionPattern=%d %p [%c] - %m%n

#log4j.logger.com.codefutures=DEBUG

　　　　步骤5：配置Run Configuration

　　　　步骤6：右键Run As - Run On Hadoop（日志我选用了DEBUG模式测试，所以会很长，但是方便测试）

　　　　此时，master hdfs多出一个文件夹存放分词结果

　　　　下列图为结果部分截图：

　　　　至此，从安装到mapreduce Wordcount测试全部结束了。

　　　　hadoop2.7.6全模式下，结合eclipse hadoop插件配置，完成Wordcount测试。

实验结果分析：

1、Wordcount项目代码是结合Map-reduce的核心思想，以及对于Java输入输出流的认识所编写，也参考了一下"大牛"博客编写的，能够基本实现分词-词频统计。

2、小项目的分词的效果显然没有Python Jieba分词来的精确，但是基于Hadoop Mapreduce的运算，分词一篇词汇众多的文档只需要5秒。（如需查看请点开。文档来源：Hadoop LICENSE.txt）

Apache License

Version 2.0, January 2004

http://www.apache.org/licenses/

TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

1. Definitions.

"License" shall mean the terms and conditions for use, reproduction,

and distribution as defined by Sections 1 through 9 of this document.

"Licensor" shall mean the copyright owner or entity authorized by

the copyright owner that is granting the License.

"Legal Entity" shall mean the union of the acting entity and all

other entities that control, are controlled by, or are under common

control with that entity. For the purposes of this definition,

"control" means (i) the power, direct or indirect, to cause the

direction or management of such entity, whether by contract or

otherwise, or (ii) ownership of fifty percent (50%) or more of the

outstanding shares, or (iii) beneficial ownership of such entity.

"You" (or "Your") shall mean an individual or Legal Entity

exercising permissions granted by this License.

"Source" form shall mean the preferred form for making modifications,

including but not limited to software source code, documentation

source, and configuration files.

"Object" form shall mean any form resulting from mechanical

transformation or translation of a Source form, including but

not limited to compiled object code, generated documentation,

and conversions to other media types.

"Work" shall mean the work of authorship, whether in Source or

Object form, made available under the License, as indicated by a

(an example is provided in the Appendix below).

"Derivative Works" shall mean any work, whether in Source or Object

form, that is based on (or derived from) the Work and for which the

editorial revisions, annotations, elaborations, or other modifications

represent, as a whole, an original work of authorship. For the purposes

of this License, Derivative Works shall not include works that remain

separable from, or merely link (or bind by name) to the interfaces of,

the Work and Derivative Works thereof.

"Contribution" shall mean any work of authorship, including

the original version of the Work and any modifications or additions

to that Work or Derivative Works thereof, that is intentionally

submitted to Licensor for inclusion in the Work by the copyright owner

or by an individual or Legal Entity authorized to submit on behalf of

the copyright owner. For the purposes of this definition, "submitted"

means any form of electronic, verbal, or written communication sent

to the Licensor or its representatives, including but not limited to

communication on electronic mailing lists, source code control systems,

and issue tracking systems that are managed by, or on behalf of, the

Licensor for the purpose of discussing and improving the Work, but

excluding communication that is conspicuously marked or otherwise

designated in writing by the copyright owner as "Not a Contribution."

"Contributor" shall mean Licensor and any individual or Legal Entity

on behalf of whom a Contribution has been received by Licensor and

subsequently incorporated within the Work.

2. Grant of Copyright License. Subject to the terms and conditions of

this License, each Contributor hereby grants to You a perpetual,

worldwide, non-exclusive, no-charge, royalty-free, irrevocable

publicly display, publicly perform, sublicense, and distribute the

Work and such Derivative Works in Source or Object form.

3. Grant of Patent License. Subject to the terms and conditions of

this License, each Contributor hereby grants to You a perpetual,

worldwide, non-exclusive, no-charge, royalty-free, irrevocable

(except as stated in this section) patent license to make, have made,

use, offer to sell, sell, import, and otherwise transfer the Work,

where such license applies only to those patent claims licensable

by such Contributor that are necessarily infringed by their

Contribution(s) alone or by combination of their Contribution(s)

with the Work to which such Contribution(s) was submitted. If You

institute patent litigation against any entity (including a

cross-claim or counterclaim in a lawsuit) alleging that the Work

or a Contribution incorporated within the Work constitutes direct

or contributory patent infringement, then any patent licenses

granted to You under this License for that Work shall terminate

as of the date such litigation is filed.

4. Redistribution. You may reproduce and distribute copies of the

Work or Derivative Works thereof in any medium, with or without

modifications, and in Source or Object form, provided that You

meet the following conditions:

(a) You must give any other recipients of the Work or

Derivative Works a copy of this License; and

(b) You must cause any modified files to carry prominent notices

stating that You changed the files; and

that You distribute, all copyright, patent, trademark, and

attribution notices from the Source form of the Work,

excluding those notices that do not pertain to any part of

the Derivative Works; and

(d) If the Work includes a "NOTICE" text file as part of its

distribution, then any Derivative Works that You distribute must

include a readable copy of the attribution notices contained

within such NOTICE file, excluding those notices that do not

pertain to any part of the Derivative Works, in at least one

of the following places: within a NOTICE text file distributed

as part of the Derivative Works; within the Source form or

documentation, if provided along with the Derivative Works; or,

within a display generated by the Derivative Works, if and

wherever such third-party notices normally appear. The contents

of the NOTICE file are for informational purposes only and

do not modify the License. You may add Your own attribution

notices within Derivative Works that You distribute, alongside

or as an addendum to the NOTICE text from the Work, provided

that such additional attribution notices cannot be construed

as modifying the License.

You may add Your own copyright statement to Your modifications and

may provide additional or different license terms and conditions

for use, reproduction, or distribution of Your modifications, or

for any such Derivative Works as a whole, provided Your use,

reproduction, and distribution of the Work otherwise complies with

the conditions stated in this License.

5. Submission of Contributions. Unless You explicitly state otherwise,

any Contribution intentionally submitted for inclusion in the Work

by You to the Licensor shall be under the terms and conditions of

this License, without any additional terms or conditions.

Notwithstanding the above, nothing herein shall supersede or modify

the terms of any separate license agreement you may have executed

with Licensor regarding such Contributions.

6. Trademarks. This License does not grant permission to use the trade

names, trademarks, service marks, or product names of the Licensor,

except as required for reasonable and customary use in describing the

origin of the Work and reproducing the content of the NOTICE file.

7. Disclaimer of Warranty. Unless required by applicable law or

agreed to in writing, Licensor provides the Work (and each

Contributor provides its Contributions) on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or

implied, including, without limitation, any warranties or conditions

of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A

PARTICULAR PURPOSE. You are solely responsible for determining the

appropriateness of using or redistributing the Work and assume any

risks associated with Your exercise of permissions under this License.

8. Limitation of Liability. In no event and under no legal theory,

whether in tort (including negligence), contract, or otherwise,

unless required by applicable law (such as deliberate and grossly

negligent acts) or agreed to in writing, shall any Contributor be

liable to You for damages, including any direct, indirect, special,

incidental, or consequential damages of any character arising as a

result of this License or out of the use or inability to use the

Work (including but not limited to damages for loss of goodwill,

work stoppage, computer failure or malfunction, or any and all

other commercial damages or losses), even if such Contributor

has been advised of the possibility of such damages.

9. Accepting Warranty or Additional Liability. While redistributing

the Work or Derivative Works thereof, You may choose to offer,

and charge a fee for, acceptance of support, warranty, indemnity,

or other liability obligations and/or rights consistent with this

License. However, in accepting such obligations, You may act only

on Your own behalf and on Your sole responsibility, not on behalf

of any other Contributor, and only if You agree to indemnify,

defend, and hold each Contributor harmless for any liability

incurred by, or claims asserted against, such Contributor by reason

of your accepting any such warranty or additional liability.

END OF TERMS AND CONDITIONS

APPENDIX: How to apply the Apache License to your work.

To apply the Apache License to your work, attach the following

boilerplate notice, with the fields enclosed by brackets "[]"

replaced with your own identifying information. (Don't include

the brackets!) The text should be enclosed in the appropriate

comment syntax for the file format. We also recommend that a

file or class name and description of purpose be included on the

same "printed page" as the copyright notice for easier

identification within third-party archives.

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

APACHE HADOOP SUBCOMPONENTS:

The Apache Hadoop project contains subcomponents with separate copyright

notices and license terms. Your use of the source code for the these

subcomponents is subject to the terms and conditions of the following

licenses.

For the org.apache.hadoop.util.bloom.* classes:

/**

* 034819 (http://www.one-lab.org)

* Redistribution and use in source and binary forms, with or

* without modification, are permitted provided that the following

* conditions are met:

* - Redistributions of source code must retain the above copyright

* notice, this list of conditions and the following disclaimer.

* - Redistributions in binary form must reproduce the above copyright

* notice, this list of conditions and the following disclaimer in

* the documentation and/or other materials provided with the distribution.

* - Neither the name of the University Catholique de Louvain - UCL

* nor the names of its contributors may be used to endorse or

* promote products derived from this software without specific prior

* written permission.

* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS

* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT

* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS

* FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE

* COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,

* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,

* BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;

* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER

* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT

* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN

* ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE

* POSSIBILITY OF SUCH DAMAGE.

For portions of the native implementation of slicing-by-8 CRC calculation

in src/main/native/src/org/apache/hadoop/util:

/**

* BSD-style license that can be found in the LICENSE file.

For src/main/native/src/org/apache/hadoop/io/compress/lz4/{lz4.h,lz4.c,lz4hc.h,lz4hc.c},

LZ4 - Fast LZ compression algorithm

Header File

BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)

Redistribution and use in source and binary forms, with or without

modification, are permitted provided that the following conditions are

met:

* Redistributions of source code must retain the above copyright

notice, this list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above

in the documentation and/or other materials provided with the

distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS

"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT

LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR

A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT

OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,

SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT

LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,

DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY

THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT

(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE

OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

You can contact the author at :

- LZ4 source repository : http://code.google.com/p/lz4/

- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c

For hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/util/tree.h

---------------------------------------------------------------------

Redistribution and use in source and binary forms, with or without

modification, are permitted provided that the following conditions

are met:

1. Redistributions of source code must retain the above copyright

notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright

notice, this list of conditions and the following disclaimer in the

documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR

IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES

OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.

IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,

INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT

NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,

DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY

THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT

(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF

THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

The binary distribution of this product bundles binaries of leveldbjni

(https://github.com/fusesource/leveldbjni), which is available under the

following license:

Redistribution and use in source and binary forms, with or without

modification, are permitted provided that the following conditions are

met:

* Redistributions of source code must retain the above copyright

notice, this list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above

in the documentation and/or other materials provided with the

distribution.

* Neither the name of FuseSource Corp. nor the names of its

contributors may be used to endorse or promote products derived from

this software without specific prior written permission.