hadoop单击模式环境搭建

一 安装jdk

下载相应版本的jdk安装到相应目录,我的安装目录是/usr/lib/jdk1.8.0_40

下载完成后,在/etc/profile中设置一下环境变量,在文件最后追加如下内容

export JAVA_HOME=/usr/lib/jdk1..0_40
export JRE_HOME=/usr/lib/jdk1..0_40/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

二 安装ssh---------sudo apt-get install ssh

主要使用其管理远端守护进程,这里是单击模式,所以,不重要.

三 下载hadoop

http://hadoop.apache.org/releases.html

建议下载稳定版本的,我下载的是hadoop2.6.4,并把它放在了/usr/local/目录下

hadoop运行在apache服务器上的,需要java环境的支持,所以,下载的hadoop需要配置java环境变量,使java认识hadoop,同时也要使hadoop放到java环境中.

1 设置 ~/.bashrc,为登录的hadoop用户设置环境变量

export JAVA_HOME=/usr/lib/jdk1..0_40
 
export HADOOP_INSTALL=/usr/local/hadoop-2.6.
 
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$JAVA_HOME/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"

设置完成之后,要运行

source ~/.bashrc

使设置的环境变量生效

2 配置hadoop

在   /usr/local/hadoop-2.6.4/etc/hadoop/下打开hadoop-env.sh

export JAVA_HOME=/usr/lib/jdk1..0_40
export JRE_HOME=/usr/lib/jdk1..0_40/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

到这里hadoop单击模式就配置好了

运行

./bin/hadoop version

可看到如下信息

Hadoop 2.6.
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 5082c73637530b0b7e115f9625ed7fac69f937e6
Compiled by jenkins on --12T09:45Z
Compiled with protoc 2.5.
From source with checksum 8dee2286ecdbbbc930a6c87b65cbc010
This command was run using /usr/local/hadoop-2.6./share/hadoop/common/hadoop-common-2.6..jar

说明hadoop配置好了

下面来运行一下hadoop自带的wordcount程序检验一下

1 在hadoop目录下创建input文件夹,将/etc/hadoop中的配置文件复制到里面作为待测文件

mkdir input

cp etc/hadoop/* input/

2 运行程序,计数

在hadoop目录下运行命令

./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6..jar grep input output '[a-z.]+'

意思是,通过example那个jar包,将a-z开头的单词数统计出来

看到如下运行信息

    File System Counters
        FILE: Number of bytes read=
        FILE: Number of bytes written=
        FILE: Number of read operations=
        FILE: Number of large read operations=
        FILE: Number of write operations=
    Map-Reduce Framework
        Map input records=
        Map output records=
        Map output bytes=
        Map output materialized bytes=
        Input split bytes=
        Combine input records=
        Combine output records=
        Reduce input groups=
        Reduce shuffle bytes=
        Reduce input records=
        Reduce output records=
        Spilled Records=
        Shuffled Maps =
        Failed Shuffles=
        Merged Map outputs=
        GC time elapsed (ms)=
        CPU time spent (ms)=
        Physical memory (bytes) snapshot=
        Virtual memory (bytes) snapshot=
        Total committed heap usage (bytes)=
    Shuffle Errors
        BAD_ID=
        CONNECTION=
        IO_ERROR=
        WRONG_LENGTH=
        WRONG_MAP=
        WRONG_REDUCE=
    File Input Format Counters
        Bytes Read=
    File Output Format Counters
        Bytes Written=

说明运行成功

查看运行结果

cat output/*

再次运行的话,需要 rm -r output/ 删除output文件夹才能再次运行