搭建hadoop2.6.0集群环境分类： A1_HADOOP 2015-04-20 07:21 459人阅读评论(0) 收藏

一、规划

（一）硬件资源

10.171.29.191 master

10.171.94.155 slave1

10.251.0.197 slave3

（二）基本资料

用户： jediael

目录：/mnt/jediael/

二、环境配置

（一）统一用户名密码，并为jediael赋予执行所有命令的权限

#passwd

# useradd jediael

# passwd jediael

# vi /etc/sudoers

增加以下一行：

jediael ALL=(ALL) ALL

（二）创建目录/mnt/jediael

$sudo chown jediael:jediael /opt

$ cd /opt

$ sudo mkdir jediael

注意:/opt必须是jediael的，否则会在format namenode时出错。

（三）修改用户名及/etc/hosts文件

1、修改/etc/sysconfig/network

NETWORKING=yes

HOSTNAME=*******

2、修改/etc/hosts

10.171.29.191 master

10.171.94.155 slave1

10.251.0.197 slave3

注意hosts文件不能有127.0.0.1 *****配置，否则会导致出现异常。org.apache.hadoop.ipc.Client: Retrying connect to server: master/10.171.29.191:9000. Already trie

3、hostname命令

hostname ****

（四）配置免密码登录

以上命令在master上使用jediael用户执行：

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

然后，将authorized_keys复制到slave1,slave2

scp ~/.ssh/authorized_keys slave1:~/.ssh/

scp ~/.ssh/authorized_keys slave2:~/.ssh/

注意

（1）若提示.ssh目录不存在，则表示此机器从未运行过ssh，因此运行一次即可创建.ssh目录。

（2）.ssh/的权限为600,authorized_keys的权限为700，权限大了小了都不行。

（五）在3台机器上分别安装java，并设置相关环境变量

参考http://blog.csdn.net/jediael_lu/article/details/38925871

（六）下载hadoop-2.6.0.tar.gz，并将其解压到/mnt/jediael

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz

tar -zxvf hadoop-2.6.0.tar.gz

三、修改配置文件

【3台机器上均要执行，一般先在一台机器上配置完成，再用scp复制到其它机器】

（一）hadoop_env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_51

（二）修改core-site.xml

        <property>

                <name>hadoop.tmp.dir</name>

                <value>/mnt/tmp</value>

                <description>Abase for other temporary directories.</description>

        </property>

        <property>

                <name>fs.defaultFS</name>

                <value>hdfs://master:9000</value>

        </property>

        <property>

                <name>io.file.buffer.size</name>

                <value>4096</value>

        </property>

（三）修改hdfs-site.xml

        <property>

                <name>dfs.replication</name>

                <value>2</value>

        </property>

（四）修改mapred-site.xml

       <property>

                <name>mapreduce.framework.name</name>

                <value>yarn</value>

                <final>true</final>

        </property>

    <property>

        <name>mapreduce.jobtracker.http.address</name>

        <value>master:50030</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.address</name>

        <value>master:10020</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.webapp.address</name>

        <value>master:19888</value>

    </property>

        <property>

                <name>mapred.job.tracker</name>

                <value>http://master:9001</value>

        </property>

（五）修改yarn.xml

        <property>

                <name>yarn.resourcemanager.hostname</name>

                <value>master</value>

        </property>

    <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

    <property>

        <name>yarn.resourcemanager.address</name>

        <value>master:8032</value>

    </property>

    <property>

        <name>yarn.resourcemanager.scheduler.address</name>

        <value>master:8030</value>

    </property>

    <property>

        <name>yarn.resourcemanager.resource-tracker.address</name>

        <value>master:8031</value>

    </property>

    <property>

        <name>yarn.resourcemanager.admin.address</name>

        <value>master:8033</value>

    </property>

    <property>

        <name>yarn.resourcemanager.webapp.address</name>

        <value>master:8088</value>

    </property>

（六）修改slaves 【不用修改masters文件？？】

slaves:

slave1

slave3

四、启动并验证

1、格式化namenode

[jediael@master hadoop-1.2.1]$ bin/hadoop namenode -format

2、启动hadoop【此步骤只需要在master上执行】

[jediael@master hadoop-1.2.1]$ bin/start-all.sh

3、验证1：向hdfs中写入内容

[jediael@master hadoop-2.6.0]$ bin/hadoop fs -ls /

[jediael@master hadoop-2.6.0]$ bin/hadoop fs -mkdir /test

[jediael@master hadoop-2.6.0]$ bin/hadoop fs -ls /

Found 1 items

drwxr-xr-x   - jediael supergroup          0 2015-04-19 23:41 /test

4、验证：登录页面

NameNode    http://ip:50070

5、查看各个主机的java进程

（1）master:

$ jps

3694 NameNode

3882 SecondaryNameNode

7216 Jps

4024 ResourceManager

（2）slave1:

$ jps

1913 NodeManager

2673 Jps

1801 DataNode

（3）slave3:

$ jps

1942 NodeManager

2252 Jps

1840 DataNode

五、运行一个完整的mapreduce程序:运行自带的wordcount程序

$ bin/hadoop fs -mkdir /input

$ bin/hadoop fs -ls /

Found 2 items

drwxr-xr-x   - jediael supergroup          0 2015-04-20 18:04 /input

drwxr-xr-x   - jediael supergroup          0 2015-04-19 23:41 /test

$ bin/hadoop fs -copyFromLocal etc/hadoop/mapred-site.xml.template /input

$ pwd

/mnt/jediael/hadoop-2.6.0/share/hadoop/mapreduce

$ /mnt/jediael/hadoop-2.6.0/bin/hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount /input /output

15/04/20 18:15:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

15/04/20 18:15:48 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id

15/04/20 18:15:48 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=

15/04/20 18:15:49 INFO input.FileInputFormat: Total input paths to process : 1

15/04/20 18:15:49 INFO mapreduce.JobSubmitter: number of splits:1

15/04/20 18:15:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local657082309_0001

15/04/20 18:15:50 INFO mapreduce.Job: The url to track the job: http://localhost:8080/

15/04/20 18:15:50 INFO mapreduce.Job: Running job: job_local657082309_0001

15/04/20 18:15:50 INFO mapred.LocalJobRunner: OutputCommitter set in config null

15/04/20 18:15:50 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

15/04/20 18:15:50 INFO mapred.LocalJobRunner: Waiting for map tasks

15/04/20 18:15:50 INFO mapred.LocalJobRunner: Starting task: attempt_local657082309_0001_m_000000_0

15/04/20 18:15:50 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]

15/04/20 18:15:50 INFO mapred.MapTask: Processing split: hdfs://master:9000/input/mapred-site.xml.template:0+2268

15/04/20 18:15:51 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

15/04/20 18:15:51 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

15/04/20 18:15:51 INFO mapred.MapTask: soft limit at 83886080

15/04/20 18:15:51 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

15/04/20 18:15:51 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

15/04/20 18:15:51 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

15/04/20 18:15:51 INFO mapred.LocalJobRunner:

15/04/20 18:15:51 INFO mapred.MapTask: Starting flush of map output

15/04/20 18:15:51 INFO mapred.MapTask: Spilling map output

15/04/20 18:15:51 INFO mapred.MapTask: bufstart = 0; bufend = 1698; bufvoid = 104857600

15/04/20 18:15:51 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26213916(104855664); length = 481/6553600

15/04/20 18:15:51 INFO mapred.MapTask: Finished spill 0

15/04/20 18:15:51 INFO mapred.Task: Task:attempt_local657082309_0001_m_000000_0 is done. And is in the process of committing

15/04/20 18:15:51 INFO mapred.LocalJobRunner: map

15/04/20 18:15:51 INFO mapred.Task: Task 'attempt_local657082309_0001_m_000000_0' done.

15/04/20 18:15:51 INFO mapred.LocalJobRunner: Finishing task: attempt_local657082309_0001_m_000000_0

15/04/20 18:15:51 INFO mapred.LocalJobRunner: map task executor complete.

15/04/20 18:15:51 INFO mapred.LocalJobRunner: Waiting for reduce tasks

15/04/20 18:15:51 INFO mapred.LocalJobRunner: Starting task: attempt_local657082309_0001_r_000000_0

15/04/20 18:15:51 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]

15/04/20 18:15:51 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@39be5e01

15/04/20 18:15:51 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10

15/04/20 18:15:51 INFO reduce.EventFetcher: attempt_local657082309_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events

15/04/20 18:15:51 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local657082309_0001_m_000000_0 decomp: 1566 len: 1570 to MEMORY

15/04/20 18:15:51 INFO reduce.InMemoryMapOutput: Read 1566 bytes from map-output for attempt_local657082309_0001_m_000000_0

15/04/20 18:15:51 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1566, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->1566

15/04/20 18:15:51 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning

15/04/20 18:15:51 INFO mapred.LocalJobRunner: 1 / 1 copied.

15/04/20 18:15:51 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs

15/04/20 18:15:51 INFO mapred.Merger: Merging 1 sorted segments

15/04/20 18:15:51 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1560 bytes

15/04/20 18:15:51 INFO reduce.MergeManagerImpl: Merged 1 segments, 1566 bytes to disk to satisfy reduce memory limit

15/04/20 18:15:51 INFO reduce.MergeManagerImpl: Merging 1 files, 1570 bytes from disk

15/04/20 18:15:51 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce

15/04/20 18:15:51 INFO mapred.Merger: Merging 1 sorted segments

15/04/20 18:15:51 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1560 bytes

15/04/20 18:15:51 INFO mapred.LocalJobRunner: 1 / 1 copied.

15/04/20 18:15:51 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords

15/04/20 18:15:51 INFO mapreduce.Job: Job job_local657082309_0001 running in uber mode : false

15/04/20 18:15:51 INFO mapreduce.Job: map 100% reduce 0%

15/04/20 18:15:51 INFO mapred.Task: Task:attempt_local657082309_0001_r_000000_0 is done. And is in the process of committing

15/04/20 18:15:51 INFO mapred.LocalJobRunner: 1 / 1 copied.

15/04/20 18:15:51 INFO mapred.Task: Task attempt_local657082309_0001_r_000000_0 is allowed to commit now

15/04/20 18:15:51 INFO output.FileOutputCommitter: Saved output of task 'attempt_local657082309_0001_r_000000_0' to hdfs://master:9000/output/_temporary/0/task_local657082309_0001_r_000000

15/04/20 18:15:51 INFO mapred.LocalJobRunner: reduce > reduce

15/04/20 18:15:51 INFO mapred.Task: Task 'attempt_local657082309_0001_r_000000_0' done.

15/04/20 18:15:51 INFO mapred.LocalJobRunner: Finishing task: attempt_local657082309_0001_r_000000_0

15/04/20 18:15:51 INFO mapred.LocalJobRunner: reduce task executor complete.

15/04/20 18:15:52 INFO mapreduce.Job: map 100% reduce 100%

15/04/20 18:15:52 INFO mapreduce.Job: Job job_local657082309_0001 completed successfully

15/04/20 18:15:52 INFO mapreduce.Job: Counters: 38

        File System Counters

                FILE: Number of bytes read=544164

                FILE: Number of bytes written=1040966

                FILE: Number of read operations=0

                FILE: Number of large read operations=0

                FILE: Number of write operations=0

                HDFS: Number of bytes read=4536

                HDFS: Number of bytes written=1196

                HDFS: Number of read operations=15

                HDFS: Number of large read operations=0

                HDFS: Number of write operations=4

        Map-Reduce Framework

                Map input records=43

                Map output records=121

                Map output bytes=1698

                Map output materialized bytes=1570

                Input split bytes=114

                Combine input records=121

                Combine output records=92

                Reduce input groups=92

                Reduce shuffle bytes=1570

                Reduce input records=92

                Reduce output records=92

                Spilled Records=184

                Shuffled Maps =1

                Failed Shuffles=0

                Merged Map outputs=1

                GC time elapsed (ms)=123

                CPU time spent (ms)=0

                Physical memory (bytes) snapshot=0

                Virtual memory (bytes) snapshot=0

                Total committed heap usage (bytes)=269361152

        Shuffle Errors

                BAD_ID=0

                CONNECTION=0

                IO_ERROR=0

                WRONG_LENGTH=0

                WRONG_MAP=0

                WRONG_REDUCE=0

        File Input Format Counters

                Bytes Read=2268

        File Output Format Counters

$ /mnt/jediael/hadoop-2.6.0/bin/hadoop fs -cat /output/*

搭建hadoop2.6.0集群环境分类： A1_HADOOP 2015-04-20 07:21 459人阅读评论(0) 收藏的更多相关文章

Prime Path 分类：搜索 POJ 2015-08-09 16:21 4人阅读评论(0) 收藏
Prime Path Time Limit: 1000MS Memory Limit: 65536K Total Submissions: 14091 Accepted: 7959 Descripti ...
A Knight's Journey 分类： POJ 搜索 2015-08-08 07:32 2人阅读评论(0) 收藏
A Knight's Journey Time Limit: 1000MS Memory Limit: 65536K Total Submissions: 35564 Accepted: 12119 ...
JAVA swing中JPanel如何实现分组框的效果以及设置边框颜色分类： Java Game 2014-08-16 12:21 198人阅读评论(0) 收藏
代码如下: import java.awt.FlowLayout; import java.awt.Frame; import java.awt.GridLayout; import javax.sw ...
菊花加载第三方--MBprogressHUD 分类： ios技术 2015-02-05 19:21 120人阅读评论(0) 收藏
上次说到了网络请求AFN,那么我们在网络请求的时候,等待期间,为了让用户不认为是卡死或程序出错,一般都会放一个菊花加载,系统有一个菊花加载类叫UIProgressHUD.但是我今天要说的是一个替代它的 ...
博弈论入门小结分类： ACM TYPE 2014-08-31 10:15 73人阅读评论(0) 收藏
文章原地址:http://blog.csdn.net/zhangxiang0125/article/details/6174639 博弈论:是二人或多人在平等的对局中各自利用对方的策略变换自己的对抗策 ...
深入N皇后问题的两个最高效算法的详解分类： C/C++ 2014-11-08 17:22 117人阅读评论(0) 收藏
N皇后问题是一个经典的问题,在一个N*N的棋盘上放置N个皇后,每行一个并使其不能互相攻击(同一行.同一列.同一斜线上的皇后都会自动攻击). 一. 求解N皇后问题是算法中回溯法应用的一个经典案例回溯算 ...
Red and Black(BFS or DFS) 分类： dfs bfs 2015-07-05 22:52 2人阅读评论(0) 收藏
Description There is a rectangular room, covered with square tiles. Each tile is colored either red ...
【从0到1学Web前端】CSS定位问题二（float和display的使用）分类： HTML+CSS 2015-05-28 22:03 812人阅读评论(1) 收藏
display 属性规定元素应该生成的框的类型. 这个属性用于定义建立布局时元素生成的显示框类型.对于 HTML 等文档类型,如果使用 display 不谨慎会很危险,因为可能违反 HTML 中已经定 ...
循环队列分类： c/c++ 2014-10-10 23:28 605人阅读评论(0) 收藏
利用线性表实现队列,为了有效利用空间,将其设计为循环结构,防止假溢出:牺牲一个存储单元以区分队空.队满. 设front队头,rear队尾,N为顺序表大小队空:rear==front 队满:(rear ...

随机推荐

POJ 3039 搜索？？？（逼近）
思路: 抄的题解这叫搜索? 难以理解我觉得就是枚举+逼近 //By SiriusRen #include <cmath> #include <cstdio> #includ ...
关于html(meta的常用的用法)
http://www.haorooms.com/post/html_meta_ds
给网站设置ICO图标
方法一: 直接在站点根目录下放入名为:favicon.ico 的图标文件(必须要为 ICO 文件,BMP 及其他格式的图片文件不行).还有将 favicon.ico 中的 favico ...
SVN无法提交,路径找不到问题的解决方案
自己有了个云服务器,只在上面搭建了几个博客网站(小雷网等). 我自己的代码,保存在SVNChina上,每年99元.(私人的付费,开源的免费) 云服务器,闲着也是闲着,决定在上面搭建一个属于自己的SVN ...
android-pulltorefresh 下拉载入中使用gif动图
效果预览: xml布局 <com.handmark.pulltorefresh.library.PullToRefreshListView xmlns:android="http:// ...
联合体union用在何处？
程序设计刚開始学习的人在学习时,总想问:"这个东东有什么用?"于是,在建设有关的教学资源时,也便总从这个角度,试图给出一些案例,这是一个将刚開始学习的人作为教学目标人群的人该干的事 ...
JavaScript全讲-架构原则解析
因为近期一直在忙,非常久没有更新,见谅. 上篇我们讲完JavaScript函数式编程的特性,今天我们就来聊聊JavaScript中的架构. 提到JavaScript架构.非常多人会认为不可思议,由于架 ...
android缩放动画的两种实现方法
在android开发.我们会常常使用到缩放动画,普通情况下缩放动画有两种实现方式.一种是直接通过java代码去实现,第二种是通过配置文件实现动画,以下是两种动画的基本是用法: Java代码实现: // ...
php中this,self,parent三个关键字的区别辨析
php中this,self,parent三个关键字的区别辨析一.总结一句话总结:this是指向对象实例的一个指针,self是对类本身的一个引用,parent是对父类的引用. 1.self关键字使用 ...
elasticsearch transport 请求发送和处理
前一篇分析对nettytransport的启动及连接,本篇主要分析transport请求的发送和处理过程.cluster中各个节点之间需要相互发送很多信息,如master检测其它节点是否存在,node ...

搭建hadoop2.6.0集群环境 分类： A1_HADOOP 2015-04-20 07:21 459人阅读 评论(0) 收藏

搭建hadoop2.6.0集群环境 分类： A1_HADOOP 2015-04-20 07:21 459人阅读 评论(0) 收藏的更多相关文章

随机推荐

热门专题

搭建hadoop2.6.0集群环境分类： A1_HADOOP 2015-04-20 07:21 459人阅读评论(0) 收藏

搭建hadoop2.6.0集群环境分类： A1_HADOOP 2015-04-20 07:21 459人阅读评论(0) 收藏的更多相关文章