3.Hadoop完全分布式搭建

1.完全分布式搭建

配置

#cd /soft/hadoop/etc/

#mv hadoop local

#cp -r local full

#ln -s full hadoop

#cd hadoop

修改core-site.xml配置文件

#vim core-site.xml

[core-site.xml配置如下]

	<?xml version="1.0"?>

   <configuration>

   	<property>

   		<name>fs.defaultFS</name>

   		<value>hdfs://hadoop-1</value>

   	</property>

   </configuration>

修改hdfs-site.xml配置文件

#vim hdfs-site.xml

[hdfs-site.xml配置如下]

<?xml version="1.0"?>

<configuration>

	<property>

		<name>dfs.replication</name>

		<value>3</value>

	</property>

	<property>

		<name>dfs.namenode.secondary.http-address</name>

		<value>hadoop-2:50090</value>

	</description>

</property>

</configuration>

修改mapred-site.xml配置文件

#cp mapred-site.xml.template mapred-site.xml

#vim mapred-site.xml

[mapred-site.xml配置如下]

<?xml version="1.0"?>

<configuration>

	<property>

		<name>mapreduce.framework.name</name>

		<value>yarn</value>

	</property>

</configuration>

修改yarn-site.xml配置文件

#vim yarn-site.xml

  [yarn-site.xml配置如下]

<?xml version="1.0"?>

<configuration>

        <property>

                <name>yarn.resourcemanager.hostname</name>

                <value>hadoop-1</value>

        </property>

        <property>

                <name>yarn.nodemanager.aux-services</name>

                <value>mapreduce_shuffle</value>

        </property>

</configuration>

修改slaves配置文件

#vim slaves

[salves]

hadoop-2

hadoop-3

hadoop-4

hadoop-5

同步到其他节点

 #scp -r /soft/hadoop/etc/full  hadoop-2:/soft/hadoop/etc/

 #scp -r /soft/hadoop/etc/full  hadoop-3:/soft/hadoop/etc/

 #scp -r /soft/hadoop/etc/full  hadoop-4:/soft/hadoop/etc/

 #scp -r /soft/hadoop/etc/full  hadoop-5:/soft/hadoop/etc/

 #ssh hadoop-2 ln -s /soft/hadoop/etc/full /soft/hadoop/etc/hadoop

 #ssh hadoop-3 ln -s /soft/hadoop/etc/full /soft/hadoop/etc/hadoop

 #ssh hadoop-4 ln -s /soft/hadoop/etc/full /soft/hadoop/etc/hadoop

 #ssh hadoop-5 ln -s /soft/hadoop/etc/full /soft/hadoop/etc/hadoop

格式化hdfs分布式文件系统
```
#hadoop namenode -format
```

启动服务

[root@hadoop-1 hadoop]# start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

Starting namenodes on [hadoop-1]

hadoop-1: starting namenode, logging to /soft/hadoop-2.7.3/logs/hadoop-root-namenode-hadoop-1.out

hadoop-2: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop-2.out

hadoop-3: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop-3.out

hadoop-4: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop-4.out

hadoop-5: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop-5.out

Starting secondary namenodes [hadoop-2]

hadoop-2: starting secondarynamenode, logging to /soft/hadoop-2.7.3/logs/hadoop-root-secondarynamenode-hadoop-2.out

starting yarn daemons

starting resourcemanager, logging to /soft/hadoop-2.7.3/logs/yarn-root-resourcemanager-hadoop-1.out

hadoop-3: starting nodemanager, logging to /soft/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop-3.out

hadoop-4: starting nodemanager, logging to /soft/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop-4.out

hadoop-2: starting nodemanager, logging to /soft/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop-2.out

hadoop-5: starting nodemanager, logging to /soft/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop-5.out

查看服务运行状态

    [root@hadoop-1 hadoop]# jps

    16358 ResourceManager

    12807 NodeManager

    16011 NameNode

    16204 SecondaryNameNode

    16623 Jps

    hadoop-5 | SUCCESS | rc=0 >>

    16993 NodeManager

    16884 DataNode

    17205 Jps

    hadoop-1 | SUCCESS | rc=0 >>

    28520 ResourceManager

    28235 NameNode

    29003 Jps

    hadoop-2 | SUCCESS | rc=0 >>

    17780 Jps

    17349 DataNode

    17529 NodeManager

    17453 SecondaryNameNode

    hadoop-4 | SUCCESS | rc=0 >>

    17105 Jps

    16875 NodeManager

    16766 DataNode

    hadoop-3 | SUCCESS | rc=0 >>

    16769 DataNode

    17121 Jps

    16878 NodeManager

登陆WEB查看

2. 完全分布式单词统计

通过hadoop自带的demo运行单词统计

#mkdir /input

#cd /input/

#echo "hello world" > file1.txt

#echo "hello world" > file2.txt

#echo "hello world" > file3.txt

#echo "hello hadoop" > file4.txt

#echo "hello hadoop" > file5.txt

#echo "hello mapreduce" > file6.txt

#echo "hello mapreduce" > file7.txt

#hadoop dfs -mkdir /input

#hdfs dfs -ls /

#hadoop fs -ls /

#hadoop fs -put /input/* /input

#hadoop fs -ls /input

开始统计

[root@hadoop-1 ~]# hadoop jar /soft/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /input/ /output

17/05/14 23:01:07 INFO client.RMProxy: Connecting to ResourceManager at hadoop-1/10.31.133.19:8032

17/05/14 23:01:09 INFO input.FileInputFormat: Total input paths to process : 7

17/05/14 23:01:10 INFO mapreduce.JobSubmitter: number of splits:7

17/05/14 23:01:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1494773207391_0001

17/05/14 23:01:10 INFO impl.YarnClientImpl: Submitted application application_1494773207391_0001

17/05/14 23:01:11 INFO mapreduce.Job: The url to track the job: http://hadoop-1:8088/proxy/application_1494773207391_0001/

17/05/14 23:01:11 INFO mapreduce.Job: Running job: job_1494773207391_0001

17/05/14 23:01:23 INFO mapreduce.Job: Job job_1494773207391_0001 running in uber mode : false

17/05/14 23:01:23 INFO mapreduce.Job:  map 0% reduce 0%

17/05/14 23:01:56 INFO mapreduce.Job:  map 43% reduce 0%

17/05/14 23:01:57 INFO mapreduce.Job:  map 100% reduce 0%

17/05/14 23:02:04 INFO mapreduce.Job:  map 100% reduce 100%

17/05/14 23:02:05 INFO mapreduce.Job: Job job_1494773207391_0001 completed successfully

17/05/14 23:02:05 INFO mapreduce.Job: Counters: 50

        File System Counters

                FILE: Number of bytes read=184

                FILE: Number of bytes written=949365

                FILE: Number of read operations=0

                FILE: Number of large read operations=0

                FILE: Number of write operations=0

                HDFS: Number of bytes read=801

                HDFS: Number of bytes written=37

            HDFS: Number of read operations=24

            HDFS: Number of large read operations=0

            HDFS: Number of write operations=2

    Job Counters

            Killed map tasks=1

            Launched map tasks=7

            Launched reduce tasks=1

            Data-local map tasks=7

            Total time spent by all maps in occupied slots (ms)=216289

            Total time spent by all reduces in occupied slots (ms)=4827

            Total time spent by all map tasks (ms)=216289

            Total time spent by all reduce tasks (ms)=4827

            Total vcore-milliseconds taken by all map tasks=216289

            Total vcore-milliseconds taken by all reduce tasks=4827

            Total megabyte-milliseconds taken by all map tasks=221479936

            Total megabyte-milliseconds taken by all reduce tasks=4942848

    Map-Reduce Framework

            Map input records=7

            Map output records=14

            Map output bytes=150

            Map output materialized bytes=220

            Input split bytes=707

            Combine input records=14

            Combine output records=14

            Reduce input groups=4

            Reduce shuffle bytes=220

            Reduce input records=14

            Reduce output records=4

            Spilled Records=28

            Shuffled Maps =7

            Failed Shuffles=0

            Merged Map outputs=7

            GC time elapsed (ms)=3616

            CPU time spent (ms)=3970

            Physical memory (bytes) snapshot=1528823808

            Virtual memory (bytes) snapshot=16635846656

            Total committed heap usage (bytes)=977825792

    Shuffle Errors

            BAD_ID=0

            CONNECTION=0

            IO_ERROR=0

            WRONG_LENGTH=0

            WRONG_MAP=0

            WRONG_REDUCE=0

    File Input Format Counters

            Bytes Read=94

    File Output Format Counters

            Bytes Written=37

查看

[root@hadoop-1 ~]# hadoop fs -ls /out/put

Found 2 items

-rw-r--r--   3 root supergroup          0 2017-05-14 23:02 /out/put/_SUCCESS

-rw-r--r--   3 root supergroup         37 2017-05-14 23:02 /out/put/part-r-00000

[root@hadoop-1 ~]# hadoop fs -cat /out/put/part-r-00000

hadoop  2

hello   7

mapreduce       2

world   3

[root@hadoop-1 ~]#

3.hadoop完全分布式搭建的更多相关文章

hadoop完全分布式搭建HA（高可用）
2018年03月25日 16:25:26 D调的Stanley 阅读数:2725 标签: hadoop HAssh免密登录hdfs HA配置hadoop完全分布式搭建zookeeper 配置更多个 ...
超详细解说Hadoop伪分布式搭建--实战验证【转】
超详细解说Hadoop伪分布式搭建原文http://www.tuicool.com/articles/NBvMv2原原文 http://wojiaobaoshanyinong.iteye.com/b ...
Hadoop伪分布式搭建（一)
下面内容主要说明在Windows虚拟机上面,怎么搭建一个Hadoop伪分布式,并如何运行wordcount程序和网页查看HDFS文件系统. 1 相关软件下载和安装 APACH官网提供hadoop版本 ...
Hadoop伪分布式搭建步骤
说明: 搭建环境是VMware10下用的是Linux CENTOS 32位,Hadoop:hadoop-2.4.1 JAVA :jdk7 32位:本文是本人在网络上收集的HADOOP系列视频所附带的 ...
Hadoop 完全分布式搭建
搭建环境 https://www.cnblogs.com/YuanWeiBlogger/p/11456623.html 修改主机名------------------- 1./etc/hostname ...
hadoop 伪分布式搭建
下载hadoop1.0.4版本,和jdk1.6版本或更高版本:1. 安装JDK,安装目录大家可以自定义,下面是我的安装目录: /usr/jdk1.6.0_22 配置环境变量: [root@hadoop ...
Hadoop完全分布式搭建过程中遇到的问题小结
前一段时间,终于抽出了点时间,在自己本地机器上尝试搭建完全分布式Hadoop集群环境,也是借助网络上虾皮的Hadoop开发指南系列书籍一步步搭建起来的,在这里仅代表hadoop初学者向虾皮表示衷心的感 ...
Hadoop完全分布式搭建流程
centos7 搭建完全分布式 Hadoop 环境 SSR 前言本次教程是以先创建四台虚拟机为基础,再配置好一台虚拟机的情况下,直接复制文件到另外的虚拟机中(这样做大大简化了安装流程) 且本次 ...
Hadoop伪分布式搭建CentOS
所需软件及版本: jdk-7u80-linux-x64.tar.gz hadoop-2.6.0.tar.gz 1.安装JDK Hadoop 在需在JDK下运行,注意JDK最好使用Oracle的否则可能 ...

随机推荐

Undefined symbols for architecture arm64: "_OBJC_CLASS_$XXX", referenced from: objc-class-ref in XXX
ld: symbol(s) not found for architecture arm64 clang: error: linker command failed with exit code 1 ...
ELK6.7.0 Windows 环境本地安装
安装环境环境准备第一次写博文,排版比较乱,理解万岁 ELK 6.7.0三件套下载地址:https://www.elastic.co/cn/downloads/ windows环境下默认已配置jdk ...
Centos6.5 VM网络故障，可以Ping 通网关，无法上网或者访问其它网段
首先查看cat /etc/sysconfig/network-scripts/ifcfg-eth0 配置是否正确查看cat /etc/udev/rules.d/70-persistent-net. ...
jQuery 常用核心方法
jQuery 常用核心方法 .each() 遍历一个jQuery对象,为每个匹配元素执行一个函数 $('p').each(function(idx,node){ $(node).text(idx + ...
『Linux基础 - 1』计算机基础概念
这篇笔记的知识点结构目录: 认识计算机: (1)什么是计算机; (2)计算机的发展过程计算机的构成: (1) 计算机硬件系统; (2) 计算机软件系统二进制: (1) 为什么用二进制 (2) 二进 ...
kali aquatone安装
https://www.jianshu.com/p/418eedb9d9c8
20145234黄斐《Java程序设计》第四周学习总结
教材学习内容总结第六章部分 - 继承与多态何谓继承继承面向对象中,子类继承父类,避免重复的行为定义.一般来说,父类的父类也称父类,且同一个子类只允许拥有一个父类,而同一个父类则可以拥有多个子类. ...
nexys4开发板使用-第一篇（未完成）
1. 下去下个原理图.今天准备研究下DDR的控制,看介绍新一代的Nexys 4 DDR最值得被关注的改良是将原先的16 MiBCellularRAM升级为128 MiB的DDR2 SDRAM内存.Di ...
nexys4-DDR开发板数码管驱动-第二篇
1. 有这个板子使用的是Artix-7系列的XC7A100T-1CSG324C芯片.作为7系列中的一款FPGA,这个芯片的结构与Kintex-7和Virtex-7几乎一样.也配备了XADC.在Arti ...
使用keytool工具产生带根CA和二级CA的用户证书
使用keytool工具产生带根CA和二级CA的用户证书 1 生成根CA 1.1 生成根CA证书根CA实际是一张自签CA,自签CA的使用者和颁发者都是它自己.使用下面的命令生成根证书,如果没有指定 ...

3.hadoop完全分布式搭建

3.Hadoop完全分布式搭建

1.完全分布式搭建

2. 完全分布式单词统计

3.hadoop完全分布式搭建的更多相关文章

随机推荐

热门专题