Hadoop环境搭建及wordcount程序
目的: 前期学习了一些机器学习基本算法,实际企业应用中算法是核心,运行的环境和数据处理的平台是基础。
手段: 搭建简易hadoop集群(由于机器限制在自己的笔记本上通过虚拟机搭建)
一、基础环境介绍
win10
vmware15.0.0
3 ubuntu 虚拟机(1 台作为master ,另外2台作为 slave1、slave2)
hadoop2.8.5
jdk1.8
二、搭建步骤
1. 安装vmware ,安装ubuntu 先安装一台,后面配置完成后直接克隆 (此处不作详细介绍,可参考其它文档进行搭建)
2. linux基础环境配置
a) 创建用户 test 执行所有安装相关操作 :
sudo useradd -m test -s /bin/bash
sudo passwd hadoop
b)安装基础软件
. 基础工具
. sudo apt-get install vim (edit tools) . sudo apt-get install openssh-client openssh-server (openssh service for log in the server via ssh) . sudo apt-get install nfs-common (for nfs mounting ) . sudo apt-get install git (for git tool) .Setup nfs service on Ubuntu for mounting
. sudo apt-get install nfs-kernel-server (install nfs server) . sudo mkdir /nfsroot; . sudo chmod /nfsroot ( create /nfsroot fold as mounting directory) . sudo vim /etc/exports (config the mount directory) add below line in /etc/exports: /nfsroot *(rw,sync,no_root_squash) . sudo service nfs-kernel-server restart (restart nfs service) . setup samba service for share folders with windows OS
. sudo apt-get install samba smbclient (install necessay tools) . sudo apt-get install samba smbclient (config the samba server) . Add following lines in /etc/samba//smb.conf: [nfsroot]
comment = nfsroot
path = /nfsroot
public = yes
guest ok = yes
browseable = yes
writeable = yes . sudo service smbd restart (restart the samba service)
c) 配置服务器之间免密互相访问(通过公钥私钥的方式)
ssh-keygen -t rsa # 会有提示,都按回车就可以
cat id_rsa.pub >> authorized_keys # 加入授权
当所有节点都克隆完成后可以测试ssh登录: ssh 192.168.xx.xxx@test
3. 配置java和hadoop软件
下载jdk1.8 解压文件放在 /opt/java 目录下,并配置环境变量 (java –version 进行测试)
下载hadoop2.8.5 解压文件放在 /opt/hadoop 目录下,并配置环境变量 (hadoop version 进行测试)
4. 克隆当前版本的linux
vmware有克隆虚拟机的功能,会将所有配置进行克隆
配置每台机器的域名
sudo hostname master (主节点)
sudo hostname slave1 (从节点)
sudo hostname slave2(从节点)
配置每台机器的固定ip地址,并增加域名解析配置: vim /etc/hosts 文件增加如下配置:
127.0.0.1 localhost
192.168.61.100 master
192.168.61.101 slave1
192.168.61.102 slave2
这里可以先配置一台,然后通过scp命令将配置复制到其他两台机器上去,后面的hdfs、yarn、MapReduce的配置同样如此。
5. 配置HDFS
到hadoop安装目录下配置: ./etc/hadoop/core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/home/test/hadoop-2.8.5/hdfs/tmp</value>
<description>A base for other temporary directories.</description>
</property> <property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
</configuration>
配置hdfs: vim ./etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/opt/hadoop-2.8.5/hdfs/name</value>
<final>true</final>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/opt/hadoop-2.8.5/hdfs/data</value>
<final>true</final>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9001</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>
6. 配置yarn
<configuration> <!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.address</name>
<value>master:18040</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:18030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:18088</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:18025</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:18141</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>1024</value>
</property> <property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property> <property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>3.0</value>
</property> <property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>1</value>
</property> <property>
<name>yarn.nodemanager.localizer.address</name>
<value>0.0.0.0:8040</value>
</property>
<property>
<description>The address of the container manager in the NM.</description>
<name>yarn.nodemanager.address</name>
<value>0.0.0.0:8041</value>
</property>
<property>
<description>NM Webapp address.</description>
<name>yarn.nodemanager.webapp.address</name>
<value>0.0.0.0:8042</value>
</property>
</configuration>
7. 配置mapreduce
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property> <property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>1024</value>
</property> <property>
<name>mapreduce.map.memory.mb</name>
<value>1024</value>
</property> <property>
<name>mapreduce.reduce.memory.mb</name>
<value>1024</value>
</property> </configuration>
8. 测试:
在master节点上运行 ./sbin/start-all.sh
通过jps 可以查看 master上的namenode和slave上的datanode (结果如下)
test@master:/opt/hadoop-2.8.5$ jps
8960 Jps
7940 NameNode
8373 ResourceManager
8206 SecondaryNameNode
slave2上运行结果如下:
test@slave2:/opt/hadoop-2.8.5/logs$ jps
7301 Jps
6938 NodeManager
6767 DataNode
三、wordcount程序
在运行完start-all.sh脚本后。 就可以运行hadoop自带的wordcount程序了。
1. 上传文件到hdfs的wc_input中
2. 执行实例程序
./bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar wordcount /wc_input /wc_output.out7
3. 执行结果如下:
18/10/21 16:13:18 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.61.100:18040
18/10/21 16:13:20 INFO input.FileInputFormat: Total input files to process : 2
18/10/21 16:13:20 INFO mapreduce.JobSubmitter: number of splits:2
18/10/21 16:13:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1540109557238_0001
18/10/21 16:13:21 INFO impl.YarnClientImpl: Submitted application application_1540109557238_0001
18/10/21 16:13:21 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1540109557238_0001/
18/10/21 16:13:21 INFO mapreduce.Job: Running job: job_1540109557238_0001
18/10/21 16:13:35 INFO mapreduce.Job: Job job_1540109557238_0001 running in uber mode : false
18/10/21 16:13:35 INFO mapreduce.Job: map 0% reduce 0%
18/10/21 16:13:42 INFO mapreduce.Job: map 50% reduce 0%
18/10/21 16:13:46 INFO mapreduce.Job: map 100% reduce 0%
18/10/21 16:13:51 INFO mapreduce.Job: map 100% reduce 100%
18/10/21 16:13:52 INFO mapreduce.Job: Job job_1540109557238_0001 completed successfully
18/10/21 16:13:52 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=93
FILE: Number of bytes written=473483
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=242
HDFS: Number of bytes written=39
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=7691
Total time spent by all reduces in occupied slots (ms)=3635
Total time spent by all map tasks (ms)=7691
Total time spent by all reduce tasks (ms)=3635
Total vcore-milliseconds taken by all map tasks=7691
Total vcore-milliseconds taken by all reduce tasks=3635
Total megabyte-milliseconds taken by all map tasks=7875584
Total megabyte-milliseconds taken by all reduce tasks=3722240
Map-Reduce Framework
Map input records=3
Map output records=8
Map output bytes=71
Map output materialized bytes=99
Input split bytes=203
Combine input records=8
Combine output records=8
Reduce input groups=6
Reduce shuffle bytes=99
Reduce input records=8
Reduce output records=6
Spilled Records=16
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=178
CPU time spent (ms)=2180
Physical memory (bytes) snapshot=721473536
Virtual memory (bytes) snapshot=5936779264
Total committed heap usage (bytes)=474480640
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=39
File Output Format Counters
Bytes Written=39
注: 配置、安装、执行过程中不可避免遇到问题,需要学会看log解决问题。
参考: https://blog.csdn.net/xiao_bai_9527/article/details/79167562
https://blog.csdn.net/qinzhaokun/article/details/47804923
Hadoop环境搭建及wordcount程序的更多相关文章
- hadoop环境搭建及Wordcount案例实验
1.Linux环境变量设置 在/etc/profile中添加环境变量 sudo vim /etc/profile PATH=$PATH:/usr/local/hadoop/bin source /et ...
- 转 史上最详细的Hadoop环境搭建
GitChat 作者:鸣宇淳 原文:史上最详细的Hadoop环境搭建 关注公众号:GitChat 技术杂谈,一本正经的讲技术 [不要错过文末活动哦] 前言 Hadoop在大数据技术体系中的地位至关重要 ...
- 【转】RHadoop实践系列之一:Hadoop环境搭建
RHadoop实践系列之一:Hadoop环境搭建 RHadoop实践系列文章,包含了R语言与Hadoop结合进行海量数据分析.Hadoop主要用来存储海量数据,R语言完成MapReduce 算法,用来 ...
- 大数据学习之Hadoop环境搭建
一.Hadoop的优势 1)高可靠性:因为Hadoop假设计算元素和存储会出现故障,因为它维护多个工作数据副本,在出现故障时可以对失败的节点重新分布处理. 2)高扩展性:在集群间分配任务数据,可方便的 ...
- hadoop环境搭建之关于NAT模式静态IP的设置 ---VMware12+CentOs7
很久没有更新了,主要是没有时间,今天挤出时间验证了一下,果然还是有些问题的,不过已经解决了,就发上来吧. PS:小豆腐看仔细了哦~ 关于hadoop环境搭建,从单机模式,到伪分布式,再到完全分布式,我 ...
- eclipse工具下hadoop环境搭建
eclipse工具下hadoop环境搭建: window10操作系统中搭建eclipse64开发系统,配置hadoop的eclipse插件,让eclipse可以查看Hdfs中的文件内容. ...
- Hadoop环境搭建、启动和管理界面查看
一.hadoop环境搭建: 1. hadoop 6个核心配置文件的作用:core-site.xml:核心配置文件,主要定义了我们文件访问的格式 hdfs://hadoop-env.sh:主要配置我们的 ...
- Ubuntu中Hadoop环境搭建
Ubuntu中Hadoop环境搭建 JDK安装 方法一:通过命令行直接安装(不建议) 有两种java可以安装oracle-java8-installer以及openjdk (1)安装oracle-ja ...
- Linux集群搭建与Hadoop环境搭建
今天是8月19日,距离开学还有15天,假期作业完成还是遥遥无期,看来开学之前的恶补是躲不过了 今天总结一下在Linux环境下安装Hadoop的过程,首先是对Linux环境的配置,设置主机名称,网络设置 ...
随机推荐
- lnmp源码编译安装zabbix
软件安装 Mysql 安装 tar xf mysql-5.7.13-1.el6.x86_64.rpm-bundle.tar -C mysql rpm -e --nodeps mysql-libs-5 ...
- 学习devexpresschartControl控件
devexpress官网chart:https://documentation.devexpress.com/WindowsForms/8117/Controls-and-Libraries/Char ...
- MATLAB拟合正态分布
clear;clc;close all format compact %% 正态分布的拟合 % 生成随机数 num = 50; y = randn(1000,1); x = 1:num; y = hi ...
- UIActivityIndicatorView 的使用
// // UIActivityIndicator.m // ToolBar // // Created by lanouhn on 15/1/3. // Copyright (c) 2015 ...
- java基础-day18
第07天 集合 今日内容介绍 u HashSet集合 u HashMap集合 第1章 HashSet集合 1.1 Set接口的特点 Set体系的集合: A:存入集合的顺序和取出集合的顺序不一 ...
- python特殊方法定制类
#coding:utf-8class RoundFloat(object): def __init__(self,val): assert isinstance(val, float),"v ...
- Hdu2204 Eddy's爱好 2017-06-27 16:11 43人阅读 评论(0) 收藏
Eddy's爱好 Time Limit : 3000/1000ms (Java/Other) Memory Limit : 32768/32768K (Java/Other) Total Subm ...
- CentOS 6.5 伪分布安装
CentOS 6.5 伪分布安装 软件准备 jdk-6u24-linux-i586.bin .hadoop-1.2.1.tar.gz.hadoop-eclipse-plugin-1.2.1.jar ...
- [php-error-report]PHP Strict Standards: Only variables should be passed by reference
// 报错代码:PHP Strict Standards: Only variables should be passed by reference $arr_userInfo['im_nation_ ...
- Java-动态代理技术
1.程序中的代理 为具有相同接口的目标类的各个方法,添加一些系统功能,如日志,异常处理,计算方法运行的 时间,事务管理等等,都可以交给另一个类去实现这些功能,该类称为代理类. 注意:为了让代理类共享目 ...