CentOS7.2配置Hadoop2.6.5
Hadoop配置文件
/etc/profile
配置Java和Hadoop环境
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export HADOOP_HOME=/usr/local/hadoop-2.6.5/hadoop-2.6.5
export PATH=$JAVA_HOME/bin:${PATH}:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
配置文件在/etc/hadoop
core-site.xml
fs.defualtFS:配置主节点namenode地址
hadoop.tmp.dir:在本地文件系统所在的NameNode的存储空间和持续化处理日志
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.94.140:9000</value>
</property> <property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-2.6.5/data</value>
</property>
</configuration>
mapred-site.xml.template
mapreduce.framework.name:执行框架设置为 Hadoop YARN
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
yarn-site.xml
yarn.resourcemanager.hostname:namenode的yarn地址
yarn.nodemanager.aux-services:Shuffle service 需要加以设置的Map Reduce的应用程序服务
<property>
<name>yarn.resourcemanager.hostname</name>
<value>CentOS7One</value>
</property> <property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
hadoop-env.sh
JAVA_HOME换成绝对路径
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
slaves
配置子节点,伪分布式,一台机器,配置子节点为自身
CentOS7One
Hadoop免密钥配置
方法1:
1.生成私钥和公钥
[root@CentOS7One ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
eb:::dc:9a::ee:f9:5d:3b:db:6f::::f9 root@CentOS7One
The key's randomart image is:
+--[ RSA ]----+
| .|
| ..|
| . . o.|
| + o o E|
| B S o |
| o o . .|
| . o. . . |
| +o . ..o o|
| ...o . oo.+.|
+-----------------+
2.将CentOS7One的id_rsa_pub拷贝到CentOSTwo中
[root@CentOS7One .ssh]#
[root@CentOS7One .ssh]# scp id_rsa.pub CentOS7Two:/root
The authenticity of host 'centos7two (192.168.94.139)' can't be established.
ECDSA key fingerprint is dd:e2::9d:e2:6e::c3:2a:::3f:f6:3a:f2:.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'centos7two,192.168.94.139' (ECDSA) to the list of known hosts.
root@centos7two's password:
id_rsa.pub % .4KB/s :
[root@CentOS7One .ssh]#
3.重复步骤1,在CentOS7Two主机上生成私钥和公钥
[root@CentOS7Two ~]# ssh-keygen -t rsa
4.在CentOS7Two中新建authorized_keys,权限为600,向其追加从CentOS7One拷贝的id_rsa.pub
[root@CentOS7Two .ssh]# touch authorized_keys
[root@CentOS7Two .ssh]# chmod authorized_keys
[root@CentOS7Two .ssh]# cat /root/id_rsa.pub >> authorized_keys
authorized_keys是被允许免密连接到CentOS7Two的主机的公钥
[root@CentOS7Two .ssh]# cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGBMCc2AOqbMUT9uB8tDyrcNkjV/nCum/Ez5OMRZlVEITERtMLN8NUZP0kQjqiRU7kr5oIlA8FoE6sgSF7ciTu1IIDb2pg3roe0PpcaIwdSjP/b6ktnnkkcIXl74194AY/I+A9UFGErdaarTzAlMEougMC6G9IYTefqyMywsUZ5lH3PP72vqQNwZZX/LZtq6AK+yZ4C2jiErfF3i7hL1bTxVDBvGmLg37U8xNhei0Z5SDq9tCGP9EFabVuaw+mehPxGwFTbyuQj6X1xDmRD8lfjRWTK7M88dVImKdrf85KJAL5kyquIQi0tSAskkSlaroIDzNh1ebacKlOuWh6eWhd root@CentOS7One
方法2:
先在本机上生成私钥和公钥
[root@CentOS7Three ~]# ssh-keygen -t rsa
然后,运行ssh-copy-id命令,快捷的配置免密登录
[root@CentOS7Three ~]# ssh-copy-id CentOS7Seven
Hadoop运行
Hadoop格式化
对Hadoop格式化,对data目录下写初始化文件
hadoop namenode -format
启动dfs
启动目录在sbin/
start-dfs.sh
Jps检验是否启动成功
[root@CentOS7One current]# jps
NameNode
DataNode
SecondaryNameNode
Jps
启动yarn
start-yarn.sh
Jps检验是否成功
[root@CentOS7One current]# jps
NameNode
DataNode
NodeManager
SecondaryNameNode
ResourceManager
Jps
Hadoop命令
上传
hadoop fs -put hadoop-2.6..tar.gz hdfs://CentOS7One:9000/
hadoop fs -copyFromLocal /usr/local/hadoop-2.6.5/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar /
下载
hadoop fs -get hdfs://CentOS7One:9000/hadoop-2.6.5.tar.gz
创建文件
路径是hdfs://CenOS7One:9000/wordcount/input的简写
hadoop fs -mkdir /wordcount/input
查看文件列表
[root@CentOS7One mapreduce]# hadoop fs -ls /wordcount/output
Found items
-rw-r--r-- root supergroup -- : /wordcount/output/_SUCCESS
-rw-r--r-- root supergroup -- : /wordcount/output/part-r-
查看文件详情
[root@CentOS7One mapreduce]# hadoop fs -cat /wordcount/output/part-r-
fuckbaby
hello
jim
kitty
tom
world
删除文件
[root@CentOS7One ~]# hadoop fs -rm -r /qingshu.txt
// :: INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = minutes, Emptier interval = minutes.
Deleted /qingshu.txt
查看文件系统空间使用情况
[root@CentOS7One data]# hadoop fs -df -h /
Filesystem Size Used Available Use%
hdfs://192.168.94.140:9000 17.5 G 210.6 M 14.1 G 1%
查看文件夹、文件具体占用多少空间
[root@CentOS7One data]# hadoop fs -du -s -h hdfs://CentOS7One:9000/*
190.4 M hdfs://CentOS7One:9000/hadoop-2.6.5.tar.gz
285.8 K hdfs://CentOS7One:9000/hadoop-mapreduce-examples-2.6.5.jar
18.1 M hdfs://CentOS7One:9000/mapreduce
hdfs://CentOS7One:9000/user
hdfs://CentOS7One:9000/wordcount
CentOS7.2配置Hadoop2.6.5的更多相关文章
- Centos7.0配置Hadoop2.7.0伪分布式
一.ssh免密登录 1.命令ssh-keygen. overwrite输入y一路回车 2.将生成的密钥发送到本机 ssh-copy-id localhost中间会询问是否继续输入“yes” 3.测试免 ...
- building hadoop2.4.1 on centos7[在centos7上面构建hadoop2.4.1]
本文介绍在centos7上面通过hadoop2.4.1源码构建hadoop distribution 版本,即hadoop的运行版本. 为何要自己building,而不用Apache的distribu ...
- Centos7网络配置,vsftpd安装及530报错解决
今天在虚拟机安装CentOS7,准备全新安装LTMP,结果又是一堆问题,不过正好因为这些出错,又给自己长了见识. 1,CentOS7网络配置 最小化安装CentOs7后,ifconfig提示comma ...
- Centos7安装配置gitlab
Centos7安装配置gitlab 这篇文字我会介绍在Centos7上安装gitlab,配置gitlab的smtp,并且创建项目demo. sudo yum install openssh-serve ...
- VMware中安装CentOS7网络配置静态IP地址,常用配置和工具安装
VMware中安装CentOS7网络配置静态IP地址,常用配置和工具安装在阿里云开源镜像地址下载镜像Index of /centos/7.2.1511/isos/x86_64/http://mirro ...
- centos7初步配置
centos7初步配置 首先安装lrzsz zip/unzip yum -y install lrzsz yum -y install zip unzip 安装vim yum install vim* ...
- centos7网络配置总结
centos7网络配置 --wang 一.通过配置文件 配置/etc/sysconfig/network-scripts/en.. 记忆信息量大,易出错,不推荐使用.配置多台电脑静态ip可以通过复制模 ...
- CentOS7基本配置一
CentOS7基本配置一 安装VMwareTools 1.点击重新安装VM-tool, 继而找到压缩文件VMwareTools-10.2.0...tar.gz,复制到桌面下,解压这么压缩文件到桌面下 ...
- Centos7安装配置Apache+PHP+Mysql+phpmyadmin
转载自: Centos7安装配置Apache+PHP+Mysql+phpmyadmin 一.安装Apache yum install httpd 安装成功后,Apache操作命令: systemctl ...
随机推荐
- Android app日志保存功能
每一个App应用应该都需要有日志保存的功能,日志保存可以记录App运行中所遇到的问题,查Bug也比较方便 等等: Android日志保存功能,保存某几天的最新日志文件到某个目录,直接看是如何代码实现的 ...
- LINQ to XML基本操作
Linq to XML同样是对原C#访问XML文件的方法的封装,简化了用xpath进行xml的查询以及增加,修改,删除xml元素的操作. LINQ to XML 三个最重要类:XElement.XAt ...
- DevOps Workshop 研发运维一体化(北京第二场) 2016.04.27
北京不亏为首都,人才济济,对微软DevOps解决方案感兴趣的人太多.我们与微软公司临时决定再家一场培训. 我之前在博客中(DevOps Workshop 研发运维一体化第一场(微软亚太研发集团总部)h ...
- linux系统编程:setjmp和longjmp函数用法
#include <stdio.h> #include <setjmp.h> //jmp_buf:数组,保存栈信息即运行环境 jmp_buf buf; double Divid ...
- [翻译]NUnit---Range and Repeat Attributes(十五)
RangeAttribute (NUnit 2.5) Range特性用于为参数话测试方法的参数的值范围指定一个值,与Random特性一样,NUnit会将每个参数的值组合为一些了测试用例,所以如果为一个 ...
- 排序算法之堆排序(Heapsort)解析
一.堆排序的优缺点(pros and cons) (还是简单的说说这个,毕竟没有必要浪费时间去理解一个糟糕的的算法) 优点: 堆排序的效率与快排.归并相同,都达到了基于比较的排序算法效率的峰值(时间复 ...
- Winform打包安装程序覆盖安装的实现
1.修改项目程序集版本号. 2.设置Version,使当前版本号大于前一个版本号. 3.RemovePreviousVersions属性设置为true. 以上三步后,生成安装程序即可实现覆盖安装. P ...
- error while loading shared libraries: libudev.so.0 的问题
在linux下运行koala的时候遇到了问题: 应该是缺少文件libudev.so.0 在/lib和/lib64目录下也都没有这个文件.然后网上找解决方案,在/lib/x86_64-linux-gnu ...
- Datetimepicker.js用法
$('.form_date').datetimepicker({//初始化 language: 'zh-CN', //weekStart: 1, //todayBtn: 1, autoclose: 1 ...
- 极大提高Web开发效率的8个工具和建议(含教程)
面对复杂的 Web 应用的开发,良好的流程和工具支持是必不可少的,它们可以让日常的开发工作更加顺畅.更加高效.本文介绍了6个Web开发利器以及相关的教程,帮助你在开发.调试.集成和发布过程极大地提高效 ...