bigdata learning unit one--Hadoop environment setting
1、配置ssh,使集群服务器之间的通讯,不再每次都输入密码进行认证。
2、
[root@hc--uatbeta2 hadoop]# start-all.sh
Starting namenodes on [hc--uatbeta2.novalocal]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [hc--uatbeta2.novalocal]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2018-08-07 06:56:20,552 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
[root@hc--uatbeta2 hadoop]#
解决:把所有环节变量都加上 vi hadoop-env.sh
export HDFS_NAMENODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_DATANODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
--------
[root@hc--uatbeta2 hadoop]# vi core-site.xml
[root@hc--uatbeta2 hadoop]# vi mapred-site.xml
[root@hc--uatbeta2 hadoop]# vi yarn-site.xml
vi hdfs-site.xml
----将hostname 改为IP------- 改为IP也不行,得修改hosts:vi /etc/hosts
2、格式化namenode,
#CD /usr/hadoop/hadoop-3.0.0
# ./bin/hdfs namenode -format
这个报错:
出现未知的主机名的问题,异常信息如下所示:
[shirdrn@localhost bin]$ hadoop namenode -format
11/06/22 07:33:31 INFO namenode.NameNode: STARTUP_MSG:
解决方案还是修改hosts;
10.167.202.135:50070
10.167.202.135:8088
10.167.202.135:50070 不能访问的话
vi /etc/selinux/config
修改 # This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing 为 # This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
设置默认访问端口 mapred-site.xml 添加下面两个
<property>
<name>mapred.job.tracker.http.address</name>
<value>0.0.0.0:50030</value>
</property>
<property>
<name>mapred.task.tracker.http.address</name>
<value>0.0.0.0:50060</value>
</property> hdfs-site.xml 添加下面配置
<property>
<name>dfs.http.address</name>
<value>0.0.0.0:50070</value>
</property>
然后停止所有进程,删除name、data文件夹下数据重新格式化,重新启动后访问正常
测试hdfs 是否可以使用:
1、上传文件:hadoop fs -put /root/服务器地址.java hdfs://localhost:9000/ localhost:9000 配置在hadoop-env.sh
2、查看文件:hadoop fs -ls /
3、删除文件:hadoop fs -rm hdfs://10.167.202.135:9000/服务器地址.java
bigdata learning unit one--Hadoop environment setting的更多相关文章
- bigdata learning unit two--Spark environment setting
1.下载 Spark安装之前的准备 文件的解压与改名 tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz rm -rf spark-2.2.0-bin-hadoop2.7. ...
- Java Environment Setting
As a non-Java developer, I am quit stuck in Java environment setting because I am not familiar with ...
- [Keras] Install and environment setting
Documentation: https://keras.io/ 1. 利用anaconda 管理python库是明智的选择. conda update conda conda update anac ...
- How to change Visual Studio default environment setting
如何改变 Visual Studio 的默认环境设置: 1. 工具栏 Tools --> Import and Export Settings... 2. 选择 Reset All Settin ...
- Deep Learning - Install the Development Environment
WLS(Windows Subsystem for Linux) Base WLS Installation Guide Initializing a newly installed distro W ...
- Mac environment setting
java 7 jdk http://www.ifunmac.com/2013/04/mac-jdk-7/ http://blog.sina.com.cn/s/blog_6dce99b101016744 ...
- Ubuntu15.10下Hadoop2.6.0伪分布式环境安装配置及Hadoop Streaming的体验
Ubuntu用的是Ubuntu15.10Beta2版本,正式的版本好像要到这个月的22号才发布.参考的资料主要是http://www.powerxing.com/install-hadoop-clus ...
- Machine Learning for Developers
Machine Learning for Developers Most developers these days have heard of machine learning, but when ...
- [C2P2] Andrew Ng - Machine Learning
##Linear Regression with One Variable Linear regression predicts a real-valued output based on an in ...
随机推荐
- 初识Xml。
/* * 一.Xml? * * 1.是什么? * Extensible markup Language 可拓展标记性语言 * 功能是 储存数据 * 1.配置文件 * 2.在网络中传输数据 * xml和 ...
- Vue之v-for、v-show使用举例
demo.html <!DOCTYPE html> <html lang="en" xmlns:v-bind="http://www.w3.org/19 ...
- JSTL 之 <c:out>
jstl的<c:out value="${hello}"></c:out> EL表达式的${hello },两者一般没什么不同,但是EL表达式输出的时候回尝 ...
- MVP, MVC, MVVM, 傻傻分不清楚~
1 简介 英文原文:MVC vs. MVP vs. MVVM 三者的目的都是分离关注,使得UI更容易变换(从Winform变为Webform),使得UI更容易进行单元测试. 2 MVC/MVP 2.1 ...
- Wiener Filter
假设分别有两个WSS process:$x[n]$,$y[n]$,这两个process之间存在某种关系,并且我们也了解这种关系.现在我们手头上有process $x[n]$,目的是要设计一个LTI系统 ...
- django---一对多和多对多字段的操作训练
建表准备: django项目models.py建表 from django.db import models class Myclass(models.Model): cname = models.C ...
- html概念
一.前端 1.什么是前端 前端即网站前台部分,运行在PC端等浏览器上展现给用户浏览的网页.随着互联网技术的发展, HTML5,CSS3,前端框架的应用,跨平台响应式网页设计能够适应各种屏幕分辨率,完美 ...
- 洛谷P1373小a和uim大逃离题解
题目 这个题好坑啊,首先是他会卡空间,然后我们就只能把一种比较好理解的状态给舍弃,因为空间开不下,然而采用一种难理解的状态就是\(dp[i][j][l][0/1]\)表示\(i\),\(j\)位置,两 ...
- Spring02-注入和注解方式操作
一. 依赖注入 测试类:Person.java 创建配置文件:applicationContext-injection.xml 创建测试代码:InjectionTest.java 1. set方法注入 ...
- Python里的单下划线,双下划线,以及前后都带下划线的意义
Python里的单下划线,双下划线,以及前后都带下划线的意义: 单下划线如:_name 意思是:不能通过from modules import * 导入,如需导入需要:from modules imp ...