1、配置ssh,使集群服务器之间的通讯,不再每次都输入密码进行认证。

2、

[root@hc--uatbeta2 hadoop]# start-all.sh
Starting namenodes on [hc--uatbeta2.novalocal]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [hc--uatbeta2.novalocal]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2018-08-07 06:56:20,552 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
[root@hc--uatbeta2 hadoop]#

解决:把所有环节变量都加上   vi hadoop-env.sh

export HDFS_NAMENODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_DATANODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

--------

[root@hc--uatbeta2 hadoop]# vi core-site.xml
[root@hc--uatbeta2 hadoop]# vi mapred-site.xml
[root@hc--uatbeta2 hadoop]# vi yarn-site.xml

vi hdfs-site.xml

----将hostname 改为IP-------  改为IP也不行,得修改hosts:vi /etc/hosts

2、格式化namenode,

#CD /usr/hadoop/hadoop-3.0.0
# ./bin/hdfs namenode -format

这个报错:

出现未知的主机名的问题,异常信息如下所示:

[shirdrn@localhost bin]$ hadoop namenode -format
11/06/22 07:33:31 INFO namenode.NameNode: STARTUP_MSG:

解决方案还是修改hosts;

10.167.202.135:50070

10.167.202.135:8088

10.167.202.135:50070  不能访问的话

vi  /etc/selinux/config

修改

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing 为 # This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled

  

设置默认访问端口

mapred-site.xml 添加下面两个
<property>
<name>mapred.job.tracker.http.address</name>
<value>0.0.0.0:50030</value>
</property>
<property>
<name>mapred.task.tracker.http.address</name>
<value>0.0.0.0:50060</value>
</property> hdfs-site.xml 添加下面配置
<property>
<name>dfs.http.address</name>
<value>0.0.0.0:50070</value>
</property>
然后停止所有进程,删除name、data文件夹下数据重新格式化,重新启动后访问正常

  

测试hdfs 是否可以使用:

1、上传文件:hadoop fs -put /root/服务器地址.java hdfs://localhost:9000/    localhost:9000 配置在hadoop-env.sh

2、查看文件:hadoop fs -ls /

3、删除文件:hadoop  fs  -rm hdfs://10.167.202.135:9000/服务器地址.java

bigdata learning unit one--Hadoop environment setting的更多相关文章

  1. bigdata learning unit two--Spark environment setting

    1.下载 Spark安装之前的准备 文件的解压与改名 tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz rm -rf spark-2.2.0-bin-hadoop2.7. ...

  2. Java Environment Setting

    As a non-Java developer, I am quit stuck in Java environment setting because I am not familiar with ...

  3. [Keras] Install and environment setting

    Documentation: https://keras.io/ 1. 利用anaconda 管理python库是明智的选择. conda update conda conda update anac ...

  4. How to change Visual Studio default environment setting

    如何改变 Visual Studio 的默认环境设置: 1. 工具栏 Tools --> Import and Export Settings... 2. 选择 Reset All Settin ...

  5. Deep Learning - Install the Development Environment

    WLS(Windows Subsystem for Linux) Base WLS Installation Guide Initializing a newly installed distro W ...

  6. Mac environment setting

    java 7 jdk http://www.ifunmac.com/2013/04/mac-jdk-7/ http://blog.sina.com.cn/s/blog_6dce99b101016744 ...

  7. Ubuntu15.10下Hadoop2.6.0伪分布式环境安装配置及Hadoop Streaming的体验

    Ubuntu用的是Ubuntu15.10Beta2版本,正式的版本好像要到这个月的22号才发布.参考的资料主要是http://www.powerxing.com/install-hadoop-clus ...

  8. Machine Learning for Developers

    Machine Learning for Developers Most developers these days have heard of machine learning, but when ...

  9. [C2P2] Andrew Ng - Machine Learning

    ##Linear Regression with One Variable Linear regression predicts a real-valued output based on an in ...

随机推荐

  1. 初识Xml。

    /* * 一.Xml? * * 1.是什么? * Extensible markup Language 可拓展标记性语言 * 功能是 储存数据 * 1.配置文件 * 2.在网络中传输数据 * xml和 ...

  2. Vue之v-for、v-show使用举例

    demo.html <!DOCTYPE html> <html lang="en" xmlns:v-bind="http://www.w3.org/19 ...

  3. JSTL 之 <c:out>

    jstl的<c:out value="${hello}"></c:out> EL表达式的${hello },两者一般没什么不同,但是EL表达式输出的时候回尝 ...

  4. MVP, MVC, MVVM, 傻傻分不清楚~

    1 简介 英文原文:MVC vs. MVP vs. MVVM 三者的目的都是分离关注,使得UI更容易变换(从Winform变为Webform),使得UI更容易进行单元测试. 2 MVC/MVP 2.1 ...

  5. Wiener Filter

    假设分别有两个WSS process:$x[n]$,$y[n]$,这两个process之间存在某种关系,并且我们也了解这种关系.现在我们手头上有process $x[n]$,目的是要设计一个LTI系统 ...

  6. django---一对多和多对多字段的操作训练

    建表准备: django项目models.py建表 from django.db import models class Myclass(models.Model): cname = models.C ...

  7. html概念

    一.前端 1.什么是前端 前端即网站前台部分,运行在PC端等浏览器上展现给用户浏览的网页.随着互联网技术的发展, HTML5,CSS3,前端框架的应用,跨平台响应式网页设计能够适应各种屏幕分辨率,完美 ...

  8. 洛谷P1373小a和uim大逃离题解

    题目 这个题好坑啊,首先是他会卡空间,然后我们就只能把一种比较好理解的状态给舍弃,因为空间开不下,然而采用一种难理解的状态就是\(dp[i][j][l][0/1]\)表示\(i\),\(j\)位置,两 ...

  9. Spring02-注入和注解方式操作

    一. 依赖注入 测试类:Person.java 创建配置文件:applicationContext-injection.xml 创建测试代码:InjectionTest.java 1. set方法注入 ...

  10. Python里的单下划线,双下划线,以及前后都带下划线的意义

    Python里的单下划线,双下划线,以及前后都带下划线的意义: 单下划线如:_name 意思是:不能通过from modules import * 导入,如需导入需要:from modules imp ...