1、配置ssh,使集群服务器之间的通讯,不再每次都输入密码进行认证。

2、

[root@hc--uatbeta2 hadoop]# start-all.sh
Starting namenodes on [hc--uatbeta2.novalocal]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [hc--uatbeta2.novalocal]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2018-08-07 06:56:20,552 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
[root@hc--uatbeta2 hadoop]#

解决:把所有环节变量都加上   vi hadoop-env.sh

export HDFS_NAMENODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_DATANODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

--------

[root@hc--uatbeta2 hadoop]# vi core-site.xml
[root@hc--uatbeta2 hadoop]# vi mapred-site.xml
[root@hc--uatbeta2 hadoop]# vi yarn-site.xml

vi hdfs-site.xml

----将hostname 改为IP-------  改为IP也不行,得修改hosts:vi /etc/hosts

2、格式化namenode,

#CD /usr/hadoop/hadoop-3.0.0
# ./bin/hdfs namenode -format

这个报错:

出现未知的主机名的问题,异常信息如下所示:

[shirdrn@localhost bin]$ hadoop namenode -format
11/06/22 07:33:31 INFO namenode.NameNode: STARTUP_MSG:

解决方案还是修改hosts;

10.167.202.135:50070

10.167.202.135:8088

10.167.202.135:50070  不能访问的话

vi  /etc/selinux/config

修改

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing 为 # This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled

  

设置默认访问端口

mapred-site.xml 添加下面两个
<property>
<name>mapred.job.tracker.http.address</name>
<value>0.0.0.0:50030</value>
</property>
<property>
<name>mapred.task.tracker.http.address</name>
<value>0.0.0.0:50060</value>
</property> hdfs-site.xml 添加下面配置
<property>
<name>dfs.http.address</name>
<value>0.0.0.0:50070</value>
</property>
然后停止所有进程,删除name、data文件夹下数据重新格式化,重新启动后访问正常

  

测试hdfs 是否可以使用:

1、上传文件:hadoop fs -put /root/服务器地址.java hdfs://localhost:9000/    localhost:9000 配置在hadoop-env.sh

2、查看文件:hadoop fs -ls /

3、删除文件:hadoop  fs  -rm hdfs://10.167.202.135:9000/服务器地址.java

bigdata learning unit one--Hadoop environment setting的更多相关文章

  1. bigdata learning unit two--Spark environment setting

    1.下载 Spark安装之前的准备 文件的解压与改名 tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz rm -rf spark-2.2.0-bin-hadoop2.7. ...

  2. Java Environment Setting

    As a non-Java developer, I am quit stuck in Java environment setting because I am not familiar with ...

  3. [Keras] Install and environment setting

    Documentation: https://keras.io/ 1. 利用anaconda 管理python库是明智的选择. conda update conda conda update anac ...

  4. How to change Visual Studio default environment setting

    如何改变 Visual Studio 的默认环境设置: 1. 工具栏 Tools --> Import and Export Settings... 2. 选择 Reset All Settin ...

  5. Deep Learning - Install the Development Environment

    WLS(Windows Subsystem for Linux) Base WLS Installation Guide Initializing a newly installed distro W ...

  6. Mac environment setting

    java 7 jdk http://www.ifunmac.com/2013/04/mac-jdk-7/ http://blog.sina.com.cn/s/blog_6dce99b101016744 ...

  7. Ubuntu15.10下Hadoop2.6.0伪分布式环境安装配置及Hadoop Streaming的体验

    Ubuntu用的是Ubuntu15.10Beta2版本,正式的版本好像要到这个月的22号才发布.参考的资料主要是http://www.powerxing.com/install-hadoop-clus ...

  8. Machine Learning for Developers

    Machine Learning for Developers Most developers these days have heard of machine learning, but when ...

  9. [C2P2] Andrew Ng - Machine Learning

    ##Linear Regression with One Variable Linear regression predicts a real-valued output based on an in ...

随机推荐

  1. delphi中如何实现DBGrid中的两列数据想减并存入另一列

    可参考下面的例子:   数据自动计算的实现:“金额”是由“单价”和“工程量”相乘直接得来的,勿需人工输入. 这可在“数据源构件”的onupdatedata例程添加如下代码实现: procedure T ...

  2. Thread的其他属性方法

    from threading import Thread,currentThread,active_count import time def task(): print('%s is running ...

  3. 【图像处理】openCV库教程

    openCV 基础学习 with:于士琪openCV基础 env:opencv3.4.0+vc2017集成开发环境 图像的表示:矩阵 1. 灰度矩阵 <br> 2. 彩色(多通道)如RGB ...

  4. python史上最全学习路线图

    ps:盘它 python入门教程 关注微信公众号,回复"python入门"获取视频下载地址

  5. python数学库math模块

    函数 数学表示 含义 .pi 圆周率π π的近似值,15位小数 .e 自然常数 e e的近似值,15位小数 ceil(x) [x] 对浮点数向上取整 floor(x) [x] 对浮点数向下取整 pow ...

  6. poj 2226 Muddy Fields(水二分图)

    Rain has pummeled the cows' field, a rectangular grid of R rows and C columns (1 <= R <= 50, 1 ...

  7. 【XSY2753】Lcm 分治 FWT FFT 容斥

    题目描述 给你\(n,k\),要你选一些互不相同的正整数,满足这些数的\(lcm\)为\(n\),且这些数的和为\(k\)的倍数. 求选择的方案数.对\(232792561\)取模. \(n\leq ...

  8. 【XSY2779】最小表示串 KMP DP polya定理

    题目描述 给你一个字符串\(s\),问你有多少个串是最小表示串且字典序\(\leq s\) \(|s|\leq 1000\) 题解 先把\(s\)变成比\(s\)小的最大的最小表示串.方法是从后枚举每 ...

  9. 【XSY1528】azelso 概率&期望DP

    题目大意 有一条很长很长的路(出题人的套路),你在\(0\)位置,你要去\(h\)位置. ​ 路上有一些不同的位置上有敌人,你要和他战斗,你有\(p\)的概率赢.若你赢,则你可以走过去,否则你会死.还 ...

  10. 洛谷P3953 逛公园(NOIP2017)(最短/长路,拓扑排序,动态规划)

    洛谷题目传送门 又是一年联赛季.NOIP2017至此收官了. 这个其实是比较套路的图论DP了,但是细节有点恶心. 先求出\(1\)到所有点的最短路\(d1\),和所有点到\(n\)的最短路\(dn\) ...