bigdata learning unit one--Hadoop environment setting
1、配置ssh,使集群服务器之间的通讯,不再每次都输入密码进行认证。
2、
[root@hc--uatbeta2 hadoop]# start-all.sh
Starting namenodes on [hc--uatbeta2.novalocal]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [hc--uatbeta2.novalocal]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2018-08-07 06:56:20,552 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
[root@hc--uatbeta2 hadoop]#
解决:把所有环节变量都加上 vi hadoop-env.sh
export HDFS_NAMENODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_DATANODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
--------
[root@hc--uatbeta2 hadoop]# vi core-site.xml
[root@hc--uatbeta2 hadoop]# vi mapred-site.xml
[root@hc--uatbeta2 hadoop]# vi yarn-site.xml
vi hdfs-site.xml
----将hostname 改为IP------- 改为IP也不行,得修改hosts:vi /etc/hosts
2、格式化namenode,
#CD /usr/hadoop/hadoop-3.0.0
# ./bin/hdfs namenode -format
这个报错:
出现未知的主机名的问题,异常信息如下所示:
[shirdrn@localhost bin]$ hadoop namenode -format
11/06/22 07:33:31 INFO namenode.NameNode: STARTUP_MSG:
解决方案还是修改hosts;
10.167.202.135:50070
10.167.202.135:8088
10.167.202.135:50070 不能访问的话
vi /etc/selinux/config
修改 # This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing 为 # This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
设置默认访问端口 mapred-site.xml 添加下面两个
<property>
<name>mapred.job.tracker.http.address</name>
<value>0.0.0.0:50030</value>
</property>
<property>
<name>mapred.task.tracker.http.address</name>
<value>0.0.0.0:50060</value>
</property> hdfs-site.xml 添加下面配置
<property>
<name>dfs.http.address</name>
<value>0.0.0.0:50070</value>
</property>
然后停止所有进程,删除name、data文件夹下数据重新格式化,重新启动后访问正常
测试hdfs 是否可以使用:
1、上传文件:hadoop fs -put /root/服务器地址.java hdfs://localhost:9000/ localhost:9000 配置在hadoop-env.sh
2、查看文件:hadoop fs -ls /
3、删除文件:hadoop fs -rm hdfs://10.167.202.135:9000/服务器地址.java
bigdata learning unit one--Hadoop environment setting的更多相关文章
- bigdata learning unit two--Spark environment setting
1.下载 Spark安装之前的准备 文件的解压与改名 tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz rm -rf spark-2.2.0-bin-hadoop2.7. ...
- Java Environment Setting
As a non-Java developer, I am quit stuck in Java environment setting because I am not familiar with ...
- [Keras] Install and environment setting
Documentation: https://keras.io/ 1. 利用anaconda 管理python库是明智的选择. conda update conda conda update anac ...
- How to change Visual Studio default environment setting
如何改变 Visual Studio 的默认环境设置: 1. 工具栏 Tools --> Import and Export Settings... 2. 选择 Reset All Settin ...
- Deep Learning - Install the Development Environment
WLS(Windows Subsystem for Linux) Base WLS Installation Guide Initializing a newly installed distro W ...
- Mac environment setting
java 7 jdk http://www.ifunmac.com/2013/04/mac-jdk-7/ http://blog.sina.com.cn/s/blog_6dce99b101016744 ...
- Ubuntu15.10下Hadoop2.6.0伪分布式环境安装配置及Hadoop Streaming的体验
Ubuntu用的是Ubuntu15.10Beta2版本,正式的版本好像要到这个月的22号才发布.参考的资料主要是http://www.powerxing.com/install-hadoop-clus ...
- Machine Learning for Developers
Machine Learning for Developers Most developers these days have heard of machine learning, but when ...
- [C2P2] Andrew Ng - Machine Learning
##Linear Regression with One Variable Linear regression predicts a real-valued output based on an in ...
随机推荐
- delphi中如何实现DBGrid中的两列数据想减并存入另一列
可参考下面的例子: 数据自动计算的实现:“金额”是由“单价”和“工程量”相乘直接得来的,勿需人工输入. 这可在“数据源构件”的onupdatedata例程添加如下代码实现: procedure T ...
- Thread的其他属性方法
from threading import Thread,currentThread,active_count import time def task(): print('%s is running ...
- 【图像处理】openCV库教程
openCV 基础学习 with:于士琪openCV基础 env:opencv3.4.0+vc2017集成开发环境 图像的表示:矩阵 1. 灰度矩阵 <br> 2. 彩色(多通道)如RGB ...
- python史上最全学习路线图
ps:盘它 python入门教程 关注微信公众号,回复"python入门"获取视频下载地址
- python数学库math模块
函数 数学表示 含义 .pi 圆周率π π的近似值,15位小数 .e 自然常数 e e的近似值,15位小数 ceil(x) [x] 对浮点数向上取整 floor(x) [x] 对浮点数向下取整 pow ...
- poj 2226 Muddy Fields(水二分图)
Rain has pummeled the cows' field, a rectangular grid of R rows and C columns (1 <= R <= 50, 1 ...
- 【XSY2753】Lcm 分治 FWT FFT 容斥
题目描述 给你\(n,k\),要你选一些互不相同的正整数,满足这些数的\(lcm\)为\(n\),且这些数的和为\(k\)的倍数. 求选择的方案数.对\(232792561\)取模. \(n\leq ...
- 【XSY2779】最小表示串 KMP DP polya定理
题目描述 给你一个字符串\(s\),问你有多少个串是最小表示串且字典序\(\leq s\) \(|s|\leq 1000\) 题解 先把\(s\)变成比\(s\)小的最大的最小表示串.方法是从后枚举每 ...
- 【XSY1528】azelso 概率&期望DP
题目大意 有一条很长很长的路(出题人的套路),你在\(0\)位置,你要去\(h\)位置. 路上有一些不同的位置上有敌人,你要和他战斗,你有\(p\)的概率赢.若你赢,则你可以走过去,否则你会死.还 ...
- 洛谷P3953 逛公园(NOIP2017)(最短/长路,拓扑排序,动态规划)
洛谷题目传送门 又是一年联赛季.NOIP2017至此收官了. 这个其实是比较套路的图论DP了,但是细节有点恶心. 先求出\(1\)到所有点的最短路\(d1\),和所有点到\(n\)的最短路\(dn\) ...