前置章节:hadoop集群namenode启动ssh免密登录(hadoop完全分布式五)|11

集群启动

配置workers(3.x之前是slaves),

删除localhost,添加102/103/104

[shaozhiqi@hadoop102 hadoop]$ pwd
/opt/module/hadoop-3.1.2/etc/hadoop
[shaozhiqi@hadoop102 hadoop]$ vi workers
[shaozhiqi@hadoop102 hadoop]$ cat workers
hadoop102
hadoop103
hadoop104
[shaozhiqi@hadoop102 hadoop]$

注:workers文件结尾不能有空格,文件中不能有空行

同步workers到其他机器

[shaozhiqi@hadoop102 hadoop]$ testxsync workers
fname=workers
pdir=/opt/module/hadoop-3.1.2/etc/hadoop
-------hadoop103 -----------------------
sending incremental file list
workers
sent 113 bytes received 37 bytes 100.00 bytes/sec
total size is 40 speedup is 0.27
------hadoop104 -----------------------
sending incremental file list
workers
sent 113 bytes received 37 bytes 100.00 bytes/sec
total size is 40 speedup is 0.27
[shaozhiqi@hadoop102 hadoop]$

验证是否同步成功

[shaozhiqi@hadoop103 hadoop]$ cat workers
hadoop102
hadoop103
hadoop104
[shaozhiqi@hadoop103 hadoop]$
[shaozhiqi@hadoop104 hadoop]$ cat workers
hadoop102
hadoop103
hadoop104
[shaozhiqi@hadoop104 hadoop]$

停止上节启动的namenode和datanode

[shaozhiqi@hadoop103 hadoop-3.1.2]$ sbin/hadoop-daemon.sh stop datanode
shaozhiqi@hadoop104 hadoop-3.1.2]$ sbin/hadoop-daemon.sh stop datanode
shaozhiqi@hadoop102 hadoop-3.1.2]$ sbin/hadoop-daemon.sh stop datanode
shaozhiqi@hadoop102 hadoop-3.1.2]$ sbin/hadoop-daemon.sh stop namenode

版本提示用这个命令:hdfs --daemon stop

启动hdfs

sbin/start-dfs.sh

(hdfs包括了namenode和datanode两个节点。具体可查看大数据及hadoop简要概念|6

[shaozhiqi@hadoop102 hadoop-3.1.2]$ sbin/start-dfs.sh
Starting namenodes on [hadoop102]
hadoop102: WARNING: /opt/module/hadoop-3.1.2/logs does not exist. Creating.
Starting datanodes
hadoop103: WARNING: /opt/module/hadoop-3.1.2/logs does not exist. Creating.
hadoop104: WARNING: /opt/module/hadoop-3.1.2/logs does not exist. Creating.
Starting secondary namenodes [hadoop104]
[shaozhiqi@hadoop102 hadoop-3.1.2]$

Jps发现其他都正常,但是namenode就是没有启动。。。

删除data logs,格式化namenode

[shaozhiqi@hadoop102 hadoop-3.1.2]$ bin/hdfs namenode –format

再次启动hdfs

[shaozhiqi@hadoop102 hadoop-3.1.2]$ sbin/start-dfs.sh
Starting namenodes on [hadoop102]
Starting datanodes
hadoop104: WARNING: /opt/module/hadoop-3.1.2/logs does not exist. Creating.
hadoop103: WARNING: /opt/module/hadoop-3.1.2/logs does not exist. Creating.
Starting secondary namenodes [hadoop104]
[shaozhiqi@hadoop102 hadoop-3.1.2]$ jps
7619 DataNode
7877 Jps
7494 NameNode
[shaozhiqi@hadoop102 hadoop-3.1.2]$

验证

102

[shaozhiqi@hadoop102 hadoop-3.1.2]$ jps
7619 DataNode
7877 Jps
7494 NameNode
[shaozhiqi@hadoop102 hadoop-3.1.2]$

103

[shaozhiqi@hadoop103 hadoop-3.1.2]$ jps
2931 Jps
2847 DataNode
[shaozhiqi@hadoop103 hadoop-3.1.2]$

104

[shaozhiqi@hadoop104 hadoop-3.1.2]$ jps
5424 SecondaryNameNode
5467 Jps
5308 DataNode
[shaozhiqi@hadoop104 hadoop-3.1.2]$

参照我们的规划配置,具体查看hadoop集群配置同步(hadoop完全分布式四)|10

配置规划

验证通过
也可以看先我们的hdfs的namenodeweb,可以看到有三台机器ok

namenodeweb

启动yarn

根据我们在[**hadoop****集群配置同步[hadoop集群配置同步(hadoop完全分布式四)|10]
的集群分配原则,我们的yarn的recourceManger在103所以yarn的启动只能在103起,否则会报错

sbin/start-yarn.sh

[shaozhiqi@hadoop103 hadoop-3.1.2]$ sbin/start-yarn.sh
Starting resourcemanager
Starting nodemanagers
[shaozhiqi@hadoop103 hadoop-3.1.2]$ jps
3154 ResourceManager
3271 NodeManager
3611 Jps
2847 DataNode
[shaozhiqi@hadoop103 hadoop-3.1.2]$

103ok

[shaozhiqi@hadoop104 hadoop-3.1.2]$ jps
5424 SecondaryNameNode
5782 Jps
5657 NodeManager
5308 DataNode
[shaozhiqi@hadoop104 hadoop-3.1.2]$

104ok

[shaozhiqi@hadoop102 hadoop-3.1.2]$ jps
7619 DataNode
8052 NodeManager
8164 Jps
7494 NameNode
[shaozhiqi@hadoop102 hadoop-3.1.2]$

102ok

启动成功
下节测试我们的hadoop集群

由于要关闭电脑,所以停止hadoop集群
启动时先起hdfs,然后yarn
先停yarn 再停hdfs

停集群yarn

[shaozhiqi@hadoop103 hadoop-3.1.2]$ sbin/stop-yarn.sh
Stopping nodemanagers
Stopping resourcemanager
[shaozhiqi@hadoop103 hadoop-3.1.2]$ jps
4145 Jps
2847 DataNode
[shaozhiqi@hadoop103 hadoop-3.1.2]$

停集群hdsf

[shaozhiqi@hadoop102 hadoop-3.1.2]$ sbin/stop-dfs.sh
Stopping namenodes on [hadoop102]
Stopping datanodes
Stopping secondary namenodes [hadoop104]
[shaozhiqi@hadoop102 hadoop-3.1.2]$ jps
8941 Jps
[shaozhiqi@hadoop102 hadoop-3.1.2]$

记得下次启动集群前格式namenode

hadoop(九)启动|关闭集群(完全分布式六)|11的更多相关文章

  1. CentOS系统下Hadoop 2.4.1集群安装配置(简易版)

    安装配置 1.软件下载 JDK下载:jdk-7u65-linux-i586.tar.gz http://www.oracle.com/technetwork/java/javase/downloads ...

  2. Hadoop 2.6.1 集群安装配置教程

    集群环境: 192.168.56.10 master 192.168.56.11 slave1 192.168.56.12 slave2 下载安装包/拷贝安装包 # 存放路径: cd /usr/loc ...

  3. Hadoop HA高可用集群搭建(Hadoop+Zookeeper+HBase)

    声明:作者原创,转载注明出处. 作者:帅气陈吃苹果 一.服务器环境 主机名 IP 用户名 密码 安装目录 master188 192.168.29.188 hadoop hadoop /home/ha ...

  4. 完全关闭及再次启动cdh集群

    关闭集群 - 关闭集群所有组件 关闭Cloudera Management Service 关闭cdh所有客户端节点 sudo /opt/cloudera-manager/cm-5.11.1/etc/ ...

  5. 【Redis学习之九】Redis集群:Twemproxy和HA

    环境 虚拟机:VMware 10 Linux版本:CentOS-6.5-x86_64 客户端:Xshell4 FTP:Xftp4 jdk8 redis-3.0.4 主从模式对写压力没有分担,解决思路就 ...

  6. shell脚本启动所有集群节点

    #profile变量追加到.bashrc中 cat /etc/profile >> ~/.bashrc #start-all-cluster.sh  启动脚本 #!/bin/bash ec ...

  7. 使用Docker搭建Hadoop集群(伪分布式与完全分布式)

    之前用虚拟机搭建Hadoop集群(包括伪分布式和完全分布式:Hadoop之伪分布式安装),但是这样太消耗资源了,自学了Docker也来操练一把,用Docker来构建Hadoop集群,这里搭建的Hado ...

  8. 大数据系列(3)——Hadoop集群完全分布式坏境搭建

    前言 上一篇我们讲解了Hadoop单节点的安装,并且已经通过VMware安装了一台CentOS 6.8的Linux系统,咱们本篇的目标就是要配置一个真正的完全分布式的Hadoop集群,闲言少叙,进入本 ...

  9. 3-3 Hadoop集群完全分布式配置部署

    Hadoop集群完全分布式配置部署 下面的部署步骤,除非说明是在哪个服务器上操作,否则默认为在所有服务器上都要操作.为了方便,使用root用户. 1.准备工作 1.1 centOS6服务器3台 手动指 ...

随机推荐

  1. Shell中的参数传递

    原文链接 我们先来定义一个方法 function methodName() { paramName1=$1 paramName2=$2 paramName3=$3 # 依此类推,参数是从1开始标号,而 ...

  2. JDK环境的配置,及运用

    JAVA为什么可以跨平台 1.JDK配置环境变量 步骤:打开控制面板中系统和安全------系统-----找到高级系统设置点击属性------高级----环境变量------系统变量(JAVA_HOM ...

  3. 一文洞悉JVM内存管理机制

    前言 本文已经收录到我的Github个人博客,欢迎大佬们光临寒舍: 我的GIthub博客 学习导图: 一.为什么要学习内存管理? Java与C++之间有一堵由内存动态分配和垃圾回收机制所围成的高墙,墙 ...

  4. shodan常用语法

    shodan常用命令: asn         区域自治编号 port        端口 org         ip所属组织机构 os          操作系统类型 http.html   网页 ...

  5. K:剑指offer-56 题解 谁说数字电路的知识不能用到算法中?从次数统计到数字电路公式推导,一文包你全懂

    前言: 本题解整理了一位大佬在leetcode中的代码的方法,该博文致力于让所有人都能够能够看懂该方法.为此,本题解将从统计数字出现次数的解题方式开始讲起,再推导出逐位统计的解题方式,期望以循序渐进的 ...

  6. 大多数人不知道的表格其他写法的onmouseover效果

    <!DOCTYPE html> <html> <head> <meta charset="utf-8"> <meta http ...

  7. 【bzoj2049】[Sdoi2008]Cave 洞穴勘测——线段树上bfs求可撤销并查集

    题面 2049: [Sdoi2008]Cave 洞穴勘测 Time Limit: 10 Sec Memory Limit: 259 MB Submit: 12030 Solved: 6024 Desc ...

  8. 滑动窗口-Substring Search Problem

    2018-07-18 11:19:19 一.Minimum Window Substring 问题描述: 问题求解: public String minWindow(String s, String ...

  9. Python第一章-编码规范

    Python的基础知识 一.编码规范 PEP8[^ 注] 编码规范 Guido的关键点之一是:代码更多是用来读而不是写.编码规范旨在改善Python代码的可读性. 风格指南强调一致性.项目.模块或函数 ...

  10. Building Applications with Force.com and VisualForce (DEV401) (二二):Visualforce Componets (Tags) Library Part II

    Dev401-023:Visualforce Pages: Visualforce Componets (Tags) Library Part II   Apex:pageBlockTable1.A ...