Hadoop启动脚本分析

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

　　　　能看到这篇博客的你估计对Hadoop已经有一个系统的了解了，最起码各种搭建方式你应该是会的，不会也没有关系，可以参考我的笔记，里面有各种搭建方式，哈哈哈~

[yinzhengjie@s101 ~]$ cat `which xcall.sh`

#!/bin/bash

#@author :yinzhengjie

#blog:http://www.cnblogs.com/yinzhengjie

#EMAIL:y1053419035@qq.com

#判断用户是否传参

if [ $# -lt  ];then

        echo "请输入参数"

        exit

fi

#获取用户输入的命令

cmd=$@

for (( i=;i<=;i++ ))

do

        #使终端变绿色

        tput setaf

        echo ============= s$i $cmd ============

        #使终端变回原来的颜色，即白灰色

        tput setaf

        #远程执行命令

        ssh s$i $cmd

        #判断命令是否执行成功

        if [ $? ==  ];then

                echo "命令执行成功"

        fi

done

[yinzhengjie@s101 ~]$

xcall.sh 脚本内容，我在测试的时候经常使用它

一.start-all.sh脚本分析

[yinzhengjie@s101 ~]$ cat `which start-all.sh`  | grep -v ^# | grep -v ^$

echo "This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/hadoop-config.sh

if [ -f "${HADOOP_HDFS_HOME}"/sbin/start-dfs.sh ]; then

  "${HADOOP_HDFS_HOME}"/sbin/start-dfs.sh --config $HADOOP_CONF_DIR

fi

if [ -f "${HADOOP_YARN_HOME}"/sbin/start-yarn.sh ]; then

  "${HADOOP_YARN_HOME}"/sbin/start-yarn.sh --config $HADOOP_CONF_DIR

fi

[yinzhengjie@s101 ~]$

　　从这个脚本中的第一行我们可以看出来，这个脚本已经过时了，取而代之的是：“This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh”，也就是 “start-dfs.sh”和“start-yarn.sh”。

二.start-dfs.sh 脚本分析

[yinzhengjie@s101 ~]$ more `which start-dfs.sh` | grep -v ^# | grep -v ^$

usage="Usage: start-dfs.sh [-upgrade|-rollback] [other options such as -clusterId]"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/hdfs-config.sh

if [[ $# -ge  ]]; then

  startOpt="$1"

  shift

  case "$startOpt" in

    -upgrade)

      nameStartOpt="$startOpt"

    ;;

    -rollback)

      dataStartOpt="$startOpt"

    ;;

    *)

      echo $usage

      exit

    ;;

  esac

fi

nameStartOpt="$nameStartOpt $@"

NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -namenodes)

echo "Starting namenodes on [$NAMENODES]"

"$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

  --config "$HADOOP_CONF_DIR" \

  --hostnames "$NAMENODES" \

  --script "$bin/hdfs" start namenode $nameStartOpt

if [ -n "$HADOOP_SECURE_DN_USER" ]; then

  echo \

    "Attempting to start secure cluster, skipping datanodes. " \

    "Run start-secure-dns.sh as root to complete startup."

else

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

    --config "$HADOOP_CONF_DIR" \

    --script "$bin/hdfs" start datanode $dataStartOpt

fi

SECONDARY_NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -secondarynamenodes >/dev/null)

if [ -n "$SECONDARY_NAMENODES" ]; then

  echo "Starting secondary namenodes [$SECONDARY_NAMENODES]"

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

      --config "$HADOOP_CONF_DIR" \

      --hostnames "$SECONDARY_NAMENODES" \

      --script "$bin/hdfs" start secondarynamenode

fi

SHARED_EDITS_DIR=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.namenode.shared.edits.dir >&-)

case "$SHARED_EDITS_DIR" in

qjournal://*)

  JOURNAL_NODES=$(echo "$SHARED_EDITS_DIR" | sed 's,qjournal://\([^/]*\)/.*,\1,g; s/;/ /g; s/:[0-9]*//g')

  echo "Starting journal nodes [$JOURNAL_NODES]"

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

      --config "$HADOOP_CONF_DIR" \

      --hostnames "$JOURNAL_NODES" \

      --script "$bin/hdfs" start journalnode ;;

esac

AUTOHA_ENABLED=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.ha.automatic-failover.enabled)

if [ "$(echo "$AUTOHA_ENABLED" | tr A-Z a-z)" = "true" ]; then

  echo "Starting ZK Failover Controllers on NN hosts [$NAMENODES]"

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

    --config "$HADOOP_CONF_DIR" \

    --hostnames "$NAMENODES" \

    --script "$bin/hdfs" start zkfc

fi

[yinzhengjie@s101 ~]$

　　以上的注释已经被我过滤掉了，从这个脚本中大致可以看出这个脚本是用来启动hdfs进程的，即分别是：NameNode，DataNode以及secondaryNameNode。

1>.单独启动NameNode脚本用法如下：

[yinzhengjie@s101 ~]$ hadoop-daemon.sh --hostnames s101 start namenode

starting namenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-namenode-s101.out

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

 NameNode

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

　　以上是单独启动NameNode节点的脚本用法，如果你想要批量启动的话可以使用hadoop-daemons.sh命令，只不过由于我部署的集群环境只有一个NameNode节点，因此看不出来有任何效果。

[yinzhengjie@s101 ~]$ hadoop-daemons.sh --hostnames  ` hdfs getconf -namenodes` start namenode

s101: starting namenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-namenode-s101.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

 NameNode

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ hadoop-daemons.sh --hostnames ` hdfs getconf -namenodes` start namenode

2>.单独启动DataNode脚本如下：

[yinzhengjie@s101 ~]$ hadoop-daemon.sh start datanode

starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s101.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

 DataNode

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

　　以上是单独启动DataNode的脚本用法，想要执行如果你想要批量启动的话可以使用hadoop-daemons.sh命令，由于我有三个节点，看起来效果就很明显了。

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$ hadoop-daemons.sh start datanode

s102: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s102.out

s104: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s104.out

s103: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s103.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

 DataNode

命令执行成功

============= s103 jps ============

 Jps

 DataNode

命令执行成功

============= s104 jps ============

 Jps

 DataNode

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ hadoop-daemons.sh start datanode

3>.单独启动secondaryNameNode

[yinzhengjie@s101 ~]$ hadoop-daemon.sh --hostnames s101 start secondarynamenode

starting secondarynamenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-secondarynamenode-s101.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 SecondaryNameNode

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

　　以上是单独启动secondaryNameNode的脚本用法，想要执行如果你想要批量启动的话可以使用hadoop-daemons.sh命令，由于我有三个节点，看起来效果就很明显了。

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ for i in `cat /soft/hadoop/etc/hadoop/slaves | grep -v ^#` ;do  hadoop-daemons.sh --hostnames $i start secondarynamenode ;done

s102: starting secondarynamenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-secondarynamenode-s102.out

s103: starting secondarynamenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-secondarynamenode-s103.out

s104: starting secondarynamenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-secondarynamenode-s104.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

 SecondaryNameNode

命令执行成功

============= s103 jps ============

 Jps

 SecondaryNameNode

命令执行成功

============= s104 jps ============

 Jps

 SecondaryNameNode

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ for i in `cat /soft/hadoop/etc/hadoop/slaves | grep -v ^#` ;do hadoop-daemons.sh --hostnames $i start secondarynamenode ;done

三.start-yarn.sh 脚本分析

[yinzhengjie@s101 ~]$ cat /soft/hadoop/sbin/start-yarn.sh | grep -v ^# | grep -v ^$

echo "starting yarn daemons"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/yarn-config.sh

"$bin"/yarn-daemon.sh --config $YARN_CONF_DIR  start resourcemanager

"$bin"/yarn-daemons.sh --config $YARN_CONF_DIR  start nodemanager

[yinzhengjie@s101 ~]$

　　其实用法跟上面的类似，单独启动进程如下：

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ yarn-daemon.sh start  nodemanager

starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s101.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 NodeManager

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ yarn-daemon.sh start nodemanager

　　如果想要想要批量启动的，实操如下：

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$ yarn-daemons.sh start  nodemanager

s102: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s102.out

s104: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s104.out

s103: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s103.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

 NodeManager

命令执行成功

============= s103 jps ============

 Jps

 NodeManager

命令执行成功

============= s104 jps ============

 Jps

 NodeManager

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ yarn-daemons.sh start nodemanager

二.stop-all.sh脚本分析

[yinzhengjie@s101 ~]$ cat `which stop-all.sh` | grep -v ^#  | grep -v ^$

echo "This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/hadoop-config.sh

if [ -f "${HADOOP_HDFS_HOME}"/sbin/stop-dfs.sh ]; then

  "${HADOOP_HDFS_HOME}"/sbin/stop-dfs.sh --config $HADOOP_CONF_DIR

fi

if [ -f "${HADOOP_HDFS_HOME}"/sbin/stop-yarn.sh ]; then

  "${HADOOP_HDFS_HOME}"/sbin/stop-yarn.sh --config $HADOOP_CONF_DIR

fi

[yinzhengjie@s101 ~]$

　　看到第一行时：echo "This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh"，估计你已经明白是怎么回事了把，就是把上面的所有start参数换成了stop参数。从这个脚本中的第一行我们可以看出来，这个脚本已经过时了，取而代之的是：“This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh”，也就是 “stop-dfs.sh”和“stop-yarn.sh”。

三.小结

　　综上所述，我们可以得到以下四个等式：

1>.start-all.sh = start-dfs.sh + start-yarn.sh

2>.stop-all.sh = stop-dfs.sh + stop-yarn.sh

3>.hadoop-damons.sh = hadoop-damon.sh + slaves

4>.yarn-damons.sh = yarn-damon.sh + slaves

Hadoop启动脚本分析的更多相关文章

  1. 4. hadoop启动脚本分析

    4. hadoop启动脚本分析 1. hadoop的端口 ``` 50070 //namenode http port 50075 //datanode http port 50090 //2name ...

  2. hadoop启动脚本分析及常见命令

    进程------------------ [hdfs]start-dfs.sh NameNode NN DataNode DN SecondaryNamenode 2NN [yarn]start-ya ...

  3. Hadoop 启动脚本分析与实战经验

    start-all.sh脚本现在已经废弃,推荐使用start-dfs.sh和start-yarn.sh分别启动HDFS和YARN. 在新一代的Hadoop里面HDFS称为了统一存储的平台,而YARN成 ...

  4. HBase配置&启动脚本分析

    本文档基于hbase-0.96.1.1-cdh5.0.2,对HBase配置&启动脚本进行分析 date:2016/8/4 author:wangxl HBase配置&启动脚本分析 剔除 ...

  5. Spark配置&启动脚本分析

    本文档基于Spark2.0,对spark启动脚本进行分析. date:2016/8/3 author:wangxl Spark配置&启动脚本分析 我们主要关注3类文件,配置文件,启动脚本文件以 ...

  6. hadoop启动脚本

    记录一下一个简单的hadoop启动脚本 就是启动zookeeper集群,hadoop的HDFS和YRAN的脚本 start-cluster.sh 关于关闭的脚本,只需要顺序换一下,然后将start改为 ...

  7. busybox rootfs 启动脚本分析(二）

    上次分析了busybox的启动脚本,这次分析一下init.d中一些脚本的内容. 参考链接 http://www.cnblogs.com/helloworldtoyou/p/6169678.html h ...

  8. busybox rootfs 启动脚本分析(一）

    imx6文件系统启动脚本分析.开机运行/sbin/init,读取/etc/inittab文件,进行初始化. 参考链接 http://blog.163.com/wghbeyond@126/blog/st ...

  9. Hadoop 2.0+YARN启动脚本分析与实战经验

    start-all.sh脚本现在已经废弃,推荐使用start-dfs.sh和start-yarn.sh分别启动HDFS和YARN. 在新一代的Hadoop里面HDFS称为了统一存储的平台,而YARN成 ...

随机推荐

  1. DWH中增量数据的抽取

    1. Truncate-Load 全量加载 简单直观.不易出错,适合数据量不太大的操作 性能问题 2. Increamental-Load 只考虑新增.修改.删除的记录    良好的数据源设计(主要是 ...

  2. kubernetes 集群安装etcd集群，带证书

    install etcd 准备证书 https://www.kubernetes.org.cn/3096.html 在master1需要安装CFSSL工具,这将会用来建立 TLS certificat ...

  3. 好程序员技术教程分享JavaScript运动框架

    好程序员技术教程分享JavaScript运动框架,有需要的朋友可以参考下. JavaScript的运动,即让某元素的某些属性由一个值变到另一个值的过程.如让div的width属性由200px变到400 ...

  4. sqlServer：行列转换之多行转一行

    1.建表:学生表(姓名,学科,成绩) CREATE TABLE teststudent(    stuname varchar(50) NULL,    subjects varchar(50) NU ...

  5. SpringBoot整合RabbitMQ-消息可靠性投递

    本系列是学习SpringBoot整合RabbitMQ的练手,包含服务安装,RabbitMQ整合SpringBoot2.x,消息可靠性投递实现等三篇博客. 学习路径:https://www.imooc. ...

  6. 【转】详解springboot-修改内置tomcat版本

    1.解析Spring Boot父级依赖 <parent> <groupId>org.springframework.boot</groupId> <artif ...

  7. 全解史上最快的JOSN解析库 - alibaba Fastjson

    JSON,全称:JavaScript Object Notation,作为一个常见的轻量级的数据交换格式,应该在一个程序员的开发生涯中是常接触的.简洁和清晰的层次结构使得 JSON 成为理想的数据交换 ...

  8. TypeError: sequence item 1: expected str instance, int found

    Error Msg Traceback (most recent call last): File "E:/code/adva_code/my_orm.py", line 108, ...

  9. 292. Nim Game(easy)

    You are playing the following Nim Game with your friend: There is a heap of stones on the table, eac ...

  10. [第二届构建之法论坛] 预培训文档(C++版)

    本博客是第二届构建之法论坛暨软件工程培训活动预培训文档中[适用于结对编程部分的C++版本],需要实验者有一部分C++基础. 目录 Part0.背景 Part1.配置环境 Part2.克隆项目 Part ...