HADOOP docker(一):安装hadoop实验集群(略操蛋)
一.环境准备
1.1.机器规划
主机名 别名 | IP | 角色 |
9321a27a2b91 hadoop1 | 172.17.0.10 | NN1 ZK RM |
7c3a3c9cd595 hadoop2 | 172.17.0.9 | NN2 ZK RM JOBHIS |
f89eaf2a2548 hadoop3 | 172.17.0.8 | DN ZK ND |
28620eee1426 hadoop4 | 172.17.0.7 | DN QJM1 ND |
ae1f06bd04c8 hadoop5 | 172.17.0.6 | DN QJM2 ND |
11c433a003b6 hadoop6 | 172.17.0.5 | DN QJM3 ND |
1.2.用户与组
用户 | 组 | 作用 |
hdfs | hadoop | 管理HDFS |
yarn | hadoop | 管理yarn |
zookeeper | hadoop | 管理zookeeper |
hvie | hadoop | 管理hvie |
hbase | hadoop | 管理hbase |
groupadd hadoop
useradd -g hadoop hdfs
passwd hdfs <<EOF
hdfs
hdfs
EOF
useradd -g hadoop yarn
passwd yarn <<EOF
yarn
yarn
EOF
useradd -g hadoop zookeeper
passwd zookeeper <<EOF
zookeeper
zookeeper
EOF
useradd -g hadoop hive
passwd hive <<EOF
hive
hive
EOF
useradd -g hadoop hbase
passwd hbase <<EOF
hbase
hbase
EOF
echo user added!
1.3.修改/etc/hosts
echo "127.0.0.1 localhost localhost">/etc/hosts
echo "172.17.0.6 9321a27a2b91 hadoop1">>/etc/hosts
echo "172.17.0.7 7c3a3c9cd595 hadoop2">>/etc/hosts
echo "172.17.0.8 f89eaf2a2548 hadoop3">>/etc/hosts
echo "172.17.0.9 28620eee1426 hadoop4">>/etc/hosts
echo "172.17.0.10 ae1f06bd04c8 hadoop5">>/etc/hosts
echo "172.17.0.11 11c433a003b6 hadoop6">>/etc/hosts
1.4. ssh 免密码登录
su hdfs
ssh-copy-id -i ~/.ssh/id_rsa.pub 172.17.0.6
ssh-copy-id -i ~/.ssh/id_rsa.pub 172.17.0.7
ssh-copy-id -i ~/.ssh/id_rsa.pub 172.17.0.8
ssh-copy-id -i ~/.ssh/id_rsa.pub 172.17.0.9
ssh-copy-id -i ~/.ssh/id_rsa.pub 172.17.0.10
ssh-copy-id -i ~/.ssh/id_rsa.pub 172.17.0.11
1.5.修改ulimit
[hdfs@9321a27a2b91 root]$ ulimit -a
core file size (blocks,-c) unlimited
data seg size (kbytes,-d) unlimited
scheduling priority (-e)0
file size (blocks,-f) unlimited
pending signals (-i)95612
max locked memory (kbytes,-l)64
max memory size (kbytes,-m) unlimited
open files (-n)65536
pipe size (512 bytes,-p)8
POSIX message queues (bytes,-q)819200
real-time priority (-r)0
stack size (kbytes,-s)8192
cpu time (seconds,-t) unlimited
max user processes (-u)1024
virtual memory (kbytes,-v) unlimited
file locks (-x) unlimited
hdfs hard nfile 65536
hdfs soft nfile 65536
yarn hard nfile 65536
yarn soft nfile 65536
......
6.关闭防火墙
service iptables stop
7.关闭seLinux
setenforce 0
二.软件准备
2.1.安装jdk
[root@9321a27a2b91 ~]#mkdir /usr/local/java
[root@9321a27a2b91 ~]#cp jdk-8u121-linux-x64.tar.gz /usr/local/java/
[root@9321a27a2b91 ~]#chown -R hdfs:hadoop /usr/local/java/
[root@9321a27a2b91 ~]#su hdfs
- [hdfs@9321a27a2b91 root]$ cd /usr/local/java/jdk-8u121-linux-x64.tar.gz
[hdfs@9321a27a2b91 java]$ tar -zxvf jdk-8u121-linux-x64.tar.gz
mkdir /usr/local/java
chown hdfs:hadoop /usr/local/java
su hdfs
scp -r hdfs@hadoop1:/usr/local/java/jdk1.8.0_121 /usr/local/java
2.2.hadoop安装包
[root@9321a27a2b91 ~]# mkdir /opt/hadoop
[root@9321a27a2b91 ~]# chown hdfs:hadoop hadoop-2.7.3.tar.gz
[root@9321a27a2b91 ~]# chown hdfs:hadoop /opt/hadoop
[root@9321a27a2b91 ~]# cp hadoop-2.7.3.tar.gz /opt/hadoop
[root@9321a27a2b91 ~]# su hdfs
[hdfs@9321a27a2b91 root]$ cd /opt/hadoop/
[hdfs@9321a27a2b91 hadoop]$ tar -zxvf hadoop-2.7.3.tar.gz
2.3.ntp服务
yum -y install ntp
#本子网内主机都可以同步
restrict 172.17.0.0 mask 255.255.0.0 nomodify
#优先时间服务器
server 172.17.0.10 prefer
#日志文件位置
logfile /var/log/ntp.log
[root@9321a27a2b91 hadoop]# service ntpd start
Starting ntpd:[ OK ]
[root@9321a27a2b91 hadoop]# service ntpd status
ntpd dead but pid file exists
3Apr11:20:08 ntpd[732]: ntp_io: estimated max descriptors:65536, initial socket boundary:16
3Apr11:20:08 ntpd[732]:Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
3Apr11:20:08 ntpd[732]:Listen and drop on 1 v6wildcard :: UDP 123
3Apr11:20:08 ntpd[732]:Listen normally on 2 lo 127.0.0.1 UDP 123
3Apr11:20:08 ntpd[732]:Listen normally on 3 eth0 172.17.0.10 UDP 123
3Apr11:20:08 ntpd[732]:Listen normally on 4 lo ::1 UDP 123
3Apr11:20:08 ntpd[732]:Listen normally on 5 eth0 fe80::42:acff:fe11:a UDP 123
3Apr11:20:08 ntpd[732]:Listening on routing socket on fd #22 for interface updates
3Apr11:20:08 ntpd[732]:0.0.0.0 c016 06 restart
3Apr11:20:08 ntpd[732]: ntp_adjtime() failed:Operation not permitted
3Apr11:20:08 ntpd[732]:0.0.0.0 c012 02 freq_set kernel 0.000 PPM
3Apr11:20:08 ntpd[732]:0.0.0.0 c011 01 freq_not_set
3Apr11:20:08 ntpd[732]: cap_set_proc() failed to drop root privileges:Operation not permitted
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"
echo "# Drop root to id 'ntp:ntp' by default.">/etc/sysconfig/ntpd
echo "#OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g" ">>/etc/sysconfig/ntpd
[root@9321a27a2b91 hadoop]# service ntpd start
Starting ntpd:[ OK ]
[root@9321a27a2b91 hadoop]# service ntpd status
ntpd (pid 796) is running..
server 172.17.0.10 prefer
2.4.mysql 数据库
三.安装hadoop及其组件
3.1 安装HDFS及YARN
3.1.1 设置环境变量.bash_profile
su hdfs
- vi ~.bash_profile
JAVA_HOME=/usr/local/java/jdk1.8.0_121
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASS
HADOOP_HOME=/opt/hadoop/hadoop-2.7.3
HADOOP_PREFIX=/opt/hadoop/hadoop-2.7.3
HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
HADOOP_YARN_HOME=$HADOOP_HOME
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_HOME/jre/lib/amd64/server
PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export PATH
su hdfs
scp -r hdfs@hadoop1:/home/hdfs/.bash_profile ~
3.1.2 设置hadoop启动的环境配置文件xxx-evn.sh
3.1.2.1 hadoop-env.s
export JAVA_HOME=
/usr/local/java/jdk1.8.0_121- export HADOOP_HOME=/opt/hadoop/hadoop-2.7.3
#hadoop进程的最大heapsize包括namenode/datanode/ secondarynamenode等,默认1000M
#export HADOOP_HEAPSIZE=
#namenode的初始heapsize,默认取上面的值,按需要分配
#export HADOOP_NAMENODE_INIT_HEAPSIZE=""
#JVM启动参数,默认为空
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
#还可以单独配置各个组件的内存:
export HADOOP_NAMENODE_OPTS=
export HADOOP_DATANODE_OPTS
export HADOOP_SECONDARYNAMENODE_OPTS
#设置hadoop日志,默认是$HADOOP_HOME/log
export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER
3.1.2.2 yarn-env.sh
export JAVA_HOME=
/usr/local/java/jdk1.8.0_121JAVA_HEAP_MAX=-Xmx1000m
# YARN_HEAPSIZE=1000 #yarn 守护进程heapsize
#export YARN_RESOURCEMANAGER_HEAPSIZE=1000 #单独设置RESOURCEMANAGER的HEAPSIZE
#export YARN_TIMELINESERVER_HEAPSIZE=1000 #单独设置TIMELINESERVER(jobhistoryServer)的HEAPSIZE
#export YARN_RESOURCEMANAGER_OPTS= #单独设置RESOURCEMANAGER的JVM选项
#export YARN_NODEMANAGER_HEAPSIZE=1000 #单独设置NODEMANAGER的HEAPSIZE
#export YARN_NODEMANAGER_OPTS= #单独设置NODEMANAGER的JVM选项
3.1.3 修改hadoop配置文件
3.1.3.1 修改core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop1:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop/hadoop-2.7.3/tmp</value>
<description>默认值/tmp/hadoop-${user.name},修改成持久化的目录</description>
</property>
</configuration>
mkdir ${HADOOP_HOME}/tmp
3.1.3.2 hdfs-site.xm
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
- <description>数据块的备份数量</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/hadoop/hadoop-2.7.3/namenodedir</value>
- <description>保存namenode元数据的目录,要自己创建</description>
</property>
<property>
<name>dfs.blocksize</name>
<value>134217728</value>
<description>数据块大小,128M</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/opt/hadoop/hadoop-2.7.3/datadir</value>
</property>
</configuration>
mkdir ${HADOOP_HOME}/datadir
mkdir
${HADOOP_HOME}/namenodedir
3.1.3.3 mapred-site.xml
Parameter | Value | Notes |
---|---|---|
mapreduce.framework.name | yarn | Execution framework set to Hadoop YARN. MR任务执行框架 |
mapreduce.map.memory.mb | 1536 | Larger resource limit for maps. map内存上限 |
mapreduce.map.java.opts | -Xmx1024M | Larger heap-size for child jvms of maps. map的子进程虚拟机heapsize |
mapreduce.reduce.memory.mb | 3072 | Larger resource limit for reduces. redouce任务内存上限 |
mapreduce.reduce.java.opts | -Xmx2560M | Larger heap-size for child jvms of reduces. redouce的子进程虚拟机heapsize |
mapreduce.task.io.sort.mb | 512 | Higher memory-limit while sorting data for efficiency. 排序内存 |
mapreduce.task.io.sort.factor | 100 | More streams merged at once while sorting files. 排序因子 |
mapreduce.reduce.shuffle.parallelcopies | 50 | Higher number of parallel copies run by reduces to fetch outputs from very large number of maps. 并行数 |
Parameter | Value | Notes |
---|---|---|
mapreduce.jobhistory.address | MapReduce JobHistory Server host:port | Default port is 10020. jobhistory地址:主机+端口 |
mapreduce.jobhistory.webapp.address | MapReduce JobHistory Server Web UI host:port | Default port is 19888. jobhistory web端口 |
mapreduce.jobhistory.intermediate-done-dir | /mr-history/tmp | Directory where history files are written by MapReduce jobs. |
mapreduce.jobhistory.done-dir | /mr-history/done | Directory where history files are managed by the MR JobHistory Server. |
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>使用yarn来管理mr</description>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop2</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop2</value>
</property>
<property>
<name>mapreduce.jobhistory.intermediate-done-dir</name>
<value>/opt/hadoop/hadoop-2.7.3/mrHtmp</value>
</property>
<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/opt/hadoop/hadoop-2.7.3/mrhHdone</value>
</property>
</configuration>
3.1.3.4 yarn-site.xml
Parameter | Value | Notes |
---|---|---|
yarn.resourcemanager.address | ResourceManager host:port for clients to submit jobs. | host:port If set, overrides the hostname set in yarn.resourcemanager.hostname. resourcemanager的地址,格式 主机:端口 |
yarn.resourcemanager.scheduler.address | ResourceManager host:port for ApplicationMasters to talk to Scheduler to obtain resources. | host:port If set, overrides the hostname set in yarn.resourcemanager.hostname. 调度器地址 ,覆盖yarn.resourcemanager.hostname |
yarn.resourcemanager.resource-tracker.address | ResourceManager host:port for NodeManagers. | host:port If set, overrides the hostname set in yarn.resourcemanager.hostname. datanode像rm报告的端口,
覆盖 yarn.resourcemanager.hostname |
yarn.resourcemanager.admin.address | ResourceManager host:port for administrative commands. | host:port If set, overrides the hostname set in yarn.resourcemanager.hostname. RM管理地址,覆盖 yarn.resourcemanager.hostname |
yarn.resourcemanager.webapp.address | ResourceManager web-ui host:port. | host:port If set, overrides the hostname set in yarn.resourcemanager.hostname. RM web地址,有默认值 |
yarn.resourcemanager.hostname | ResourceManager host. | host Single hostname that can be set in place of setting allyarn.resourcemanager*address resources. Results in default ports for ResourceManager components. RM的主机,使用默认端口 |
yarn.resourcemanager.scheduler.class | ResourceManager Scheduler class. | CapacityScheduler (recommended), FairScheduler (also recommended), or FifoScheduler |
yarn.scheduler.minimum-allocation-mb | Minimum limit of memory to allocate to each container request at the Resource Manager. | In MBs 最小容器内存(每个container最小内存) |
yarn.scheduler.maximum-allocation-mb | Maximum limit of memory to allocate to each container request at the Resource Manager. | In MBs 最大容器内存(每个container最大内存) |
yarn.resourcemanager.nodes.include-path /yarn.resourcemanager.nodes.exclude-path | List of permitted/excluded NodeManagers. | If necessary, use these files to control the list of allowable NodeManagers. 哪些datanode可以被RM管理 |
yarn.nodemanager.resource.memory-mb | Resource i.e. available physical memory, in MB, for given NodeManager | Defines total available resources on the NodeManager to be made available to running containers Yarn在NodeManager最大内存 |
yarn.nodemanager.vmem-pmem-ratio | Maximum ratio by which virtual memory usage of tasks may exceed physical memory | The virtual memory usage of each task may exceed its physical memory limit by this ratio. The total amount of virtual memory used by tasks on the NodeManager may exceed its physical memory usage by this ratio. 任务使用的虚拟内存超过被允许的推理内存的比率,超过则kill掉 |
yarn.nodemanager.local-dirs | Comma-separated list of paths on the local filesystem where intermediate data is written. | Multiple paths help spread disk i/o. datamanager的本地目录 |
yarn.nodemanager.log-dirs | Comma-separated list of paths on the local filesystem where logs are written. | Multiple paths help spread disk i/o. datamanager日志目录 |
yarn.nodemanager.log.retain-seconds | 10800 | Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled. |
yarn.nodemanager.remote-app-log-dir | /logs | HDFS directory where the application logs are moved on application completion. Need to set appropriate permissions. Only applicable if log-aggregation is enabled. |
yarn.nodemanager.remote-app-log-dir-suffix | logs | Suffix appended to the remote log dir. Logs will be aggregated to ${yarn.nodemanager.remote-app-log-dir}/${user}/${thisParam} Only applicable if log-aggregation is enabled. |
yarn.nodemanager.aux-services | mapreduce_shuffle | Shuffle service that needs to be set for Map Reduce applications. shuffle服务类型 |
yarn.acl.enable | true /false | Enable ACLs? Defaults to false. 是否开启ACL |
yarn.admin.acl | Admin ACL | ACL to set admins on the cluster. ACLs are of for comma-separated-usersspacecomma-separated-groups. Defaults to special value of * which meansanyone. Special value of just space means no one has access. ACL用户,用,分隔 如root,yarn |
yarn.log-aggregation-enable | false | Configuration to enable or disable log aggregation 启用日志聚集.日志聚焦到一个节点 |
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop1</value>
<description>设置resourcemanager节点</description>
</property>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>设置nodemanager的aux服务</description>
</property>
</configuration>
3.1.4 设置slaves文件
vi $HADOOP_HOME/et/hadoop/slaves
hadoop3
hadoop4
hadoop5
hadoop6
3.1.5 启动HADOOP
3.1.5.1 把hadoop配置复制到其它机器上
mkdir /opt/hadoop
chown hdfs:hadoop /opt/hadoop
su hdfs
scp -r hdfs@hadoop1:/opt/hadoop/hadoop-2.7.3 /opt/hadoop
$HADOOP_HOME/bin/hdfs namenode -format
3.1.5.3 启动hdfs
[hdfs@9321a27a2b91 hadoop-2.7.3]$ start-dfs.sh
Starting namenodes on [hadoop1]
hadoop1: starting namenode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-namenode-9321a27a2b91.out
hadoop3: starting datanode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-datanode-f89eaf2a2548.out
hadoop4: starting datanode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-datanode-28620eee1426.out
hadoop5: starting datanode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-datanode-ae1f06bd04c8.out
hadoop6: starting datanode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-datanode-11c433a003b6.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-secondarynamenode-9321a27a2b91.out
[hdfs@9321a27a2b91 hadoop]$ jps
11105 Jps
10981 SecondaryNameNode
10777 NameNode
[hdfs@9321a27a2b91 hadoop-2.7.3]$ hdfs dfs -put NOTICE.txt /
[hdfs@9321a27a2b91 hadoop-2.7.3]$ hdfs dfs -ls /
Found 1 items
-rw-r--r-- 3 hdfs supergroup 14978 2017-04-03 19:15 /NOTICE.txt
[root@9321a27a2b91 hdfs]# curl hadoop1:50070
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
................

3.1.5.4 启动yarn
[hdfs@9321a27a2b91 hadoop]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop/hadoop-2.7.3/logs/yarn-hdfs-resourcemanager-9321a27a2b91.out
hadoop5: starting nodemanager, logging to /opt/hadoop/hadoop-2.7.3/logs/yarn-hdfs-nodemanager-ae1f06bd04c8.out
hadoop6: starting nodemanager, logging to /opt/hadoop/hadoop-2.7.3/logs/yarn-hdfs-nodemanager-11c433a003b6.out
hadoop3: starting nodemanager, logging to /opt/hadoop/hadoop-2.7.3/logs/yarn-hdfs-nodemanager-f89eaf2a2548.out
hadoop4: starting nodemanager, logging to /opt/hadoop/hadoop-2.7.3/logs/yarn-hdfs-nodemanager-28620eee1426.out
[hdfs@9321a27a2b91 hadoop]$ jps
11105 Jps
10981 SecondaryNameNode
10777 NameNode
10383 ResourceManager


[hdfs@9321a27a2b91 hadoop-2.7.3]$ bin/hdfs dfs -mkdir /user
[hdfs@9321a27a2b91 hadoop-2.7.3]$ bin/hdfs dfs -mkdir /user/hdfs
[hdfs@9321a27a2b91 hadoop-2.7.3]$ bin/hdfs dfs -put etc/hadoop input
...............
17/04/12 12:38:24 INFO mapreduce.JobSubmitter: number of splits:30
17/04/12 12:38:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1491968887469_0003
17/04/12 12:38:24 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/hdfs/.staging/job_1491968887469_0003
java.lang.IllegalArgumentException: Does not contain a valid host:port authority: hadoop2
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:213)
........................
[hdfs@9321a27a2b91 hadoop]$ start-dfs.sh
Starting namenodes on [9321a27a2b91]
The authenticity of host '9321a27a2b91 (172.17.0.10)' can't be established.
RSA key fingerprint is 60:0c:61:73:2c:49:ef:e3:f7:61:c9:27:93:5a:1d:c7.
Are you sure you want to continue connecting (yes/no)?
3.1.5.6 各个节点上启动hdfs守护进程
[hdfs@11c433a003b6 hadoop-2.7.3]$ $HADOOP_HOME/sbin/hadoop-daemons.sh start datanode
The authenticity of host '28620eee1426 (172.17.0.7)' can't be established.
RSA key fingerprint is 60:0c:61:73:2c:49:ef:e3:f7:61:c9:27:93:5a:1d:c7.
Are you sure you want to continue connecting (yes/no)? The authenticity of host '11c433a003b6 (172.17.0.5)' can't be established.
RSA key fingerprint is 60:0c:61:73:2c:49:ef:e3:f7:61:c9:27:93:5a:1d:c7.
Are you sure you want to continue connecting (yes/no)? The authenticity of host 'ae1f06bd04c8 (172.17.0.6)' can't be established.
RSA key fingerprint is 60:0c:61:73:2c:49:ef:e3:f7:61:c9:27:93:5a:1d:c7.
Are you sure you want to continue connecting (yes/no)? f89eaf2a2548: datanode running as process 5764. Stop it first.
[hdfs@11c433a003b6 hadoop-2.7.3]$ $HADOOP_HOME/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs start datanode
cat: /opt/hadoop/hadoop-2.7.3/etc/hadoop/slaves: No such file or directory
usage="Usage: hadoop-daemons.sh [--config confdir] [--hosts hostlistfile] [start|stop] command args..."
# if no args specified, show usage
if[ $# -le 1 ]; then
echo $usage
exit 1
fi
bin=`dirname "${BASH_SOURCE-$0}"`
bin=`cd "$bin"; pwd`
DEFAULT_LIBEXEC_DIR="$bin"/../libexec
HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}
. $HADOOP_LIBEXEC_DIR/hadoop-config.sh
exec "$bin/slaves.sh"--config $HADOOP_CONF_DIR cd "$HADOOP_PREFIX" \; "$bin/hadoop-daemon.sh"--config $HADOOP_CONF_DIR "$@"

[hdfs@9321a27a2b91 hadoop]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode
starting namenode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-namenode-9321a27a2b91.out
以下节点启动nodemanager:
[hdfs@f89eaf2a2548 hadoop-2.7.3]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager
starting resourcemanager, logging to /opt/hadoop/hadoop-2.7.3/logs/yarn-hdfs-resourcemanager-f89eaf2a2548.out

[hdfs@9321a27a2b91 hadoop-2.7.3]$ bin/hdfs dfs -put etc/hadoop input
...............
17/04/1212:38:24 INFO mapreduce.JobSubmitter: number of splits:30
17/04/1212:38:24 INFO mapreduce.JobSubmitter:Submitting tokens for job: job_1491968887469_0003
17/04/1212:38:24 INFO mapreduce.JobSubmitter:Cleaning up the staging area /tmp/hadoop-yarn/staging/hdfs/.staging/job_1491968887469_0003
java.lang.IllegalArgumentException:Does not contain a valid host:port authority: hadoop2
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:213)
........................
<property>
<name>mapreduce.jobhistory.address</name>
<value>7c3a3c9cd595</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>7c3a3c9cd595</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>7c3a3c9cd595:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>7c3a3c9cd595:19888</value>
</property>
2017-04-0319:13:12,328 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:IOExceptionin offerService
java.io.EOFException:End of FileException between local host is:"ae1f06bd04c8/172.17.0.6"; destination host is:"hadoop1":9000;: java.io.EOFException;For more details see: http://wiki.a
pache.org/hadoop/EOFException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
四.总结
4.1.hadoop的安装步骤
4.2问题总结
HADOOP docker(一):安装hadoop实验集群(略操蛋)的更多相关文章
- 使用docker安装部署Spark集群来训练CNN(含Python实例)
使用docker安装部署Spark集群来训练CNN(含Python实例) http://blog.csdn.net/cyh_24/article/details/49683221 实验室有4台神服务器 ...
- Hadoop 2.6.4单节点集群配置
1.安装配置步骤 # wget http://download.oracle.com/otn-pub/java/jdk/8u91-b14/jdk-8u91-linux-x64.rpm # rpm -i ...
- 基于Hadoop 2.2.0的高可用性集群搭建步骤(64位)
内容概要: CentSO_64bit集群搭建, hadoop2.2(64位)编译,安装,配置以及测试步骤 新版亮点: 基于yarn计算框架和高可用性DFS的第一个稳定版本. 注1:官网只提供32位re ...
- Hadoop加zookeeper构建高可靠集群
事前准备 1.更改Linux主机名,每个人都有配置 vim /etc/sysconfig/network NETWORKING=yes HOSTNAME=hadoop-server1 2.改动IP / ...
- Hadoop及Zookeeper+HBase完全分布式集群部署
Hadoop及HBase集群部署 一. 集群环境 系统版本 虚拟机:内存 16G CPU 双核心 系统: CentOS-7 64位 系统下载地址: http://124.202.164.6/files ...
- Hadoop入门 完全分布式运行模式-集群配置
目录 集群配置 集群部署规划 配置文件说明 配置集群 群起集群 1 配置workers 2 启动集群 总结 3 集群基本测试 上传文件到集群 查看数据真实存储路径 下载 执行wordcount程序 配 ...
- [Hadoop] - Win7下提交job到集群上去
一般我们采用win开发+linux hadoop集群的方式进行开发,使用插件:hadoop-***-eclipse-plugin. 运行程序的时候,我们一般采用run as application或者 ...
- Docker安装部署es集群
Docker安装部署es集群:环境准备:已安装docker的centos服务器一台1. 拉取es版本docker pull elasticsearch:5.6.82. 新建文件夹 数据挂载目录 和 配 ...
- docker安装Elasticsearch7.6集群并设置密码
docker安装Elasticsearch7.6集群并设置密码 Elasticsearch从6.8开始, 允许免费用户使用X-Pack的安全功能, 以前安装es都是裸奔.接下来记录配置安全认证的方法. ...
随机推荐
- Json中dumps、loads、dump、load函数实例讲解
1.dumps() 1. json.dumps() 用于将字典(dic)类型的数据转成字符串(str),直接将dict类型的数据写入json文件中会发生报错,因此在将数据写入时需要用到该函数. imp ...
- LightOJ 1203--Guarding Bananas(二维凸包+内角计算)
1203 - Guarding Bananas PDF (English) Statistics Forum Time Limit: 3 second(s) Memory Limit: 32 M ...
- Spring Cloud 微服务入门(一)--初识分布式及其发展历程
分布式开发出现背景 当有计算机出现一段时间之后就开始有人去想如何将不同的电脑进行网络连接,而网络连接之后对于web的项目开发就探索所谓的分布式设计,同时人们也意识到重要的数据必须多份存在.所以分布式就 ...
- MySQL---下载安装、数据库基本操作
1.下载安装 1.1 下载: http://dev.mysql.com/downloads/mysql/ 1.2 解压 1.3 初始化 cd c:\mysql-5.7.16-winx64\bin ( ...
- 万恶的a标签
相信很多人碰见过这些问题吧 给某个a标签套的元素中添加点击事件 在外面就能获取到但是点击事件不生效把 或者在页面中点击一个a标签元素发现页面返回了最顶端 然后就开始郁闷了 哈哈 其实这些看似神奇的 ...
- yii 后台配置独立子域名方法
我这里安装的是宝塔面板集成的环境WNMP,官网上虽然也有,但是写的并不明确,对我这种用YII的新手来说也很头疼,折腾了半天终于弄好,记录一下. 首先解析一个子域名:back.domain.com: 用 ...
- rails应用中各数据平台的对接
1.mongo #Gemfile添加如下两个gem包gem 'mongoid', '~> 5.1.0' gem 'mongo', '~> 2.4’ @client = Mongo::Cli ...
- (cmath)关于头文件cmath
关于头文件cmath // -*- C++ -*- C forwarding header. // Copyright (C) 1997-2014 Free Software Foundation, ...
- day 2 给程序传递参数
1.如何实现变化name name = "alex" print("欢迎%s前来指导学习"%name) 欢迎alex前来指导学习 2.sys.argv impo ...
- dvs-panotracking编译运行
编译运行dvs-panotracking > 编译dvs-panotracking之前首先需要安装imageutilities . 源码下载 https://github.com/VLOGrou ...