HADOOP docker(一):安装hadoop实验集群(略操蛋)
一.环境准备
1.1.机器规划
主机名 别名 | IP | 角色 |
9321a27a2b91 hadoop1 | 172.17.0.10 | NN1 ZK RM |
7c3a3c9cd595 hadoop2 | 172.17.0.9 | NN2 ZK RM JOBHIS |
f89eaf2a2548 hadoop3 | 172.17.0.8 | DN ZK ND |
28620eee1426 hadoop4 | 172.17.0.7 | DN QJM1 ND |
ae1f06bd04c8 hadoop5 | 172.17.0.6 | DN QJM2 ND |
11c433a003b6 hadoop6 | 172.17.0.5 | DN QJM3 ND |
1.2.用户与组
用户 | 组 | 作用 |
hdfs | hadoop | 管理HDFS |
yarn | hadoop | 管理yarn |
zookeeper | hadoop | 管理zookeeper |
hvie | hadoop | 管理hvie |
hbase | hadoop | 管理hbase |
groupadd hadoop
useradd -g hadoop hdfs
passwd hdfs <<EOF
hdfs
hdfs
EOF
useradd -g hadoop yarn
passwd yarn <<EOF
yarn
yarn
EOF
useradd -g hadoop zookeeper
passwd zookeeper <<EOF
zookeeper
zookeeper
EOF
useradd -g hadoop hive
passwd hive <<EOF
hive
hive
EOF
useradd -g hadoop hbase
passwd hbase <<EOF
hbase
hbase
EOF
echo user added!
1.3.修改/etc/hosts
echo "127.0.0.1 localhost localhost">/etc/hosts
echo "172.17.0.6 9321a27a2b91 hadoop1">>/etc/hosts
echo "172.17.0.7 7c3a3c9cd595 hadoop2">>/etc/hosts
echo "172.17.0.8 f89eaf2a2548 hadoop3">>/etc/hosts
echo "172.17.0.9 28620eee1426 hadoop4">>/etc/hosts
echo "172.17.0.10 ae1f06bd04c8 hadoop5">>/etc/hosts
echo "172.17.0.11 11c433a003b6 hadoop6">>/etc/hosts
1.4. ssh 免密码登录
su hdfs
ssh-copy-id -i ~/.ssh/id_rsa.pub 172.17.0.6
ssh-copy-id -i ~/.ssh/id_rsa.pub 172.17.0.7
ssh-copy-id -i ~/.ssh/id_rsa.pub 172.17.0.8
ssh-copy-id -i ~/.ssh/id_rsa.pub 172.17.0.9
ssh-copy-id -i ~/.ssh/id_rsa.pub 172.17.0.10
ssh-copy-id -i ~/.ssh/id_rsa.pub 172.17.0.11
1.5.修改ulimit
[hdfs@9321a27a2b91 root]$ ulimit -a
core file size (blocks,-c) unlimited
data seg size (kbytes,-d) unlimited
scheduling priority (-e)0
file size (blocks,-f) unlimited
pending signals (-i)95612
max locked memory (kbytes,-l)64
max memory size (kbytes,-m) unlimited
open files (-n)65536
pipe size (512 bytes,-p)8
POSIX message queues (bytes,-q)819200
real-time priority (-r)0
stack size (kbytes,-s)8192
cpu time (seconds,-t) unlimited
max user processes (-u)1024
virtual memory (kbytes,-v) unlimited
file locks (-x) unlimited
hdfs hard nfile 65536
hdfs soft nfile 65536
yarn hard nfile 65536
yarn soft nfile 65536
......
6.关闭防火墙
service iptables stop
7.关闭seLinux
setenforce 0
二.软件准备
2.1.安装jdk
[root@9321a27a2b91 ~]#mkdir /usr/local/java
[root@9321a27a2b91 ~]#cp jdk-8u121-linux-x64.tar.gz /usr/local/java/
[root@9321a27a2b91 ~]#chown -R hdfs:hadoop /usr/local/java/
[root@9321a27a2b91 ~]#su hdfs
- [hdfs@9321a27a2b91 root]$ cd /usr/local/java/jdk-8u121-linux-x64.tar.gz
[hdfs@9321a27a2b91 java]$ tar -zxvf jdk-8u121-linux-x64.tar.gz
mkdir /usr/local/java
chown hdfs:hadoop /usr/local/java
su hdfs
scp -r hdfs@hadoop1:/usr/local/java/jdk1.8.0_121 /usr/local/java
2.2.hadoop安装包
[root@9321a27a2b91 ~]# mkdir /opt/hadoop
[root@9321a27a2b91 ~]# chown hdfs:hadoop hadoop-2.7.3.tar.gz
[root@9321a27a2b91 ~]# chown hdfs:hadoop /opt/hadoop
[root@9321a27a2b91 ~]# cp hadoop-2.7.3.tar.gz /opt/hadoop
[root@9321a27a2b91 ~]# su hdfs
[hdfs@9321a27a2b91 root]$ cd /opt/hadoop/
[hdfs@9321a27a2b91 hadoop]$ tar -zxvf hadoop-2.7.3.tar.gz
2.3.ntp服务
yum -y install ntp
#本子网内主机都可以同步
restrict 172.17.0.0 mask 255.255.0.0 nomodify
#优先时间服务器
server 172.17.0.10 prefer
#日志文件位置
logfile /var/log/ntp.log
[root@9321a27a2b91 hadoop]# service ntpd start
Starting ntpd:[ OK ]
[root@9321a27a2b91 hadoop]# service ntpd status
ntpd dead but pid file exists
3Apr11:20:08 ntpd[732]: ntp_io: estimated max descriptors:65536, initial socket boundary:16
3Apr11:20:08 ntpd[732]:Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
3Apr11:20:08 ntpd[732]:Listen and drop on 1 v6wildcard :: UDP 123
3Apr11:20:08 ntpd[732]:Listen normally on 2 lo 127.0.0.1 UDP 123
3Apr11:20:08 ntpd[732]:Listen normally on 3 eth0 172.17.0.10 UDP 123
3Apr11:20:08 ntpd[732]:Listen normally on 4 lo ::1 UDP 123
3Apr11:20:08 ntpd[732]:Listen normally on 5 eth0 fe80::42:acff:fe11:a UDP 123
3Apr11:20:08 ntpd[732]:Listening on routing socket on fd #22 for interface updates
3Apr11:20:08 ntpd[732]:0.0.0.0 c016 06 restart
3Apr11:20:08 ntpd[732]: ntp_adjtime() failed:Operation not permitted
3Apr11:20:08 ntpd[732]:0.0.0.0 c012 02 freq_set kernel 0.000 PPM
3Apr11:20:08 ntpd[732]:0.0.0.0 c011 01 freq_not_set
3Apr11:20:08 ntpd[732]: cap_set_proc() failed to drop root privileges:Operation not permitted
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"
echo "# Drop root to id 'ntp:ntp' by default.">/etc/sysconfig/ntpd
echo "#OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g" ">>/etc/sysconfig/ntpd
[root@9321a27a2b91 hadoop]# service ntpd start
Starting ntpd:[ OK ]
[root@9321a27a2b91 hadoop]# service ntpd status
ntpd (pid 796) is running..
server 172.17.0.10 prefer
2.4.mysql 数据库
三.安装hadoop及其组件
3.1 安装HDFS及YARN
3.1.1 设置环境变量.bash_profile
su hdfs
- vi ~.bash_profile
JAVA_HOME=/usr/local/java/jdk1.8.0_121
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASS
HADOOP_HOME=/opt/hadoop/hadoop-2.7.3
HADOOP_PREFIX=/opt/hadoop/hadoop-2.7.3
HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
HADOOP_YARN_HOME=$HADOOP_HOME
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_HOME/jre/lib/amd64/server
PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export PATH
su hdfs
scp -r hdfs@hadoop1:/home/hdfs/.bash_profile ~
3.1.2 设置hadoop启动的环境配置文件xxx-evn.sh
3.1.2.1 hadoop-env.s
export JAVA_HOME=
/usr/local/java/jdk1.8.0_121- export HADOOP_HOME=/opt/hadoop/hadoop-2.7.3
#hadoop进程的最大heapsize包括namenode/datanode/ secondarynamenode等,默认1000M
#export HADOOP_HEAPSIZE=
#namenode的初始heapsize,默认取上面的值,按需要分配
#export HADOOP_NAMENODE_INIT_HEAPSIZE=""
#JVM启动参数,默认为空
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
#还可以单独配置各个组件的内存:
export HADOOP_NAMENODE_OPTS=
export HADOOP_DATANODE_OPTS
export HADOOP_SECONDARYNAMENODE_OPTS
#设置hadoop日志,默认是$HADOOP_HOME/log
export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER
3.1.2.2 yarn-env.sh
export JAVA_HOME=
/usr/local/java/jdk1.8.0_121JAVA_HEAP_MAX=-Xmx1000m
# YARN_HEAPSIZE=1000 #yarn 守护进程heapsize
#export YARN_RESOURCEMANAGER_HEAPSIZE=1000 #单独设置RESOURCEMANAGER的HEAPSIZE
#export YARN_TIMELINESERVER_HEAPSIZE=1000 #单独设置TIMELINESERVER(jobhistoryServer)的HEAPSIZE
#export YARN_RESOURCEMANAGER_OPTS= #单独设置RESOURCEMANAGER的JVM选项
#export YARN_NODEMANAGER_HEAPSIZE=1000 #单独设置NODEMANAGER的HEAPSIZE
#export YARN_NODEMANAGER_OPTS= #单独设置NODEMANAGER的JVM选项
3.1.3 修改hadoop配置文件
3.1.3.1 修改core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop1:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop/hadoop-2.7.3/tmp</value>
<description>默认值/tmp/hadoop-${user.name},修改成持久化的目录</description>
</property>
</configuration>
mkdir ${HADOOP_HOME}/tmp
3.1.3.2 hdfs-site.xm
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
- <description>数据块的备份数量</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/hadoop/hadoop-2.7.3/namenodedir</value>
- <description>保存namenode元数据的目录,要自己创建</description>
</property>
<property>
<name>dfs.blocksize</name>
<value>134217728</value>
<description>数据块大小,128M</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/opt/hadoop/hadoop-2.7.3/datadir</value>
</property>
</configuration>
mkdir ${HADOOP_HOME}/datadir
mkdir
${HADOOP_HOME}/namenodedir
3.1.3.3 mapred-site.xml
Parameter | Value | Notes |
---|---|---|
mapreduce.framework.name | yarn | Execution framework set to Hadoop YARN. MR任务执行框架 |
mapreduce.map.memory.mb | 1536 | Larger resource limit for maps. map内存上限 |
mapreduce.map.java.opts | -Xmx1024M | Larger heap-size for child jvms of maps. map的子进程虚拟机heapsize |
mapreduce.reduce.memory.mb | 3072 | Larger resource limit for reduces. redouce任务内存上限 |
mapreduce.reduce.java.opts | -Xmx2560M | Larger heap-size for child jvms of reduces. redouce的子进程虚拟机heapsize |
mapreduce.task.io.sort.mb | 512 | Higher memory-limit while sorting data for efficiency. 排序内存 |
mapreduce.task.io.sort.factor | 100 | More streams merged at once while sorting files. 排序因子 |
mapreduce.reduce.shuffle.parallelcopies | 50 | Higher number of parallel copies run by reduces to fetch outputs from very large number of maps. 并行数 |
Parameter | Value | Notes |
---|---|---|
mapreduce.jobhistory.address | MapReduce JobHistory Server host:port | Default port is 10020. jobhistory地址:主机+端口 |
mapreduce.jobhistory.webapp.address | MapReduce JobHistory Server Web UI host:port | Default port is 19888. jobhistory web端口 |
mapreduce.jobhistory.intermediate-done-dir | /mr-history/tmp | Directory where history files are written by MapReduce jobs. |
mapreduce.jobhistory.done-dir | /mr-history/done | Directory where history files are managed by the MR JobHistory Server. |
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>使用yarn来管理mr</description>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop2</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop2</value>
</property>
<property>
<name>mapreduce.jobhistory.intermediate-done-dir</name>
<value>/opt/hadoop/hadoop-2.7.3/mrHtmp</value>
</property>
<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/opt/hadoop/hadoop-2.7.3/mrhHdone</value>
</property>
</configuration>
3.1.3.4 yarn-site.xml
Parameter | Value | Notes |
---|---|---|
yarn.resourcemanager.address | ResourceManager host:port for clients to submit jobs. | host:port If set, overrides the hostname set in yarn.resourcemanager.hostname. resourcemanager的地址,格式 主机:端口 |
yarn.resourcemanager.scheduler.address | ResourceManager host:port for ApplicationMasters to talk to Scheduler to obtain resources. | host:port If set, overrides the hostname set in yarn.resourcemanager.hostname. 调度器地址 ,覆盖yarn.resourcemanager.hostname |
yarn.resourcemanager.resource-tracker.address | ResourceManager host:port for NodeManagers. | host:port If set, overrides the hostname set in yarn.resourcemanager.hostname. datanode像rm报告的端口,
覆盖 yarn.resourcemanager.hostname |
yarn.resourcemanager.admin.address | ResourceManager host:port for administrative commands. | host:port If set, overrides the hostname set in yarn.resourcemanager.hostname. RM管理地址,覆盖 yarn.resourcemanager.hostname |
yarn.resourcemanager.webapp.address | ResourceManager web-ui host:port. | host:port If set, overrides the hostname set in yarn.resourcemanager.hostname. RM web地址,有默认值 |
yarn.resourcemanager.hostname | ResourceManager host. | host Single hostname that can be set in place of setting allyarn.resourcemanager*address resources. Results in default ports for ResourceManager components. RM的主机,使用默认端口 |
yarn.resourcemanager.scheduler.class | ResourceManager Scheduler class. | CapacityScheduler (recommended), FairScheduler (also recommended), or FifoScheduler |
yarn.scheduler.minimum-allocation-mb | Minimum limit of memory to allocate to each container request at the Resource Manager. | In MBs 最小容器内存(每个container最小内存) |
yarn.scheduler.maximum-allocation-mb | Maximum limit of memory to allocate to each container request at the Resource Manager. | In MBs 最大容器内存(每个container最大内存) |
yarn.resourcemanager.nodes.include-path /yarn.resourcemanager.nodes.exclude-path | List of permitted/excluded NodeManagers. | If necessary, use these files to control the list of allowable NodeManagers. 哪些datanode可以被RM管理 |
yarn.nodemanager.resource.memory-mb | Resource i.e. available physical memory, in MB, for given NodeManager | Defines total available resources on the NodeManager to be made available to running containers Yarn在NodeManager最大内存 |
yarn.nodemanager.vmem-pmem-ratio | Maximum ratio by which virtual memory usage of tasks may exceed physical memory | The virtual memory usage of each task may exceed its physical memory limit by this ratio. The total amount of virtual memory used by tasks on the NodeManager may exceed its physical memory usage by this ratio. 任务使用的虚拟内存超过被允许的推理内存的比率,超过则kill掉 |
yarn.nodemanager.local-dirs | Comma-separated list of paths on the local filesystem where intermediate data is written. | Multiple paths help spread disk i/o. datamanager的本地目录 |
yarn.nodemanager.log-dirs | Comma-separated list of paths on the local filesystem where logs are written. | Multiple paths help spread disk i/o. datamanager日志目录 |
yarn.nodemanager.log.retain-seconds | 10800 | Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled. |
yarn.nodemanager.remote-app-log-dir | /logs | HDFS directory where the application logs are moved on application completion. Need to set appropriate permissions. Only applicable if log-aggregation is enabled. |
yarn.nodemanager.remote-app-log-dir-suffix | logs | Suffix appended to the remote log dir. Logs will be aggregated to ${yarn.nodemanager.remote-app-log-dir}/${user}/${thisParam} Only applicable if log-aggregation is enabled. |
yarn.nodemanager.aux-services | mapreduce_shuffle | Shuffle service that needs to be set for Map Reduce applications. shuffle服务类型 |
yarn.acl.enable | true /false | Enable ACLs? Defaults to false. 是否开启ACL |
yarn.admin.acl | Admin ACL | ACL to set admins on the cluster. ACLs are of for comma-separated-usersspacecomma-separated-groups. Defaults to special value of * which meansanyone. Special value of just space means no one has access. ACL用户,用,分隔 如root,yarn |
yarn.log-aggregation-enable | false | Configuration to enable or disable log aggregation 启用日志聚集.日志聚焦到一个节点 |
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop1</value>
<description>设置resourcemanager节点</description>
</property>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>设置nodemanager的aux服务</description>
</property>
</configuration>
3.1.4 设置slaves文件
vi $HADOOP_HOME/et/hadoop/slaves
hadoop3
hadoop4
hadoop5
hadoop6
3.1.5 启动HADOOP
3.1.5.1 把hadoop配置复制到其它机器上
mkdir /opt/hadoop
chown hdfs:hadoop /opt/hadoop
su hdfs
scp -r hdfs@hadoop1:/opt/hadoop/hadoop-2.7.3 /opt/hadoop
$HADOOP_HOME/bin/hdfs namenode -format
3.1.5.3 启动hdfs
[hdfs@9321a27a2b91 hadoop-2.7.3]$ start-dfs.sh
Starting namenodes on [hadoop1]
hadoop1: starting namenode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-namenode-9321a27a2b91.out
hadoop3: starting datanode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-datanode-f89eaf2a2548.out
hadoop4: starting datanode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-datanode-28620eee1426.out
hadoop5: starting datanode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-datanode-ae1f06bd04c8.out
hadoop6: starting datanode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-datanode-11c433a003b6.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-secondarynamenode-9321a27a2b91.out
[hdfs@9321a27a2b91 hadoop]$ jps
11105 Jps
10981 SecondaryNameNode
10777 NameNode
[hdfs@9321a27a2b91 hadoop-2.7.3]$ hdfs dfs -put NOTICE.txt /
[hdfs@9321a27a2b91 hadoop-2.7.3]$ hdfs dfs -ls /
Found 1 items
-rw-r--r-- 3 hdfs supergroup 14978 2017-04-03 19:15 /NOTICE.txt
[root@9321a27a2b91 hdfs]# curl hadoop1:50070
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
................

3.1.5.4 启动yarn
[hdfs@9321a27a2b91 hadoop]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop/hadoop-2.7.3/logs/yarn-hdfs-resourcemanager-9321a27a2b91.out
hadoop5: starting nodemanager, logging to /opt/hadoop/hadoop-2.7.3/logs/yarn-hdfs-nodemanager-ae1f06bd04c8.out
hadoop6: starting nodemanager, logging to /opt/hadoop/hadoop-2.7.3/logs/yarn-hdfs-nodemanager-11c433a003b6.out
hadoop3: starting nodemanager, logging to /opt/hadoop/hadoop-2.7.3/logs/yarn-hdfs-nodemanager-f89eaf2a2548.out
hadoop4: starting nodemanager, logging to /opt/hadoop/hadoop-2.7.3/logs/yarn-hdfs-nodemanager-28620eee1426.out
[hdfs@9321a27a2b91 hadoop]$ jps
11105 Jps
10981 SecondaryNameNode
10777 NameNode
10383 ResourceManager


[hdfs@9321a27a2b91 hadoop-2.7.3]$ bin/hdfs dfs -mkdir /user
[hdfs@9321a27a2b91 hadoop-2.7.3]$ bin/hdfs dfs -mkdir /user/hdfs
[hdfs@9321a27a2b91 hadoop-2.7.3]$ bin/hdfs dfs -put etc/hadoop input
...............
17/04/12 12:38:24 INFO mapreduce.JobSubmitter: number of splits:30
17/04/12 12:38:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1491968887469_0003
17/04/12 12:38:24 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/hdfs/.staging/job_1491968887469_0003
java.lang.IllegalArgumentException: Does not contain a valid host:port authority: hadoop2
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:213)
........................
[hdfs@9321a27a2b91 hadoop]$ start-dfs.sh
Starting namenodes on [9321a27a2b91]
The authenticity of host '9321a27a2b91 (172.17.0.10)' can't be established.
RSA key fingerprint is 60:0c:61:73:2c:49:ef:e3:f7:61:c9:27:93:5a:1d:c7.
Are you sure you want to continue connecting (yes/no)?
3.1.5.6 各个节点上启动hdfs守护进程
[hdfs@11c433a003b6 hadoop-2.7.3]$ $HADOOP_HOME/sbin/hadoop-daemons.sh start datanode
The authenticity of host '28620eee1426 (172.17.0.7)' can't be established.
RSA key fingerprint is 60:0c:61:73:2c:49:ef:e3:f7:61:c9:27:93:5a:1d:c7.
Are you sure you want to continue connecting (yes/no)? The authenticity of host '11c433a003b6 (172.17.0.5)' can't be established.
RSA key fingerprint is 60:0c:61:73:2c:49:ef:e3:f7:61:c9:27:93:5a:1d:c7.
Are you sure you want to continue connecting (yes/no)? The authenticity of host 'ae1f06bd04c8 (172.17.0.6)' can't be established.
RSA key fingerprint is 60:0c:61:73:2c:49:ef:e3:f7:61:c9:27:93:5a:1d:c7.
Are you sure you want to continue connecting (yes/no)? f89eaf2a2548: datanode running as process 5764. Stop it first.
[hdfs@11c433a003b6 hadoop-2.7.3]$ $HADOOP_HOME/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs start datanode
cat: /opt/hadoop/hadoop-2.7.3/etc/hadoop/slaves: No such file or directory
usage="Usage: hadoop-daemons.sh [--config confdir] [--hosts hostlistfile] [start|stop] command args..."
# if no args specified, show usage
if[ $# -le 1 ]; then
echo $usage
exit 1
fi
bin=`dirname "${BASH_SOURCE-$0}"`
bin=`cd "$bin"; pwd`
DEFAULT_LIBEXEC_DIR="$bin"/../libexec
HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}
. $HADOOP_LIBEXEC_DIR/hadoop-config.sh
exec "$bin/slaves.sh"--config $HADOOP_CONF_DIR cd "$HADOOP_PREFIX" \; "$bin/hadoop-daemon.sh"--config $HADOOP_CONF_DIR "$@"

[hdfs@9321a27a2b91 hadoop]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode
starting namenode, logging to /opt/hadoop/hadoop-2.7.3/logs/hadoop-hdfs-namenode-9321a27a2b91.out
以下节点启动nodemanager:
[hdfs@f89eaf2a2548 hadoop-2.7.3]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager
starting resourcemanager, logging to /opt/hadoop/hadoop-2.7.3/logs/yarn-hdfs-resourcemanager-f89eaf2a2548.out

[hdfs@9321a27a2b91 hadoop-2.7.3]$ bin/hdfs dfs -put etc/hadoop input
...............
17/04/1212:38:24 INFO mapreduce.JobSubmitter: number of splits:30
17/04/1212:38:24 INFO mapreduce.JobSubmitter:Submitting tokens for job: job_1491968887469_0003
17/04/1212:38:24 INFO mapreduce.JobSubmitter:Cleaning up the staging area /tmp/hadoop-yarn/staging/hdfs/.staging/job_1491968887469_0003
java.lang.IllegalArgumentException:Does not contain a valid host:port authority: hadoop2
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:213)
........................
<property>
<name>mapreduce.jobhistory.address</name>
<value>7c3a3c9cd595</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>7c3a3c9cd595</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>7c3a3c9cd595:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>7c3a3c9cd595:19888</value>
</property>
2017-04-0319:13:12,328 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:IOExceptionin offerService
java.io.EOFException:End of FileException between local host is:"ae1f06bd04c8/172.17.0.6"; destination host is:"hadoop1":9000;: java.io.EOFException;For more details see: http://wiki.a
pache.org/hadoop/EOFException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
四.总结
4.1.hadoop的安装步骤
4.2问题总结
HADOOP docker(一):安装hadoop实验集群(略操蛋)的更多相关文章
- 使用docker安装部署Spark集群来训练CNN(含Python实例)
使用docker安装部署Spark集群来训练CNN(含Python实例) http://blog.csdn.net/cyh_24/article/details/49683221 实验室有4台神服务器 ...
- Hadoop 2.6.4单节点集群配置
1.安装配置步骤 # wget http://download.oracle.com/otn-pub/java/jdk/8u91-b14/jdk-8u91-linux-x64.rpm # rpm -i ...
- 基于Hadoop 2.2.0的高可用性集群搭建步骤(64位)
内容概要: CentSO_64bit集群搭建, hadoop2.2(64位)编译,安装,配置以及测试步骤 新版亮点: 基于yarn计算框架和高可用性DFS的第一个稳定版本. 注1:官网只提供32位re ...
- Hadoop加zookeeper构建高可靠集群
事前准备 1.更改Linux主机名,每个人都有配置 vim /etc/sysconfig/network NETWORKING=yes HOSTNAME=hadoop-server1 2.改动IP / ...
- Hadoop及Zookeeper+HBase完全分布式集群部署
Hadoop及HBase集群部署 一. 集群环境 系统版本 虚拟机:内存 16G CPU 双核心 系统: CentOS-7 64位 系统下载地址: http://124.202.164.6/files ...
- Hadoop入门 完全分布式运行模式-集群配置
目录 集群配置 集群部署规划 配置文件说明 配置集群 群起集群 1 配置workers 2 启动集群 总结 3 集群基本测试 上传文件到集群 查看数据真实存储路径 下载 执行wordcount程序 配 ...
- [Hadoop] - Win7下提交job到集群上去
一般我们采用win开发+linux hadoop集群的方式进行开发,使用插件:hadoop-***-eclipse-plugin. 运行程序的时候,我们一般采用run as application或者 ...
- Docker安装部署es集群
Docker安装部署es集群:环境准备:已安装docker的centos服务器一台1. 拉取es版本docker pull elasticsearch:5.6.82. 新建文件夹 数据挂载目录 和 配 ...
- docker安装Elasticsearch7.6集群并设置密码
docker安装Elasticsearch7.6集群并设置密码 Elasticsearch从6.8开始, 允许免费用户使用X-Pack的安全功能, 以前安装es都是裸奔.接下来记录配置安全认证的方法. ...
随机推荐
- mysql 配置安装
1. 下载: mysql-5.7.20是解压版免安装的,mysql-5.7.20下载地址:http://dev.mysql.com/downloads/mysql/ 直接下载,无需注册和登录. ...
- IIS网站的应用程序与虚拟目录的区别及应用
IIS网站 一个网站可以新建无数个应用程序和目录 应用程序 同一域名下程序的独立开发,独立部署的最佳应用策略. 应用程序的应用场景: 1. 域名的分布 比如:www.baidu.com,对于后台,我们 ...
- 获取地图的信息到input里
在最近项目中,我接触了百度地图的API写法,对其中的代码有了一点兴趣,所以我在完成任务后,在办公室里学习了百度地图的相关引用,并申请了服务秘钥: E7PCho0sv3FdzmjC901ttP0HrS9 ...
- border-radius__边框圆角
1.四个参数: border-radius: 值1 值2 值3 值4;顺序:从左开始,顺时针顺 div{ width: 200px; height: 100px; background-color: ...
- go语言学习(一):数组/切片
学习区块链之后,发现还要学习加密,接触到的视频是:Go的crypto包,所以开始接触Go.因为和solidity有些相似,但是比solidity简单一些,就开始放松的心态去学习.每天翻着go菜鸟教程, ...
- yii学习笔记(4),获取请求数据的request组件
yii在控制器中获取请求数据需要通过request组件来完成 <?php namespace app\controllers; use yii; use yii\web\Controller; ...
- C#的哈希表Hashtable同步方法
在多线程环境的操作中对Hashtable进行操作需要进行同步控制,有两种方法,一种是由.Net自动控制:一种是在代码中自己控制. 1.使用Hashtable.Synchronized进行同步 Hash ...
- STM32F407+STemwin学习笔记之STemwin移植补充Touch
原文地址:http://www.cnblogs.com/NickQ/p/8857213.html 环境:keil5.20 STM32F407ZGT6 LCD(320*240) STemwin:S ...
- linux 下的torrent下载器qBitTorrent
BT下载利器--Qbittorrent完全攻 Ubuntu使用命令安装qBittorrent的方法 源码下载
- AtCoder Grant Contest 10.F 博弈
题意:给定一棵树,每个节点有个权值,Alice和Bob轮流进行操作,给定游戏起点指针指向节点C.不断进行下述操作. 1.将当前节点权值-1,然后将指针从该节点移动到相邻节点,出现一方不能移动(即指针指 ...