(三)Spark-Hadoop集群搭建-Java&Python版Spark
Spark-Hadoop集群搭建
视频教程:
1、优酷
2、YouTube
配置java
启动ftp
[root@master ~]# /etc/init.d/vsftpd restart
关闭 vsftpd: [失败]
为 vsftpd 启动 vsftpd: [确定]
默认情况下 root不允许使用ftp
[root@master vsftpd]# pwd
/etc/vsftpd
[root@master vsftpd]# ls
ftpusers user_list
将这两个文件中的root注释掉。
然后重启ftp
在两台机器上创建文件夹
[root@master ~]# mkdir installer
[root@master ~]#
上传jdk
E:\开发工具
jdk-7u79-linux-i586.rpm
到installer目录
Master机器
[root@master installer]# rpm -ivh jdk-7u79-linux-i586.rpm
Preparing... ########################################### [100%]
1:jdk ########################################### [100%]
Unpacking JAR files...
rt.jar...
jsse.jar...
charsets.jar...
tools.jar...
localedata.jar...
jfxrt.jar...
plugin.jar...
javaws.jar...
Deploy.jar...
Slave机器
[root@slave installer]# rpm -ivh jdk-7u79-linux-i586.rpm
Preparing... ########################################### [100%]
1:jdk ########################################### [100%]
Unpacking JAR files...
rt.jar...
jsse.jar...
charsets.jar...
tools.jar...
localedata.jar...
jfxrt.jar...
plugin.jar...
javaws.jar...
deploy.jar...
添加hadoop用户
[root@master ~]# useradd hadoop
[root@master ~]# passwd hadoop
Changing password for user hadoop.
New UNIX password:
BAD PASSWORD: it is based on a dictionary word
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
[root@slave ~]# useradd hadoop
[root@slave ~]# passwd hadoop
Changing password for user hadoop.
New UNIX password:
BAD PASSWORD: it is based on a dictionary word
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
并且在slave1和slave2使用hadoop用户创建installer目录
[root@master ~]# su - hadoop
[hadoop@master ~]$ mkdir installer
[root@slave ~]# su - hadoop
[hadoop@slave ~]$ mkdir installer
配置ssh等效性
一路回车键
[hadoop@master ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
32:da:01:64:6a:48:4c:f0:58:4f:60:2e:ea:18:a0:5d hadoop@master
[hadoop@slave ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
80:49:e7:58:18:0b:9c:1f:ee:d8:96:5a:c5:50:a5:bb hadoop@slave
Master
[hadoop@master ~]$ cd .ssh/
[hadoop@master .ssh]$ ls
id_rsa id_rsa.pub
[hadoop@master .ssh]$ cat id_rsa.pub > authorized_keys
[hadoop@master .ssh]$ scp authorized_keys slave:~/.ssh/
The authenticity of host 'slave (192.168.3.101)' can't be established.
RSA key fingerprint is de:5a:2e:67:62:a4:27:2c:d7:de:85:eb:c7:6c:93:c8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave,192.168.3.101' (RSA) to the list of known hosts.
hadoop@slave's password:
authorized_keys
Slave
[hadoop@slave ~]$ cd .ssh/
[hadoop@slave .ssh]$ ls
authorized_keys id_rsa id_rsa.pub
[hadoop@slave .ssh]$ cat id_rsa.pub >> authorized_keys
[hadoop@slave .ssh]$ scp authorized_keys master:~/.ssh/
The authenticity of host 'master (192.168.3.100)' can't be established.
RSA key fingerprint is de:5a:2e:67:62:a4:27:2c:d7:de:85:eb:c7:6c:93:c8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,192.168.3.100' (RSA) to the list of known hosts.
hadoop@master's password:
authorized_keys
最后
[hadoop@master .ssh]$ chmod 600 authorized_keys
[hadoop@slave .ssh]$ chmod 600 authorized_keys
到这里配置完毕,可以直接使用
[hadoop@master ~]$ ssh slave
[hadoop@slave ~]$ ssh master
配置hadoop
解压缩
[hadoop@master installer]$ tar -zxvf hadoop-2.6.0.tar.gz
[hadoop@master installer]$ mv hadoop-2.6.0 hadoop2
[hadoop@master installer]$ ls
hadoop2 hadoop-2.6.0.tar.gz
配置环境变量
[hadoop@master ~]$ vim .bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# User specific aliases and functions
export JAVA_HOME=/usr/java/jdk1.7.0_79
export HADOOP_HOME=/home/hadoop/installer/hadoop2
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export CLASSPATH=$CLASSPATH:$HADOOP_HOME/lib:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
让bashrc生效
[hadoop@master ~]$ . .bashrc
[hadoop@master ~]$ scp .bashrc slave:~
.bashrc 100% 341 0.3KB/s 00:00
配置hadoop文件
[hadoop@master hadoop]$ vim hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_79
#export HADOOP_HEAPSIZE=
export HADOOP_HEAPSIZE=100
[hadoop@master hadoop]$ vim yarn-env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_79
JAVA_HEAP_MAX=-Xmx300m
YARN_HEAPSIZE=100
[hadoop@master hadoop]$ vim mapred-env.sh
export JAVA_HOME=/usr/java/jdk1.6.0_27
export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=100
[hadoop@master hadoop]$ vim slaves
slave
[hadoop@master hadoop]$ vim core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop/tmp</value>
</property>
</configuration>
创建tmp目录
[hadoop@master ~]$ mkdir tmp
[hadoop@slave ~]$ mkdir tmp
[hadoop@master hadoop]$ vim hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:50090</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/data/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/data/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
创建目录
[hadoop@master ~]$ mkdir -p data/dfs/data
[hadoop@master ~]$ mkdir -p data/dfs/name
[hadoop@slave ~]$ mkdir -p data/dfs/data
[hadoop@slave ~]$ mkdir -p data/dfs/name
[hadoop@master hadoop]$ cp mapred-site.xml.template mapred-site.xml
[hadoop@master hadoop]$ vim mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>300</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>300</value>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>100</value>
</property>
</configuration>
[hadoop@master hadoop]$ vim yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<description>The hostname of the RM.</description>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<description>The address of the applications manager interface in the RM.</description>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
<description>The address of the scheduler interface.</description>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
<description>The http address of the RM web application.</description>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<description>The https adddress of the RM web application.</description>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
<description>The address of the RM admin interface.</description>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
<description>The minimum allocation for every container request at the RM,
in MBs. Memory requests lower than this won't take effect,
and the specified value will get allocated at minimum.</description>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>300</value>
</property>
<property>
<description>The maximum allocation for every container request at the RM,
in MBs. Memory requests higher than this won't take effect,
and will get capped to this value.</description>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>1024</value>
</property>
<property>
<description>Ratio between virtual memory to physical memory when
setting memory limits for containers. Container allocations are
expressed in terms of physical memory, and virtual memory usage
is allowed to exceed this allocation by this ratio.
</description>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>6.1</value>
</property>
</configuration>
[hadoop@master installer]$ cd installer/
[hadoop@master installer]$ scp -r hadoop2 slave:~/installer/
Slave机器上执行
[hadoop@slave ~]$ . .bashrc
格式化
[hadoop@master ~]$ hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
16/11/03 19:42:11 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = master/192.168.3.100
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.6.0
STARTUP_MSG: classpath = /home/hadoop/installer/hadoop2/etc/hadoop:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/avro-1.7.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/curator-client-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/junit-4.11.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/htrace-core-3.0.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-el-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/hadoop-annotations-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/curator-framework-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/hadoop-auth-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/xz-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jetty-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/hadoop-common-2.6.0-tests.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/hadoop-nfs-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/hadoop-common-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/hadoop-hdfs-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/hadoop-hdfs-2.6.0-tests.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jline-0.9.94.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-registry-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-client-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-api-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0.jar:/home/hadoop/installer/hadoop2/contrib/capacity-scheduler/*.jar:/home/hadoop/installer/hadoop2/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1; compiled by 'jenkins' on 2014-11-13T21:10Z
STARTUP_MSG: java = 1.7.0_79
************************************************************/
16/11/03 19:42:11 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
16/11/03 19:42:11 INFO namenode.NameNode: createNameNode [-format]
16/11/03 19:42:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/11/03 19:42:11 WARN common.Util: Path /home/hadoop/data/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
16/11/03 19:42:11 WARN common.Util: Path /home/hadoop/data/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
Formatting using clusterid: CID-9c79097a-7f44-48cc-9308-ea9c487a685e
16/11/03 19:42:11 INFO namenode.FSNamesystem: No KeyProvider found.
16/11/03 19:42:11 INFO namenode.FSNamesystem: fsLock is fair:true
16/11/03 19:42:11 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
16/11/03 19:42:11 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
16/11/03 19:42:11 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
16/11/03 19:42:11 INFO blockmanagement.BlockManager: The block deletion will start around 2016 十一月 03 19:42:11
16/11/03 19:42:11 INFO util.GSet: Computing capacity for map BlocksMap
16/11/03 19:42:11 INFO util.GSet: VM type = 32-bit
16/11/03 19:42:11 INFO util.GSet: 2.0% max memory 96.7 MB = 1.9 MB
16/11/03 19:42:11 INFO util.GSet: capacity = 2^19 = 524288 entries
16/11/03 19:42:11 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
16/11/03 19:42:11 INFO blockmanagement.BlockManager: defaultReplication = 1
16/11/03 19:42:11 INFO blockmanagement.BlockManager: maxReplication = 512
16/11/03 19:42:11 INFO blockmanagement.BlockManager: minReplication = 1
16/11/03 19:42:11 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
16/11/03 19:42:11 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
16/11/03 19:42:11 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
16/11/03 19:42:11 INFO blockmanagement.BlockManager: encryptDataTransfer = false
16/11/03 19:42:11 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
16/11/03 19:42:12 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE)
16/11/03 19:42:12 INFO namenode.FSNamesystem: supergroup = supergroup
16/11/03 19:42:12 INFO namenode.FSNamesystem: isPermissionEnabled = true
16/11/03 19:42:12 INFO namenode.FSNamesystem: HA Enabled: false
16/11/03 19:42:12 INFO namenode.FSNamesystem: Append Enabled: true
16/11/03 19:42:12 INFO util.GSet: Computing capacity for map INodeMap
16/11/03 19:42:12 INFO util.GSet: VM type = 32-bit
16/11/03 19:42:12 INFO util.GSet: 1.0% max memory 96.7 MB = 990.1 KB
16/11/03 19:42:12 INFO util.GSet: capacity = 2^18 = 262144 entries
16/11/03 19:42:12 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/11/03 19:42:12 INFO util.GSet: Computing capacity for map cachedBlocks
16/11/03 19:42:12 INFO util.GSet: VM type = 32-bit
16/11/03 19:42:12 INFO util.GSet: 0.25% max memory 96.7 MB = 247.5 KB
16/11/03 19:42:12 INFO util.GSet: capacity = 2^16 = 65536 entries
16/11/03 19:42:12 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/11/03 19:42:12 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
16/11/03 19:42:12 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
16/11/03 19:42:12 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/11/03 19:42:12 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
16/11/03 19:42:12 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/11/03 19:42:12 INFO util.GSet: VM type = 32-bit
16/11/03 19:42:12 INFO util.GSet: 0.029999999329447746% max memory 96.7 MB = 29.7 KB
16/11/03 19:42:12 INFO util.GSet: capacity = 2^13 = 8192 entries
16/11/03 19:42:12 INFO namenode.NNConf: ACLs enabled? false
16/11/03 19:42:12 INFO namenode.NNConf: XAttrs enabled? true
16/11/03 19:42:12 INFO namenode.NNConf: Maximum size of an xattr: 16384
16/11/03 19:42:12 INFO namenode.FSImage: Allocated new BlockPoolId: BP-942661971-192.168.3.100-1478173332298
16/11/03 19:42:12 INFO common.Storage: Storage directory /home/hadoop/data/dfs/name has been successfully formatted.
16/11/03 19:42:12 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/11/03 19:42:12 INFO util.ExitUtil: Exiting with status 0
16/11/03 19:42:12 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.3.100
************************************************************/
没有任何错误,格式化成功。
[hadoop@master ~]$ start-dfs.sh
[hadoop@master ~]$ start-yarn.sh
[hadoop@master ~]$ jps
17769 ResourceManager
18020 Jps
17443 NameNode
17631 SecondaryNameNode
[hadoop@slave ~]$ jps
13538 Jps
13297 DataNode
13408 NodeManager
[hadoop@master mapreduce]$ ls
hadoop-mapreduce-client-app-2.6.0.jar hadoop-mapreduce-client-hs-plugins-2.6.0.jar hadoop-mapreduce-examples-2.6.0.jar
hadoop-mapreduce-client-common-2.6.0.jar hadoop-mapreduce-client-jobclient-2.6.0.jar lib
hadoop-mapreduce-client-core-2.6.0.jar hadoop-mapreduce-client-jobclient-2.6.0-tests.jar lib-examples
hadoop-mapreduce-client-hs-2.6.0.jar hadoop-mapreduce-client-shuffle-2.6.0.jar sources
[hadoop@master mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount /data/ /output
16/11/03 20:01:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/11/03 20:01:45 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.3.100:8032
16/11/03 20:01:49 INFO input.FileInputFormat: Total input paths to process : 2
16/11/03 20:01:49 INFO mapreduce.JobSubmitter: number of splits:2
16/11/03 20:01:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1478174305039_0001
16/11/03 20:01:54 INFO impl.YarnClientImpl: Submitted application application_1478174305039_0001
16/11/03 20:01:55 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1478174305039_0001/
16/11/03 20:01:55 INFO mapreduce.Job: Running job: job_1478174305039_0001
16/11/03 20:02:06 INFO mapreduce.Job: Job job_1478174305039_0001 running in uber mode : false
16/11/03 20:02:06 INFO mapreduce.Job: map 0% reduce 0%
16/11/03 20:02:06 INFO mapreduce.Job: Job job_1478174305039_0001 failed with state FAILED due to: Application application_1478174305039_0001 failed 2 times due to AM Container for appattempt_1478174305039_0001_000002 exited with exitCode: -103
For more detailed output, check application tracking page:http://master:8088/proxy/application_1478174305039_0001/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=11781,containerID=container_1478174305039_0001_02_000001] is running beyond virtual memory limits. Current usage: 41.4 MB of 100 MB physical memory used; 1.2 GB of 210.0 MB virtual memory used. Killing container.
Dump of the process-tree for container_1478174305039_0001_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 11781 11780 11781 11781 (bash) 0 0 65404928 274 /bin/bash -c /usr/java/jdk1.7.0_79/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop/installer/hadoop2/logs/userlogs/application_1478174305039_0001/container_1478174305039_0001_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/home/hadoop/installer/hadoop2/logs/userlogs/application_1478174305039_0001/container_1478174305039_0001_02_000001/stdout 2>/home/hadoop/installer/hadoop2/logs/userlogs/application_1478174305039_0001/container_1478174305039_0001_02_000001/stderr
|- 11790 11781 11781 11781 (java) 75 38 1248215040 10317 /usr/java/jdk1.7.0_79/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop/installer/hadoop2/logs/userlogs/application_1478174305039_0001/container_1478174305039_0001_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
16/11/03 20:02:06 INFO mapreduce.Job: Counters: 0
[hadoop@master hadoop2]$ vim ./share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
<property>
<description>Ratio between virtual memory to physical memory when
setting memory limits for containers. Container allocations are
expressed in terms of physical memory, and virtual memory usage
is allowed to exceed this allocation by this ratio.
</description>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.1</value> 这里的2.1是一个系数
</property>
标红错误说 物理内存乘以2.1 得到210M,因为一开始我们设置了物理内存是100M
调整方法要么把2.1调整大,要么调整大物理内存
修改之前最好先停掉yarn
[hadoop@master hadoop]$ vim yarn-site.xml
<property>
<description>The minimum allocation for every container request at the RM,
in MBs. Memory requests lower than this won't take effect,
and the specified value will get allocated at minimum.</description>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>200</value>
</property>
<property>
<description>Ratio between virtual memory to physical memory when
setting memory limits for containers. Container allocations are
expressed in terms of physical memory, and virtual memory usage
is allowed to exceed this allocation by this ratio.
</description>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>6.1</value>
</property>
[hadoop@master hadoop]$ pwd
/home/hadoop/install/hadoop2/etc/hadoop
[hadoop@master hadoop]$ scp yarn-site.xml slave1:/home/hadoop/hadoop2/etc/hadoop
yarn-site.xml 100% 3165 3.1KB/s 00:00
[hadoop@master hadoop]$ scp yarn-site.xml slave2:/home/hadoop/hadoop2/etc/hadoop
yarn-site.xml 100% 3165 3.1KB/s 00:00
可以不用停止hdfs服务
继续测试
[hadoop@master hadoop]$ start-yarn.sh
[hadoop@master mapreduce]$ jps
16511 Jps
14927 NameNode
15112 SecondaryNameNode
[hadoop@master mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount /data/ /output
16/11/03 20:10:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/11/03 20:10:01 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.3.100:8032
16/11/03 20:10:02 INFO input.FileInputFormat: Total input paths to process : 2
16/11/03 20:10:02 INFO mapreduce.JobSubmitter: number of splits:2
16/11/03 20:10:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1478174985535_0001
16/11/03 20:10:03 INFO impl.YarnClientImpl: Submitted application application_1478174985535_0001
16/11/03 20:10:03 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1478174985535_0001/
16/11/03 20:10:03 INFO mapreduce.Job: Running job: job_1478174985535_0001
16/11/03 20:10:13 INFO mapreduce.Job: Job job_1478174985535_0001 running in uber mode : false
16/11/03 20:10:13 INFO mapreduce.Job: map 0% reduce 0%
16/11/03 20:10:54 INFO mapreduce.Job: map 100% reduce 0%
16/11/03 20:11:01 INFO mapreduce.Job: map 100% reduce 100%
16/11/03 20:11:02 INFO mapreduce.Job: Job job_1478174985535_0001 completed successfully
16/11/03 20:11:02 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=62
FILE: Number of bytes written=316503
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=226
HDFS: Number of bytes written=30
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=75720
Total time spent by all reduces in occupied slots (ms)=3903
Total time spent by all map tasks (ms)=75720
Total time spent by all reduce tasks (ms)=3903
Total vcore-seconds taken by all map tasks=75720
Total vcore-seconds taken by all reduce tasks=3903
Total megabyte-seconds taken by all map tasks=22716000
Total megabyte-seconds taken by all reduce tasks=1170900
Map-Reduce Framework
Map input records=4
Map output records=7
Map output bytes=66
Map output materialized bytes=68
Input split bytes=188
Combine input records=7
Combine output records=5
Reduce input groups=4
Reduce shuffle bytes=68
Reduce input records=5
Reduce output records=4
Spilled Records=10
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=1281
CPU time spent (ms)=9240
Physical memory (bytes) snapshot=394346496
Virtual memory (bytes) snapshot=1360240640
Total committed heap usage (bytes)=257433600
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=38
File Output Format Counters
Bytes Written=30
[hadoop@master ~]$ hadoop dfs -cat /output/*
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
16/11/03 20:14:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
123 2
hadoop 2
java 2
spark 1
这样表示hadoop安装成功
(三)Spark-Hadoop集群搭建-Java&Python版Spark的更多相关文章
- (四)Spark集群搭建-Java&Python版Spark
Spark集群搭建 视频教程 1.优酷 2.YouTube 安装scala环境 下载地址http://www.scala-lang.org/download/ 上传scala-2.10.5.tgz到m ...
- 三节点Hadoop集群搭建
1. 基础环境搭建 新建3个CentOS6.5操作系统的虚拟机,命名(可自定)为masternode.slavenode1和slavenode2.该过程参考上一篇博文CentOS6.5安装配置详解 2 ...
- Hadoop 集群搭建和维护文档
一.前言 -- 基础环境准备 节点名称 IP NN DN JNN ZKFC ZK RM NM Master Worker master1 192.168.8.106 * * * * * * maste ...
- Hadoop集群搭建安装过程(三)(图文详解---尽情点击!!!)
Hadoop集群搭建安装过程(三)(图文详解---尽情点击!!!) 一.JDK的安装 安装位置都在同一位置(/usr/tools/jdk1.8.0_73) jdk的安装在克隆三台机器的时候可以提前安装 ...
- Linux环境下Hadoop集群搭建
Linux环境下Hadoop集群搭建 前言: 最近来到了武汉大学,在这里开始了我的研究生生涯.昨天通过学长们的耐心培训,了解了Hadoop,Hdfs,Hive,Hbase,MangoDB等等相关的知识 ...
- 大数据初级笔记二:Hadoop入门之Hadoop集群搭建
Hadoop集群搭建 把环境全部准备好,包括编程环境. JDK安装 版本要求: 强烈建议使用64位的JDK版本,这样的优势在于JVM的能够访问到的最大内存就不受限制,基于后期可能会学习到Spark技术 ...
- Hadoop集群搭建(完全分布式版本) VMWARE虚拟机
Hadoop集群搭建(完全分布式版本) VMWARE虚拟机 一.准备工作 三台虚拟机:master.node1.node2 时间同步 ntpdate ntp.aliyun.com 调整时区 cp /u ...
- Hadoop 集群搭建
Hadoop 集群搭建 2016-09-24 杜亦舒 目标 在3台服务器上搭建 Hadoop2.7.3 集群,然后测试验证,要能够向 HDFS 上传文件,并成功运行 mapreduce 示例程序 搭建 ...
- Hadoop集群搭建安装过程(一)(图文详解---尽情点击!!!)
Hadoop集群搭建(一)(上篇中讲到了Linux虚拟机的安装) 一.安装所需插件(以hadoop2.6.4为例,如果需要可以到官方网站进行下载:http://hadoop.apache.org) h ...
随机推荐
- 疯狂的JSONP
何为跨域?何为JSONP?JSONP技术能实现什么?是否有必要使用JSONP技术? 跨域 就是由于JavaScript同源策略的限制,使得a.com域名下的js无法操作b.com或c.a.com域名下 ...
- 工欲善其事,必先利其器 之 WPF篇: 随着开发轨迹来看高效WPF开发的工具和技巧
之前一篇<工欲善其事,必先利其器.VS2013全攻略(安装,技巧,快捷键,插件)!> 看到很多朋友回复和支持,非常感谢,尤其是一些拍砖的喷油,感谢你们的批评,受益良多. 我第一份工作便是W ...
- HTML5_04之SVG绘图
1.关于Canvas绘制图像: 问题:需要绘制多张图片时,必须等待所有图片加载完成才能开始绘制:而每张图片都是异步请求,彼此没有先后顺序,哪一张先加载完成完全无法预测: 方案: var progres ...
- 解析大型.NET ERP系统 业务逻辑设计与实现
根据近几年的制造业软件开发经验,以我开发人员的理解角度,简要说明功能(Feature)是如何设计与实现的,供参考. 因架构的不同,技术实现上会有所差异,我的经验仅限定于Windows Form程序. ...
- 小知识:C#可选参数的一个陷阱
一.背景: 互联网行业,为了降低程序维护.升级的部署风险,往往会将程序拆分成很多项目,编译成多个dll部署,这样发布的时候,只需要部署修改过的dll即可. 二.问题: 有一个函数,在很多个地方被使 ...
- git did not exit cleanly
exit code 1 1.鼠标右键 -> TortoiseGit -> Settings -> Network 2.SSH client was pointing to C:\Pr ...
- MVC5 网站开发之四 业务逻辑层的架构和基本功能
业务逻辑层在Ninesky.Core中实现,主要功能封装一些方法通过调用数据存储层,向界面层提供服务. 目录 奔跑吧,代码小哥! MVC5网站开发之一 总体概述 MVC5 网站开发之二 创建项目 ...
- ASP.NET MVC5+EF6+EasyUI 后台管理系统(59)-BLL层重构
系列目录 前言: 这应该是本系统最后一次重构,将重构BLL层和Model层.来完全取代代码生成器生成的BLL层和DAL层.完全废掉了代码生成器的DAL,BLL,MODEL层. 全自动生成增,删,改 ...
- AFNetworking 3.0 源码解读(三)之 AFURLRequestSerialization
这篇就讲到了跟请求相关的类了 关于AFNetworking 3.0 源码解读 的文章篇幅都会很长,因为不仅仅要把代码进行详细的的解释,还会大概讲解和代码相关的知识点. 上半篇: URI编码的知识 关于 ...
- 完整记录一则Oracle 11.2.0.4单实例打PSU补丁的过程
本文记录了打PSU的全过程,意在体会数据库打PSU补丁的整个过程. 1.OPatch替换为最新版本2.数据库软件应用19121551补丁程序3.数据库应用补丁4.验证PSU补丁是否应用成功 1.OPa ...