一,准备环境

  • PC基本配置如下:
处理器:Intel(R) Core(TM) i5-3230M CPU @ 2.6GHz 2.60GHz
安装内存(RAM): 12.0GB
系统类型:64位操作系统

  • 初始化四台Ubuntu-14.04_x64虚拟机,配置如下:
内存:2GB
处理器:1
硬盘:40G
网络适配器:NAT
系统:Linux ubuntu 4.4.0-142-generic #168~14.04.1-Ubuntu SMP Sat Jan 19 11:26:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

  • 修改系统时区
 ~ sudo timedatectl set-timezone "Asia/Shanghai"
  • 为方便使用建议如下配置:

安装oh-my-zsh插件;

设置VIM行号;

安装SSH插件服务;

安装vsftpd插件服务并加以配置,方便文件上传下载;

PC上安装XSHELL客户端;

PC上安装FTP客户端。

  • 需要的软件:
jdk-7u51-linux-x64.gz  链接:http://pan.baidu.com/s/1dFFT1GP  密码:cc5t

hadoop-2.2.0-x64.tar.gz https://download.csdn.net/download/wwyymmddbb/10203840
  • 在虚拟机做如下步骤:创建目录,存储工具包/home/zhangbocheng,并利用FTP上传相关软件包。



二,安装单机环境

  • 安装Java1.7.0
~ mkdir java
~ cd java
➜ java tar -xf /home/zhangbocheng/jdk-7u51-linux-x64.gz
➜ java ln -s jdk1.7.0_51 JDK
➜ java vi ~/.bashrc
➜ java vi ~/.zshrc
export JAVA_HOME=$HOME/java/java
export PATH=$JAVA_HOME/bin:$PATH :wq
➜ java source ~/.bashrc
➜ java source ~/.zshrc
➜ java java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

  • 安装Hadoop2.2.0
➜  ~ tar -xf /home/zhangbocheng/hadoop-2.2.0-x64.tar.gz
➜ ~ mv hadoop-2.2.0 hadoop2.2.0
➜ ~ mkdir hadoop2.2.0/hdfs
➜ ~ mkdir hadoop2.2.0/hdfs/name
➜ ~ mkdir hadoop2.2.0/hdfs/data
➜ ~ mkdir hadoop2.2.0/logs
➜ ~ mkdir hadoop2.2.0/tmp
  • 配置环境
➜  ~ vi .zshrc
export JAVA_HOME=$HOME/java/jdk
export CLASSPATH=$JAVA_HOME/lib/tool.jar:$JAVA_HOME/lib/dt.jar
export PATH=$JAVA_HOME/bin:$PATH
➜ ~ source .zshrc
➜ ~ echo $CLASSPATH
/home/zhangbc/java/jdk/lib/tool.jar:/home/zhangbc/java/jdk/lib/dt.jar
➜ ~ vi .zshrc
export HADOOP_HOME=$HOME/hadoop2.2.0
export PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$JAVA_HOME/bin:$PATH
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export YARN_LOG_DIR=$HADOOP_LOG_DIR
➜ ~ source .zshrc
➜ ~ hadoop version
Hadoop 2.2.0
Subversion Unknown -r Unknown
Compiled by root on 2014-09-21T22:41Z
Compiled with protoc 2.5.0
From source with checksum 79e53ce7994d1628b240f09af91e1af4
This command was run using /home/zhangbc/hadoop2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar

  • 修改Hadoop2.2.0配置文件
# 检查并修改以下三个文件中JAVA_HOME的值
➜ ~ vi hadoop2.2.0/etc/hadoop/hadoop-env.sh
➜ ~ vi hadoop2.2.0/etc/hadoop/yarn-env.sh
➜ ~ vi hadoop2.2.0/etc/hadoop/mapred-env.sh # 只需要去掉注释加以修改
export JAVA_HOME=${JAVA_HOME} # 错误
export JAVA_HOME=/home/zhangbc/java/jdk # 正确 # 添加集群的slave节点
➜ ~ vi hadoop2.2.0/etc/hadoop/slaves
# localhost
slave_1
slave_2
slave_3 ➜ ~ vi hadoop2.2.0/etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000/</value>
<descrption>设定namenode的主机名及其端口</descrption>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/zhangbc/hadoop2.2.0/tmp/hadoop-${user.name}</value>
<descrption>存储临时文件</descrption>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
</configuration> ➜ ~ vi hadoop2.2.0/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.http-address</name>
<value>master:50070</value>
<descrption>设定NameNode地址及其端口</descrption>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>slave_1:50090</value>
<descrption>设定SecondNameNode地址及其端口</descrption>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
<descrption>设定HDFS存储文件的副本个数,默认为3</descrption>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/zhangbc/hadoop2.2.0/hdfs/name</value>
<descrption>设置NameNode用来持续存储命名空间和交换日志的本地文件系统路径</descrption>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/zhangbc/hadoop2.2.0/hdfs/data</value>
<descrption>设置DataNode在本地存储文件的目录列表</descrption>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file:///home/zhangbc/hadoop2.2.0/hdfs/namesecondary</value>
<descrption>设置SecondaryNameNode存储临时镜像的本地文件系统路径,
若这是一个用逗号分隔的列表,则镜像会冗余复制到所有目录</descrption>
</property>
<property>
<name>dfs.webhdfs.enable</name>
<value>true</value>
<descrption>是否允许网页浏览HDFS文件</descrption>
</property>
<property>
<name>dfs.stream-buffer-size</name>
<value>131072</value>
<descrption>默认为4KB,作为Hadoop的缓冲区,用于Hapdoop读写HDFS的文件,
还有map的输出都用到了这个缓冲区容量,131072=128KB</descrption>
</property>
</configuration> ➜ ~ vi hadoop2.2.0/etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration> ➜ ~ vi hadoop2.2.0/etc/hadoop/yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
  • 修改主机名称(千万不要含有下划线_
➜  ~ sudo hostname master   # 只对当前状态生效
[sudo] password for zhangbc:
➜ ~ hostname
master
➜ ~ sudo vi /etc/hostname # 永久修改 master
:wq
  • 关闭防火墙
➜  ~ service ufw status
ufw start/running
➜ ~ sudo service ufw stop
[sudo] password for zhangbc:
ufw stop/waiting
➜ ~ service ufw status
ufw stop/waiting

三,克隆VM

通过 VMware Workstation工具,关闭当前虚拟机,对其克隆三台虚拟机作为从机使用。

克隆方法选择“创建完整克隆(F)”,如图所示:

四,搭建集群

  • 修改三台从机slavehost,并再重启使之生效。
➜  ~ sudo vi /etc/hostname
➜ ~ sudo vi /etc/hosts
➜ ~ sudo reboot
  • 对所有集群中的服务器进行检查,关闭防火墙并禁止掉。
➜  ~ sudo service ufw status
[sudo] password for zhangbc:
ufw start/running
➜ ~ sudo service ufw stop
ufw stop/waiting
➜ ~ sudo service ufw status
ufw stop/waiting
➜ ~ sudo ufw disable
Firewall stopped and disabled on system startup
  • 对所有集群中的服务器绑定hostnameIP
➜  ~ sudo vi /etc/hosts
192.168.71.128 master
192.168.71.129 slave_1
192.168.71.130 slave_2
192.168.71.131 slave_3
  • 对所有集群中的服务器创建SSH密钥,完成相关验证,注意保留原有的其他密钥,以备他用
➜  .ssh mv id_rsa id_rsa_git
➜ .ssh mv id_rsa.pub id_rsa_git.pub
➜ .ssh ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
# 生成authorized_keys
➜ .ssh cat id_rsa.pub >> authorized_keys
# 设置权限
➜ .ssh sudo chmod 600 authorized_keys
# ssh登录本机,并退出
➜ .ssh ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is b6:fa:8d:2b:2d:0d:e4:fd:4f:44:ed:37:3f:79:b6:ce.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.6 LTS (GNU/Linux 4.4.0-142-generic x86_64) * Documentation: https://help.ubuntu.com/ New release '16.04.6 LTS' available.
Run 'do-release-upgrade' to upgrade to it. Your Hardware Enablement Stack (HWE) is supported until April 2019.
Last login: Wed Nov 13 20:17:41 2019 from 192.168.71.1
➜ ~ exit
Connection to localhost closed.
➜ .ssh
  • 配置各个节点之间免密登录
# 将slave_1节点rsa通过ssh-copy-id分别复制到master,slave_2,slave_3
➜ ~ ssh-copy-id -i ~/.ssh/id_rsa.pub master
➜ ~ ssh-copy-id -i ~/.ssh/id_rsa.pub slave_2
➜ ~ ssh-copy-id -i ~/.ssh/id_rsa.pub slave_3
# 验证登录
➜ ~ ssh master
➜ ~ ssh slave_2
➜ ~ ssh slave_3
# 其他节点同步骤

五,Hadoop启动与测试

  • 格式化文件系统
➜  ~ hdfs namenode -format
19/11/13 21:57:48 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = master/192.168.71.128
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.2.0
.........
19/11/13 21:57:55 INFO util.ExitUtil: Exiting with status 0 # 表示成功
.........
  • 启动HDFS
zhangbc@master:~$ start-dfs.sh
Starting namenodes on [master]
master: starting namenode, logging to /home/zhangbc/hadoop2.2.0/logs/hadoop-zhangbc-namenode-master.out
slave_1: starting datanode, logging to /home/zhangbc/hadoop2.2.0/logs/hadoop-zhangbc-datanode-slave_1.out
slave_3: starting datanode, logging to /home/zhangbc/hadoop2.2.0/logs/hadoop-zhangbc-datanode-slave_3.out
slave_2: starting datanode, logging to /home/zhangbc/hadoop2.2.0/logs/hadoop-zhangbc-datanode-slave_2.out zhangbc@master:~$ jps
6524 Jps
5771 NameNode
zhangbc@slave_1:~$ jps
4919 Jps
4818 DataNode
zhangbc@slave_2:~$ jps
4919 Jps
4801 DataNode
zhangbc@slave_3:~$ jps
4705 DataNode
4800 Jps

WEB验证:http://192.168.71.128:50070

  • 启动Yarn
zhangbc@master:~$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/zhangbc/hadoop2.2.0/logs/yarn-zhangbc-resourcemanager-master.out
slave_2: starting nodemanager, logging to /home/zhangbc/hadoop2.2.0/logs/yarn-zhangbc-nodemanager-slave_2.out
slave_1: starting nodemanager, logging to /home/zhangbc/hadoop2.2.0/logs/yarn-zhangbc-nodemanager-slave_1.out
slave_3: starting nodemanager, logging to /home/zhangbc/hadoop2.2.0/logs/yarn-zhangbc-nodemanager-slave_3.out zhangbc@master:~$ jps
5771 NameNode
6642 Jps
zhangbc@slave_1:~$ jps
5099 Jps
4818 DataNode
5011 NodeManager
zhangbc@slave_2:~$ jps
5101 Jps
5016 NodeManager
4801 DataNode
zhangbc@slave_2:~$ jps
5101 Jps
5016 NodeManager
4801 DataNode

WEB验证:http://192.168.71.128:8088

  • 管理JobHistory Server
zhangbc@master:~$ mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /home/zhangbc/hadoop2.2.0/logs/mapred-zhangbc-historyserver-master.out
zhangbc@master:~$ mr-jobhistory-daemon.sh stop historyserver

WEB验证:http://192.168.71.128:19888

  • 集群验证
# 创建目录
zhangbc@master:~$ hdfs dfs -mkdir -p /data/wordscount
zhangbc@master:~$ hdfs dfs -mkdir -p /output # 查看数据目录
zhangbc@master:~$ hdfs dfs -ls /data # 上传本地文件
zhangbc@master:~$ hdfs dfs -put hadoop2.2.0/etc/hadoop/core-site.xml /data/wordscount # 执行
zhangbc@master:~$ hadoop jar hadoop2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /data/wordscount /output/wordscount
...............................
19/11/14 13:04:45 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1573705994579_0001/
19/11/14 13:04:45 INFO mapreduce.Job: Running job: job_1573705994579_0001
19/11/14 13:04:59 INFO mapreduce.Job: Job job_1573705994579_0001 running in uber mode : false
19/11/14 13:04:59 INFO mapreduce.Job: map 0% reduce 0%
19/11/14 13:05:14 INFO mapreduce.Job: map 100% reduce 0%
19/11/14 13:05:27 INFO mapreduce.Job: map 100% reduce 100%
19/11/14 13:05:27 INFO mapreduce.Job: Job job_1573705994579_0001 completed successfully
19/11/14 13:05:27 INFO mapreduce.Job: Counters: 43
............................................ # 查看运行结果
zhangbc@master:~$ hdfs dfs -cat /output/wordscount/part-r-00000

六,安装过程中遇到的问题及其解决方案

  • 问题1:上传文件报错
zhangbc@master:~$ hdfs dfs -put hadoop2.2.0/etc/hadoop/core-site.xml /data/wordcount
19/11/14 10:13:24 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOExcept、ion): File /data/wordcount/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:330)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1226)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)
put: File /data/wordcount/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
19/11/14 10:13:24 ERROR hdfs.DFSClient: Failed to close file /data/wordcount/core-site.xml._COPYING_
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /data/wordcount/core-site.xml._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2477)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:330)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1226)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)

主要原因是重新格式化文件系统,导致master节点下的hadoop2.2.0/hdfs/name/current/VERSION中的clusterIDSlave节点下的hadoop2.2.0/hdfs/data/current/VERSION中的clusterID不一致。在浏览器输入master:50070可以发现Live Nodes为0。

解决方案是修改master节点下的clusterID使之与Slave节点下的clusterID一致,然后重启服务即可。

  • 问题2:执行JAR报错问题
Container launch failed for container_1573700741821_0001_01_000007 : java.lang.IllegalArgumentException: Does not contain a valid host:port authority: slave_1:33775

主要原因Hadoop nodemanager结点主机名不能带下划线_

解决方案:修改主机名称。

  • 问题3:绑定主机名引起的问题:sudo: unable to resolve host master

解决方案如下:

➜  ~ sudo vi /etc/hosts

  1 127.0.0.1       localhost
2 127.0.1.1 ubuntu
3 127.0.1.1 master
:wq!

通过本次实验,对集群概念有个基本的认识,在搭建过程中遇到问题不算太多,主要是对局域网组建缺乏认识深度,本集群环境可以进一步扩展,如动态增减节点,借助Zookeeper技术加以融合等在企业中是比较常见的做法。

分享一份关于Hadoop2.2.0集群环境搭建文档的更多相关文章

  1. Linux下Hadoop2.6.0集群环境的搭建

    本文旨在提供最基本的,可以用于在生产环境进行Hadoop.HDFS分布式环境的搭建,对自己是个总结和整理,也能方便新人学习使用. 基础环境 JDK的安装与配置 现在直接到Oracle官网(http:/ ...

  2. 搭建hadoop2.6.0集群环境

    一.规划 (一)硬件资源 10.171.29.191 master 10.171.94.155  slave1 10.251.0.197 slave3 (二)基本资料 用户:  jediael 目录: ...

  3. 搭建hadoop2.6.0集群环境 分类: A1_HADOOP 2015-04-20 07:21 459人阅读 评论(0) 收藏

    一.规划 (一)硬件资源 10.171.29.191 master 10.171.94.155  slave1 10.251.0.197 slave3 (二)基本资料 用户:  jediael 目录: ...

  4. Hadoop-2.7.1集群环境搭建

    摘自:http://blog.csdn.net/u014039577/article/details/49813531 由于日志数据量越来越大,数据处理的逻辑越来越复杂,同时还涉及到大量日志需要批处理 ...

  5. ElasticSearch 5.0.0 集群安装部署文档

    1.  搭建环境 3台物理机 操作系统 centos7 es1   192.168.31.141   4g内存   2核 es2   192.168.31.142   4g内存   2核 es3    ...

  6. Hadoop 2.6.0-cdh5.4.0集群环境搭建和Apache-Hive、Sqoop的安装

    搭建此环境主要用来hadoop的学习,因此我们的操作直接在root用户下,不涉及HA. Software: Hadoop 2.6.0-cdh5.4.0 Apache-hive-2.1.0-bin Sq ...

  7. Ubuntu12.04-x64编译Hadoop2.2.0和安装Hadoop2.2.0集群

      本文Blog地址:http://www.cnblogs.com/fesh/p/3766656.html   本文对Hadoop-2.2.0源码进行重新编译(64位操作系统下不重新编译会有版本问题) ...

  8. Kafka:ZK+Kafka+Spark Streaming集群环境搭建(二)安装hadoop2.9.0

    如何搭建配置centos虚拟机请参考<Kafka:ZK+Kafka+Spark Streaming集群环境搭建(一)VMW安装四台CentOS,并实现本机与它们能交互,虚拟机内部实现可以上网.& ...

  9. Linux基于Hadoop2.8.0集群安装配置Hive2.1.1及基础操作

    前言 安装Apache Hive前提是要先安装hadoop集群,并且hive只需要在hadoop的namenode节点集群里安装即可,安装前需保证Hadoop已启(动文中用到了hadoop的hdfs命 ...

随机推荐

  1. CS184.1X 计算机图形学导论L3V2和L3V3(部分)

    组合变换 连接矩阵的优点是可以使用这些矩阵单独操作. 多个变换依然是一个矩阵. 连接矩阵不可交换,因为矩阵乘法不具有交换性. X3=RX2 X2=SX1 X3=R(SX1)=(RS)X1 X3≠SRX ...

  2. 渗透测试-基于白名单执行payload--Forfiles

    0x01 Forfiles简介: Forfiles为Windows默认安装的文件操作搜索工具之一,可根据日期,后缀名,修改日期为条件.常与批处理配合使用. 微软官方文档:https://docs.mi ...

  3. Ajax:后台jquery实现ajax无刷新删除数据及demo

    aaarticlea/png;base64,iVBORw0KGgoAAAANSUhEUgAAA8gAAAFSCAIAAAChUmFZAAAgAElEQVR4nO29z4scWZbn2/+Hb30zi8

  4. sqlserver 查看最耗时的前10个存储过程

    SELECT TOP OBJECT_NAME(a.object_id,database_id) SP_Name, DB_NAME(a.database_id) Database_Name, a.cac ...

  5. C++ Web框架::cintara

    1.Cinatra是由C++开源社区purecpp发起的一个开源项目,是一个现代C++写的Web框架,旨在给用户提供一个易用.灵活和高性能的Web框架,让用户能完全专注于核心逻辑而无需关注http细节 ...

  6. php有orm吗

    ORM是通过使用描述对象和数据库之间映射的元数据,将程序中的对象自动持久化到关系数据库中.本质上就是将数据从一种形式转换到另外一种形式. ORM提供了所有SQL语句的生成,代码人员远离了数据库概念.从 ...

  7. ESP8266开发之旅 网络篇⑩ UDP服务

    授人以鱼不如授人以渔,目的不是为了教会你具体项目开发,而是学会学习的能力.希望大家分享给你周边需要的朋友或者同学,说不定大神成长之路有博哥的奠基石... QQ技术互动交流群:ESP8266&3 ...

  8. 云计算 docker 容器部署

    什么是docker容器: 容器就是在隔离的环境中运行的一个进程,如果进程停止,容器就会退出.隔离的环境拥有自己的系统文件,ip地址,主机名等kvm虚拟机,linux,系统文件 容器和虚拟化的区别 : ...

  9. Nginx在Window上简单的使用

    先上Nginx在Window上的基本常用指令: IP_hase也可以解决Session共享的问题:不过不推荐这样使用,建议使用 Memcache/redis来处理 session共享的问题 轮询还是权 ...

  10. js响应式布局

    <!DOCTYPE html> <html class="mobile"> <head> <meta charset="UTF- ...