centos7配置hadoop集群
一：测试环境搭建规划：
| 主机名称 | IP | 用户 | HDFS | YARN |
| hadoop11 | 192.168.1.101 | hadoop | NameNode,DataNode | NodeManager |
| hadoop12 | 192.168.1.102 | hadoop | DataNode | NodeManager |
| hadoop13 | 192.168.1.103 | hadoop | DataNode,SecondaryNameNode | NodeManager |
| hadoop14 | 192.168.1.104 | hadoop | DataNode | ResourceManager,NodeManager |
二：hadoop安装：
1:新建hadoop用户：
useradd hadoop
passwd hadoop
su hadoop
mkdir /home/hadoop/apps
mkdir /home/hadoop/data
三：关闭防火墙：
systemctl stop firewalld
systemctl disable firewalld
四：关闭selinux：
vim /etc/sysconfig/selinux
修改SELINUX=enforcing为SELINUX=disabled
五：安装Java：
5-1下载JDK：
https://www.oracle.com/technetwork/java/javase/downloads/index.html
https://www.oracle.com/technetwork/java/javase/downloads/jdk11-downloads-5066655.html
点选 Accept License Agreement
下载jdk-11.0.1_linux-x64_bin.tar.gz
把下载好的jdk-11.0.1_linux-x64_bin.tar.gz，上传到服务器 /usr/local目录
tar -zxvf jdk-11.0.1_linux-x64_bin.tar.gz
5-2添加环境变量：
vim /etc/profile
在结尾添加：
#java
export JAVA_HOME='/usr/local/jdk-11.0.1'
export PATH=$JAVA_HOME/bin:$PATH
保存退出后，执行：
source /etc/profile
查看版本，确定环境变量是否生效：
java -version
六：安装hadoop
6-1下载hadoop：
https://hadoop.apache.org/releases.html
选择binary文件
拷贝镜像地址，下载
wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz
6-2 解压到指定目录：
tar -zxvf hadoop-2.8.5.tar.gz -C /home/hadoop/apps
6-3修改配置文件：
cd /home/hadoop/apps/hadoop-2.8.5/etc/hadoop
配置Hadoop JDK路径修改hadoop-env.sh、mapred-env.sh、yarn-env.sh文件中的JDK路径：
#export JAVA_HOME='/usr/local/jdk-11.0.1'
export JAVA_HOME='/usr/local/jdk1.8.0_191'
6-3-1 vim core-site.xml
<!-- Put site-specific property overrides in this file. --> <configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop11.zm.com:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/data/hadoop/tmp</value>
</property>
</configuration>
fs.fefaultFS是NameNode的地址，
hadoop.tmp.dir是hadoop临时目录的地址，
应保证目录是已存在的，如不存在，需先创建。
6-3-2 vim hdfs-site.xml
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/data/hadoop/name</value>
<description>为了保证元数据的安全一般配置多个不同目录</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/data/hadoop/data</value>
<description>datanode 的数据存储目录</description>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
<description>HDFS 的数据块的副本存储个数, 默认是3</description>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop13.zm.com:50090</value>
</property>
</configuration>
dfs.namenode.name.dir：namenode数据的存放地点。也就是namenode元数据存放的地方，记录了hdfs系统中文件的元数据。
dfs.datanode.data.dir： datanode数据的存放地点。也就是block块存放的目录了。
dfs.replication：hdfs的副本数设置。也就是上传一个文件，其分割为block块后，每个block的冗余副本个数，默认配置是3。
dfs.secondary.http.address：secondarynamenode 运行节点的信息，和 namenode 不同节点
dfs.namenode.secondary.http-address是指定secondaryNameNode的http访问地址和端口号，因为在规划中，我们将hadoop13规划为SecondaryNameNode服务器。
所以这里设置为：hadoop13.zm.com:50090
6-3-3 配置slave文件,slaves文件是指定HDFS上有哪些DataNode节点：
cd /home/hadoop/apps/hadoop-2.8.5
[hadoop@hadoop11 hadoop-2.8.5]$ vim slaves
hadoop11.zm.com
hadoop12.zm.com
hadoop13.zm.com
hadoop14.zm.com
6-3-4 cd /home/hadoop/apps/hadoop-2.8.5/etc/hadoop
vim yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop14.zm.com</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value></value>
</property>
</configuration>
yarn.resourcemanager.hostname这个指定resourcemanager服务器指向hadoop14.zm.com，
yarn.log-aggregation-enable是配置是否启用日志聚集功能，
yarn.log-aggregation.retain-seconds是配置聚集的日志在HDFS上最多保存多长时间。
6-3-5 cp mapred-site.xml.template mapred-site.xml
vim mapred-site.xml
<!-- Put site-specific property overrides in this file. --> <configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop11.zm.com:</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop11.zm.com:</value>
</property>
</configuration>
mapreduce.framework.name设置mapreduce任务运行在yarn上。
mapreduce.jobhistory.address是设置mapreduce的历史服务器安装在hadoop11机器上。
mapreduce.jobhistory.webapp.address是设置历史服务器的web页面地址和端口号。
七：配置hosts文件：
回到root权限：su
vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
:: localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.201 hadoop11 hadoop11.zm.com 192.168.1.202 hadoop12 hadoop12.zm.com
192.168.1.203 hadoop13 hadoop13.zm.com 192.168.1.204 hadoop14 hadoop14.zm.com
八：SSH免密登陆
su hadoop
所有hadoop集群服务器：
su hadoop
cd /home/hadoop/.ssh
ssh-keygen -t rsa
将所有终端公钥文件拷贝到hadoop11：
所有终端运行：
ssh-copy-id -i hadoop11
登陆hadoop11终端的/home/hadoop/.ssh修改权限：
chmod 600 authorized_keys
分发授权文件到其他终端：
scp /home/hadoop/.ssh/authorized_keys hadoop12:/home/hadoop/.ssh/
scp /home/hadoop/.ssh/authorized_keys hadoop13:/home/hadoop/.ssh/
scp /home/hadoop/.ssh/authorized_keys hadoop14:/home/hadoop/.ssh/
每台机器用ssh命令连接下所有机器，第一次连接要输入yes，以后不用输入
九：hadoop分发到其他终端
拷贝hadoop11 /home/hadoop/apps目录内容到其他终端:
scp -rq data hadoop12:/home/hadoop/
拷贝hadoop11 /home/hadoop/data目录内容到其他终端:
scp -rq data hadoop12:/home/hadoop/
十：配置hadoop环境变量（所有节点均需配置）
注意：
1、如果使用root用户进行安装。 vi /etc/profile 即可 系统变量
2、如果使用普通用户进行安装。 vi ~/.bashrc 用户变量
vim .bashrc
# User specific aliases and functions export HADOOP_HOME=/home/hadoop/apps/hadoop-2.8.
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
[hadoop@hadoop11 ~]$ source .bashrc
查看hadoop版本：hadoop version
[hadoop@hadoop11 ~]$ hadoop version
Hadoop 2.8.
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 0b8464d75227fcee2c6e7f2410377b3d53d3d5f8
Compiled by jdu on --10T03:32Z
Compiled with protoc 2.5.
From source with checksum 9942ca5c745417c14e318835f420733
This command was run using /home/hadoop/apps/hadoop-2.8./share/hadoop/common/hadoop-common-2.8..jar
十一：hadoop初始化
HDFS初始化只能在主节点上进行
如果需要重新初始化，先删除相关目录文件，
粗略做法是删除/home/hadoop/data/hadoop路径下
[hadoop@hadoop14 hadoop]$ ll
总用量
drwx------. hadoop hadoop 10月 : data
drwxrwxr-x. hadoop hadoop 10月 : name
drwxrwxr-x. hadoop hadoop 10月 : tmp
[hadoop@hadoop14 hadoop]$ pwd
/home/hadoop/data/hadoop
这几个目录的子目录；
[hadoop@hadoop11 ~]$ hadoop namenode -format
// :: INFO namenode.FSImage: Allocated new BlockPoolId: BP--198.8.8.201-
// :: INFO common.Storage: Storage directory /home/hadoop/data/hadoop/name has been successfully formatted.
// :: INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/data/hadoop/name/current/fsimage.ckpt_0000000000000000000 using no compression
// :: INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/data/hadoop/name/current/fsimage.ckpt_0000000000000000000 of size bytes saved in seconds.
// :: INFO namenode.NNStorageRetentionManager: Going to retain images with txid >=
// :: INFO util.ExitUtil: Exiting with status
// :: INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop11/198.8.8.201
************************************************************/
初始化后会自动在/home/hadoop/data/hadoop/name路径
建立current目录
生成：
-rw-rw-r-- hadoop hadoop 10月 : fsimage_0000000000000000000
-rw-rw-r-- hadoop hadoop 10月 : fsimage_0000000000000000000.md5
-rw-rw-r-- hadoop hadoop 10月 : seen_txid
-rw-rw-r-- hadoop hadoop 10月 : VERSION
十二：启动HDFS：
可以在集群中任何节点执行：
[hadoop@hadoop12 ~]$ start-dfs.sh
报错
[hadoop@hadoop12 ~]$ start-dfs.sh
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/home/hadoop/apps/hadoop-2.8./share/hadoop/common/lib/hadoop-auth-2.8..jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Starting namenodes on [hadoop11.zm.com]
The authenticity of host 'hadoop11.zm.com (198.8.8.201)' can't be established.
ECDSA key fingerprint is SHA256:RlH2tSlCWblQ5jZ6DMuCt4+yEmYuWo5MZuqUTGesM9I.
ECDSA key fingerprint is MD5:de:e9:::c7:::0c:e6:8a::0e::5f:2b:8d.
Are you sure you want to continue connecting (yes/no)? no
关闭HDFS:
stop-dfs.sh
降低java版本到jdk1.8.0_191
如步骤6-3修改集群所有节点配置文件：
cd /home/hadoop/apps/hadoop-2.8.5/etc/hadoop
配置Hadoop JDK路径修改hadoop-env.sh、mapred-env.sh、yarn-env.sh文件中的JDK路径：
#export JAVA_HOME='/usr/local/jdk-11.0.1'
export JAVA_HOME='/usr/local/jdk1.8.0_191'
再次启动HDFS（任何节点均可执行）:
start-dfs.sh
[hadoop@hadoop11 sbin]$ start-dfs.sh
Starting namenodes on [hadoop11.zm.com]
hadoop11.zm.com: starting namenode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-namenode-hadoop11.out
hadoop11: starting datanode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-datanode-hadoop11.out
hadoop13: starting datanode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-datanode-hadoop13.out
hadoop14: starting datanode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-datanode-hadoop14.out
hadoop12: starting datanode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-datanode-hadoop12.out
Starting secondary namenodes [hadoop13.zm.com]
hadoop13.zm.com: starting secondarynamenode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-secondarynamenode-hadoop13.out
在其他节点启动时会报询问信息
The authenticity of host 'hadoop13.zm.com (198.8.8.203)' can't be established.
ECDSA key fingerprint is SHA256:RlH2tSlCWblQ5jZ6DMuCt4+yEmYuWo5MZuqUTGesM9I.
ECDSA key fingerprint is MD5:de:e9:::c7:::0c:e6:8a::0e::5f:2b:8d.
Are you sure you want to continue connecting (yes/no)?
解决办法（所有节点执行）：
进入root用户
su
vim /etc/ssh/ssh_config
加入如下两句
# StrictHostKeyChecking ask
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
start-dfs.sh执行后：
在/home/hadoop/data/hadoop/data中自动生成
[hadoop@hadoop11 data]$ ll
总用量
drwxrwxr-x hadoop hadoop 10月 : current
-rw-rw-r-- hadoop hadoop 10月 : in_use.lock
在/home/hadoop/data/hadoop/name中自动生成
[hadoop@hadoop11 name]$ ll
总用量
drwxrwxr-x hadoop hadoop 10月 : current
-rw-rw-r-- hadoop hadoop 10月 : in_use.lock
十三：启动YARN（只能在ResourceManager主节点执行）：
[hadoop@hadoop14 ~]$ start-yarn.sh
[hadoop@hadoop14 ~]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-resourcemanager-hadoop11.out
hadoop12: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop12.out
hadoop13: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop13.out
hadoop14: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop14.out
hadoop11: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop11.out
start-yarn.sh执行后：
会在/home/hadoop/data/hadoop/tmp中自动生成
[hadoop@hadoop14 tmp]$ ll
总用量
drwxr-xr-x hadoop hadoop 10月 : nm-local-dir
[hadoop@hadoop14 nm-local-dir]$ ll
总用量
drwxr-xr-x hadoop hadoop 10月 : filecache
drwx------ hadoop hadoop 10月 : nmPrivate
drwxr-xr-x hadoop hadoop 10月 : usercache
此配置启动后，ResourceManager没有启动，无法访问：http://198.8.8.204:8088/
需要在所有节点添加ResourceManager配置：
vim vim yarn-site.xml
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration> <!-- Site specific YARN configuration properties --> <property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop14</value>
</property> <property>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:</value>
#ResourceManager 对客户端暴露的地址。客户端通过该地址向RM提交应用程序，杀死应用程序等。
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:</value>
#ResourceManager 对ApplicationMaster暴露的访问地址。ApplicationMaster通过该地址向RM申请资源、释放资源等。
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:</value>
#ResourceManager 对NodeManager暴露的地址。NodeManager通过该地址向RM汇报心跳，领取任务等。
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:</value>
#ResourceManager 对管理员暴露的访问地址。管理员通过该地址向RM发送管理命令等。
</property> <property>
<name>yarn.resourcemanager.webapp.address</name>
# <value>198.8.8.204:</value>
<value>${yarn.resourcemanager.hostname}:</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value></value>
</property> </configuration>
保存配置，
在ResourceManager主节点hadoop14重启yarn：
stop-yarn.sh
start-yarn.sh
[hadoop@hadoop14 hadoop]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-resourcemanager-hadoop14.out
hadoop11: Warning: Permanently added 'hadoop11,198.8.8.201' (ECDSA) to the list of known hosts.
hadoop12: Warning: Permanently added 'hadoop12,198.8.8.202' (ECDSA) to the list of known hosts.
hadoop14: Warning: Permanently added 'hadoop14,198.8.8.204' (ECDSA) to the list of known hosts.
hadoop13: Warning: Permanently added 'hadoop13,198.8.8.203' (ECDSA) to the list of known hosts.
hadoop14: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop14.out
hadoop11: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop11.out
hadoop13: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop13.out
hadoop12: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop12.out
[hadoop@hadoop14 hadoop]$ jps
ResourceManager
DataNode
NodeManager
Jps
可以访问：http://198.8.8.204:8088/
十四：查看所有节点进程
jps
[hadoop@hadoop11 nm-local-dir]$ jps
NodeManager
DataNode
Jps
NameNode
[hadoop@hadoop12 nm-local-dir]$ jps
NodeManager
DataNode
Jps
[hadoop@hadoop13 nm-local-dir]$ jps
NodeManager
DataNode
Jps
SecondaryNameNode
[hadoop@hadoop14 nm-local-dir]$ jps
DataNode
NodeManager
Jps
十五：HDFS和YARN的WEB管理界面：
http://198.8.8.201:50070
十六：hadoop简单应用：
创建目录
[hadoop@hadoop11 ~]$ hadoop fs -mkdir -p /test/input
查看创建文件
hadoop fs -ls /
新建文件
vim words.txt
hello zhangsan
hello lisi
hello wangwu
上传文件
hadoop fs -put ~/words.txt /test/input
查看是否上传成功
hadoop fs -ls /test/input
下载文件
hadoop fs -get /test/input/words.txt /home/hadoop/
查看是否下载成功
cd /home/hadoop
ll
用wordcount测试：
[hadoop@hadoop11 hadoop]$ hadoop jar /home/hadoop/apps/hadoop-2.8.5/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar wordcount /test/input /test/output
// :: INFO client.RMProxy: Connecting to ResourceManager at hadoop14/198.8.8.204:
// :: INFO input.FileInputFormat: Total input files to process :
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1541142769004_0001
// :: INFO impl.YarnClientImpl: Submitted application application_1541142769004_0001
// :: INFO mapreduce.Job: The url to track the job: http://hadoop14:8088/proxy/application_1541142769004_0001/
// :: INFO mapreduce.Job: Running job: job_1541142769004_0001
// :: INFO mapreduce.Job: Job job_1541142769004_0001 running in uber mode : false
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: Job job_1541142769004_0001 completed successfully
// :: INFO mapreduce.Job: Counters:
File System Counters
FILE: Number of bytes read=
FILE: Number of bytes written=
FILE: Number of read operations=
FILE: Number of large read operations=
FILE: Number of write operations=
HDFS: Number of bytes read=
HDFS: Number of bytes written=
HDFS: Number of read operations=
HDFS: Number of large read operations=
HDFS: Number of write operations=
Job Counters
Launched map tasks=
Launched reduce tasks=
Data-local map tasks=
Total time spent by all maps in occupied slots (ms)=
Total time spent by all reduces in occupied slots (ms)=
Total time spent by all map tasks (ms)=
Total time spent by all reduce tasks (ms)=
Total vcore-milliseconds taken by all map tasks=
Total vcore-milliseconds taken by all reduce tasks=
Total megabyte-milliseconds taken by all map tasks=
Total megabyte-milliseconds taken by all reduce tasks=
Map-Reduce Framework
Map input records=
Map output records=
Map output bytes=
Map output materialized bytes=
Input split bytes=
Combine input records=
Combine output records=
Reduce input groups=
Reduce shuffle bytes=
Reduce input records=
Reduce output records=
Spilled Records=
Shuffled Maps =
Failed Shuffles=
Merged Map outputs=
GC time elapsed (ms)=
CPU time spent (ms)=
Physical memory (bytes) snapshot=
Virtual memory (bytes) snapshot=
Total committed heap usage (bytes)=
Shuffle Errors
BAD_ID=
CONNECTION=
IO_ERROR=
WRONG_LENGTH=
WRONG_MAP=
WRONG_REDUCE=
File Input Format Counters
Bytes Read=
File Output Format Counters
Bytes Written=
查看输出文件：
[hadoop@hadoop11 hadoop]$ hadoop fs -ls /test/output
[hadoop@hadoop11 hadoop]$ hadoop fs -ls /test/output
Found items
-rw-r--r-- hadoop supergroup -- : /test/output/_SUCCESS
-rw-r--r-- hadoop supergroup -- : /test/output/part-r-
[hadoop@hadoop11 hadoop]$ hadoop fs -cat /test/output/part-r-
hello
lisi
wangwu
zhangsan
参考文档：
https://www.cnblogs.com/qingyunzong/p/8496127.html
https://blog.csdn.net/hliq5399/article/details/78193113
centos7配置hadoop集群的更多相关文章
- centos7配置Hadoop集群环境
参考: https://blog.csdn.net/pucao_cug/article/details/71698903 设置免密登陆后,必须重启ssh服务 systermctl restart ss ...
- Linux中安装配置hadoop集群
一. 简介 参考了网上许多教程,最终把hadoop在ubuntu14.04中安装配置成功.下面就把详细的安装步骤叙述一下.我所使用的环境:两台ubuntu 14.04 64位的台式机,hadoop选择 ...
- centos7配置fastdfs集群(5.09)
centos7配置fastdfs集群(5.09) 2017年03月10日 23:34:26 带鱼兄 阅读数 1564 版权声明:本文为博主原创文章,转载请注明出处. https://blog.c ...
- 初学Hadoop：利用VMWare+CentOS7搭建Hadoop集群
一.前言 开始学习数据处理相关的知识了,第一步是搭建一个Hadoop集群.搭建一个分布式集群需要多台电脑,在此我选择采用VMWare+CentOS7搭建一个三台虚拟机组成的Hadoop集群. 注:1 ...
- hadoop集群环境搭建之安装配置hadoop集群
在安装hadoop集群之前,需要先进行zookeeper的安装,请参照hadoop集群环境搭建之zookeeper集群的安装部署 1 将hadoop安装包解压到 /itcast/ (如果没有这个目录 ...
- CentOS7 安装Hadoop集群环境
先按照上一篇安装与配置好CentOS以及zookeeper http://www.cnblogs.com/dopeter/p/4609276.html 本章介绍在CentOS搭建Hadoop集群环境 ...
- 在 Linux 服务器上搭建和配置 Hadoop 集群
实验条件:3台centos服务器,jdk版本1.8.0,Hadoop 版本2.8.0 注:hadoop安装和搭建过程中都是在用户lb的home目录下,master的主机名为host98,slave的主 ...
- 安装和配置hadoop集群步骤
hadoop集群的安装步骤和配置 hadoop是由java语言编写的,首先我们肯定要在电脑中安装jdk,配置好jdk的环境,接下来就是安装hadoop集群的步骤了,在安装之前需要创建hadoop用户组 ...
- linux配置hadoop集群
①安装虚拟机 ②为虚拟机添加共享文件 右击已经安装好的虚拟机 设置—>选项—>共享文件 ③配置映射 sudo nano /etc/network/interfaces 重启网络:sud ...
随机推荐
- 【转】 C语言文件操作详解
转自:http://www.cnblogs.com/likebeta/archive/2012/06/16/2551780.html C语言中没有输入输出语句,所有的输入输出功能都用 ANSI C提供 ...
- CentOS优化
一.CentOS6.x优化 #.更改yum源 mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup ...
- 机器学习 之k-means和DBSCAN的区别
目录 1.定义和区别(优缺点对比) 2.kmeans原理 3.DBSCAN原理 1.定义和区别(优缺点对比) 聚类分为:基于划分.层次.密度.图形和模型五大类: 均值聚类k-means是基于划分的聚类 ...
- 【洛谷p1601】A+B Problem（高精）
高精度加法的思路还是很简单容易理解的 A+B Problem(高精)[传送门] 洛谷算法标签: 附上代码(最近懒得一批) #include<iostream> #include<cs ...
- Financiers Game CodeForces - 737D (博弈论)
直接暴力区间DP的话是$O(n^3)$, 关键注意到每步走的距离差不超过1, 所以差最大是$O(\sqrt{n})$的, 所以实际上有用的状态是$O(n^2)$的, 可以通过.
- NIM 博弈 牛客小白月赛2 E-是是非非
题目链接 分析:一个裸的NIM博弈 对于一个Nim游戏的局面(a1,a2,...,an),它是P-position(即当前局面先手必败)当且仅当a1^a2^...^an=0,其中^表示异或(xor)运 ...
- 2018 Benelux Algorithm Programming Contest (BAPC 18)I In Case of an Invasion, Please. . .
题意:一副无向有权图,每个点有一些人,某些点是避难所(有容量),所有人要去避难所,问最小时间所有人都到达避难所, 题解:dij+二分+最大流check,注意到避难所最多10个,先挨个dij求到避难所的 ...
- spring boot 2.0(二)动态banner的支持
Spring Boot 2.0 提供了很多新特性,其中就有一个小彩蛋:动态 Banner,今天我们就先拿这个来尝尝鲜. 配置依赖 使用 Spring Boot 2.0 首先需要将项目依赖包替换为刚刚发 ...
- 关于POD和非POD类型中，list initialization和constructor initialization（未解决）
如果你的成员是POD类型的,那么list initialization和constructor initialization没有任何区别 #include<iostream> using ...
- TCP如何保证可靠性
如何保证可靠性? 1.校验和.在TCP的首部中有一个占据16为的空间用来放置校验和的结果. 这是一个端到端的检验和,目的是检测数据在传输过程中的任何变化.如果收到段的检验和有差错,TCP将丢弃这个报文 ...