CentOS 7安装Hadoop集群
准备三台虚拟机，ip分别为192.168.220.10（master）、192.168.220.11（slave1）、192.168.220.12（slave2）
准备好jdk-6u45-linux-x64.bin和hadoop-1.2.1-bin.tar.gz，放在/usr/local/src/目录下
安装JDK（每台虚拟机都安装）
1.进入到/usr/local/src/目录，执行./jdk-6u45-linux-x64.bin
2.修改~/.bashrc，在文件末尾增加三行
export JAVA_HOME=/usr/local/src/jdk1.6.0_45
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin
3.使环境变量生效，执行source ~/.bashrc
安装Hadoop
在192.168.220.10机器上安装hadoop
1.进入到/usr/local/src/目录，解压hadoop-1.2.1-bin.tar.gz，执行tar -zxf hadoop-1.2.1-bin.tar.gz
2.修改配置文件
masters文件
master
slaves文件
slave1
slave2
core-site.xml文件
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/src/hadoop-1.2.1/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.220.10:9000</value>
</property>
</configuration>
mapred-site.xml文件
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>http://192.168.220.10:9001</value>
</property>
</configuration>
hdfs-site.xml文件
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
hadoop-env.sh文件，在后面添加一行
export JAVA_HOME=/usr/local/src/jdk1.6.0_45
3.将/usr/local/src/hadoop-1.2.1目录拷贝到192.168.220.11、192.168.220.12机器上
配置hostname
配置192.168.220.10的主机名为master
1.执行hostname master
2.修改/etc/hostname文件
master
修改192.168.220.11的主机名为slave1，修改192.168.220.12的主机名为slave2
配置host文件
三台机器的/etc/hosts文件末尾添加以下代码
192.168.220.10 master
192.168.220.11 slave1
192.168.220.12 slave2
关闭防火墙和SELinux
分别在每台机器上执行
systemctl stop firewalld.service
setenforce
配置SSH
1.在192.168.220.10上执行ssh-keygen，在~目录下新增.ssh目录，目录中的文件为id_rsa，id_rsa.pub
2.将id_rsa.pub拷贝为authorized_keys
cp id_rsa.pub authorized_keys
3.在192.168.220.11和192.168.220.12上分别执行ssh-keygen
4.将192.168.220.11和192.168.220.12上id_rsa.pub的内容分别拷贝到192.168.220.10的authorized_keys文件中，如下：
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9mGRhFOdcoHw9GUnKQmqThNKpsyah93Dtq/d8RICGWIHDRJ3GXd0sEcb743ejwbuCMmtlhheXcU0FuyA6Cm0jvMyvDfaPKArtxl6KT7Z93uC0VDCXDRomueux81HAIVjc7ZqlXwVeYs1LITxEeJykKlFOXvK7JexWhWGdMMADwxbFMbaNsZ9EwRxcFLFtNg65FQ+u8CIV9KR3D02kemwLCsP+xiRcgs+wirQPm5JM+2cJoLsVQBz3Hk335IsEhc1Xb9Cralo8Tt8gh/ho8K/1pVjvyW1b0LkP9HGNdwVYD9wkWdEJRkryLXBEXpjk4xu+riF+N4rOzJD root@master
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn79fdfR/NjzPVD3NPj1vBBfQdVOrv7jeb4UJCOsd7xioPRiz8gOQnOmhu5C+GchbyGA+tg5pXwnNJTOO2wn32U4lOPndW0okN/wqyN4vgq/taJi7JgY/8rneBiGaIIdNIy/pAGlMwb53Qn766adetMhsxYMD2l4uxmbVVjzCRb8QP5EsAYTmmFOODzJsPm70uF3j1Q8zGavYg0wFSYR/yECQns4DBSuBJNxdGY6PskBXqurahwi5yaR3vWV1Ix4wtB6BYuQomEnGdzOSfrBMZ/yc5tXo0xmEfY7wFkize6z9Pm2E3oDoMR18YkwT1Cz6fHikVILA9cldtL root@slave1
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCydYCvASzCZggks4hMqOcYSGLO2eAvocWezNOMwspTfpJ105Jumb/vf5h6cRZeckq56IvhSV6t6mytk4pZoZjjZPSmWvCwLtMRMPShNbA3BYtj5V3WRKV8ZcMrNdD//U7iHHoJm57vI/m+XO42YSYjPw7JDkb8Ij9b6zgI3fyvbSSYeXb451PlyJLHdxIzRMAaZDSbAML9e7EO8VJB9Wf9bXpow4+VeP33it3kgMNUlHQtyqduSwYGxVVtGsUTJkxnuRsbWeeA1/pp8MNFKUgBTMALTVHByglgZqwGcbblJxsG832PIZNRECIFqorm6odftjnT4DR7/0yR root@slave2
5.将192.168.220.10的authorized_keys文件拷贝到192.168.220.11、192.168.220.12机器上
配置完成后三台机器可以互相访问不用密码
好了，到这里Hadoop集群就配置完了，让我们来使用下吧
在192.168.220.10上格式化namenode
进入到/usr/local/src/hadoop-1.2.1/bin目录，执行
./hadoop namenode -format
出现
// :: INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = master/192.168.220.10
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.6.0_45
************************************************************/
// :: INFO util.GSet: Computing capacity for map BlocksMap
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 2.0% max memory =
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO util.GSet: recommended=, actual=
// :: INFO namenode.FSNamesystem: fsOwner=root
// :: INFO namenode.FSNamesystem: supergroup=supergroup
// :: INFO namenode.FSNamesystem: isPermissionEnabled=true
// :: INFO namenode.FSNamesystem: dfs.block.invalidate.limit=
// :: INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval= min(s), accessTokenLifetime= min(s)
// :: INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length =
// :: INFO namenode.NameNode: Caching file names occuring more than times
// :: INFO common.Storage: Image file /usr/local/src/hadoop-1.2./tmp/dfs/name/current/fsimage of size bytes saved in seconds.
// :: INFO namenode.FSEditLog: closing edit log: position=, editlog=/usr/local/src/hadoop-1.2./tmp/dfs/name/current/edits
// :: INFO namenode.FSEditLog: close success: truncate to , editlog=/usr/local/src/hadoop-1.2./tmp/dfs/name/current/edits
// :: INFO common.Storage: Storage directory /usr/local/src/hadoop-1.2./tmp/dfs/name has been successfully formatted.
// :: INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.220.10
************************************************************/
用jps查看进程
[root@master bin]# jps
JobTracker
NameNode
SecondaryNameNode
Jps
在192.168.220.11查看进程
DataNode
Jps
TaskTracker
在192.168.220.12查看进程
DataNode
TaskTracker
Jps
测试下：
执行./hadoop fs -ls /
drwxr-xr-x - root supergroup -- : /usr
上传文件，执行./hadoop fs -put /root/w.txt /
查看文件执行./hadoop fs -cat /w.txt，显示
ddd
成功
注：
问题：上传文件时出现以下错误
[root@master bin]# ./hadoop fs -put /root/a.txt /
// :: WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /a.txt could only be replicated to nodes, instead of
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)
at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:) at org.apache.hadoop.ipc.Client.call(Client.java:)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:)
at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:)
at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:) // :: WARN hdfs.DFSClient: Error Recovery for null bad datanode[] nodes == null
// :: WARN hdfs.DFSClient: Could not get block locations. Source file "/a.txt" - Aborting...
put: java.io.IOException: File /a.txt could only be replicated to nodes, instead of
// :: ERROR hdfs.DFSClient: Failed to close file /a.txt
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /a.txt could only be replicated to nodes, instead of
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)
at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:) at org.apache.hadoop.ipc.Client.call(Client.java:)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:)
at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:)
at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:)
解决方法：关闭防火墙
CentOS 7安装Hadoop集群的更多相关文章
- CentOS 6 安装Hadoop集群
hadoop2.6.4安装包(链接:https://pan.baidu.com/s/15qHpdoLBQHP4HdxOJzNcsg 密码:5hel) 1.准备Linux环境 1.0先将虚拟机的网络模式 ...
- docker安装hadoop集群
docker安装hadoop集群?图啥呢?不图啥,就是图好玩.本篇博客主要是来教大家如何搭建一个docker的hadoop集群.不要问 为什么我要做这么无聊的事情,答案你也许知道,因为没有女票.... ...
- CentOS7 搭建Ambari-Server，安装Hadoop集群（一）
2017-07-05:修正几处拼写错误,之前没发现,抱歉! 第一次在cnblogs上发表文章,效果肯定不会好,希望各位多包涵. 编写这个文档的背景是月中的时候,部门老大希望我们能够抽时间学习一下Had ...
- 在CentOS上安装ZooKeeper集群
一共准备3个CentOS虚拟机 172.16.9.194 172.16.9.195 172.16.9.196 上传zookeeper-3.3.6.tar.gz到服务器并解压,3台服务器的目录结构如下 ...
- 安装Hadoop集群的最快的软件
Quick Hadoop是一款安装Hadoop集群的桌面软件,只需要点两下鼠标,一分钟之内安装Hadoop到集群上,超快! 还在每台主机的Shell里一行一行地敲安装Hadoop的命令?别苦逼了! 用 ...
- 通过ambari安装hadoop集群,ZT
通过ambari安装hadoop集群,ZT http://www.cnblogs.com/cenyuhai/p/3295635.html http://www.cnblogs.com/cenyuhai ...
- Linux上安装Hadoop集群(CentOS7+hadoop-2.8.0)--------hadoop环境的搭建
Linux上安装Hadoop集群(CentOS7+hadoop-2.8.0)------https://blog.csdn.net/pucao_cug/article/details/71698903 ...
- 安装hadoop集群--hdfs
安装hadoop集群--hdfs 大数据软件 链接:https://pan.baidu.com/s/1-3PYLHMgvvONawJq55hstQ 提取码:izqf 准备一台干净的虚拟机-centos ...
- 图文讲解基于centos虚拟机的Hadoop集群安装，并且使用Mahout实现贝叶斯分类实例 （7）
接下来,我们开启hadoop集群. 如果之前打开过Hadoop,可能会发生lock的问题,解决方案:http://blog.csdn.net/caoshichaocaoshichao/article/ ...
随机推荐
- Maven 打war包
使用maven将项目达成war包 右击项目的跟项目,选择run as 选择maven build.... 进入窗口,在 Goals 输入命令 clean package,选择 skip Tes ...
- PAT 甲级 1036 Boys vs Girls (25 分)（简单题）
1036 Boys vs Girls (25 分) This time you are asked to tell the difference between the lowest grade ...
- javascript从网络下载随机笑话
/*** * 讲笑话函数(调试用) * @param callback 回调函数 */ function randomText(callback) { var result =''; $.ajax({ ...
- Oracle中查询当前时间、时间格式化方法
Oracle中如何获取系统当前时间 select to_char(sysdate,'yyyy-mm-dd hh24:mi:ss') from dual; ORACLE里获取一个时间的年.季.月.周. ...
- mysql命令、mysqldump命令找不到解决
1.解决bash: mysql: command not found 的方法 [root@DB-02 ~]# mysql -u root -bash: mysql: command not found ...
- 红绿灯 promise和原始方式实现
Promise 方式 async+await function sleep(duration){ return new Promise(function(resolve){ setTimeout(re ...
- vue-slicksort拖拽组件
vue-slicksort拖拽组件 安装 通过npm安装 $ npm install vue-slicksort --save 通过yarn安装 $ yarn add vue-slicksort 插件 ...
- 无权限安装vim8
本文介绍Ubuntu14.04在没有sudo权限的情况下安装vim8 1.默认vim版本为7.3 $ vim --versionVIM - Vi IMproved 7.3 (2010 Aug 15, ...
- python3.7.3安装beautifulsoup4出现版本不兼容的问题
今天想安装一个beautifulsoup4,结果一直出错,好多教程总是有各种坑……找了很多个教程,为了记录方法也为了分享给大家,简单些一个.但是是真真不想再费劲写一遍了……直接贴链接,亲测有效,但是底 ...
- vue启动时报 This relative module was not found
This relative module was not found: * ../../vue-temp/vue-editor-bridge in ./node_modules/babel-loade ...