安装CDH5.11.2集群

master  192.168.1.30

saver1  192.168.1.40

saver2  192.168.1.50

首先，时间同步

然后，ssh互通

接下来开始：

1.安装MySQL5.6.44

tar -xvf MySQL-5.6.44-1.el6.x86_64.rpm-bundle.tar

rpm -ivh MySQL-server-5.6.44-1.el6.x86_64.rpm

rpm -ivh MySQL-devel-5.6.44-1.el6.x86_64.rpm

启动MySQL

[root@master soft]# cp /usr/share/mysql/my-default.cnf /etc/my.cnf

初始化MySQL

/usr/bin/mysql_install_db

[root@master ~]# service mysql start

查看密码

mysql -u root -pIBPFigA5SLmylk0m

mysql> SET PASSWORD = PASSWORD('mysql');

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+
4 rows in set (0.00 sec)
MySQL安装完成。

2.安装Oracle1.7 JDK（三台机器都需要执行）

[root@master soft]# rpm -ivh oracle-j2sdk1.-1.7.+update67-.x86_64.rpm

[root@master hadoop]# vim /etc/profile
export JAVA_HOmE=/usr/java/jdk1..0_67-cloudera
export PATH=.:$JAVA_HOmE/bin:$PATH
export CLASSPATH=.:$JAVA_HOmE/lib/dt.jar:$JAVA_HOmE/lib/tools.jar

[root@master hadoop]# source /etc/profile
[root@master soft]# java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

3.关闭防火墙
[root@master soft]# service iptables status
iptables: Firewall is not running.

4.调整Linux参数
echo 0 >/proc/sys/vm/swappiness
echo "vm.swappiness=0" >>/etc/sysctl.conf
echo "echo 0 > /proc/sys/vm/swappiness" >>/etc/rc.d/rc.local
cat /proc/sys/vm/swappiness

5.禁用hugepage透明大页，因为它可能带来CPU利用过高的问题
echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >>/etc/rc.d/rc.local
echo "echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag" >>/etc/rc.d/rc.local

6.修改最大文件句柄数
echo "* soft nofile 128000" >> /etc/security/limits.conf
echo "* hard nofile 128000" >> /etc/security/limits.conf
echo "* soft nproc  128000" >> /etc/security/limits.conf
echo "* hard nproc  128000" >> /etc/security/limits.conf
sed -i 's/1024/unlimited/' /etc/security/limits.d/90-nproc.conf
ulimit -SHn 128000
ulimit -SHu 128000

7.安装httpd启动
[root@master soft]# ps -ef|grep http
root       5695   5630  0 19:39 pts/1    00:00:00 grep http
[root@master soft]# service httpd  status
httpd is stopped
[root@master soft]# yum install httpd
[root@master soft]# chkconfig httpd on
[root@master soft]# service httpd start
Starting httpd: httpd: Could not reliably determine the servers fully qualified domain name, using 192.168.1.30 for ServerName
                                                           [  OK  ]
[root@master soft]# service httpd  status
httpd (pid  5727) is running...
[root@master soft]#  ps -ef|grep http
root       5727      1  0 19:42 ?        00:00:00 /usr/sbin/httpd
apache     5730   5727  0 19:42 ?        00:00:00 /usr/sbin/httpd
apache     5731   5727  0 19:42 ?        00:00:00 /usr/sbin/httpd
apache     5732   5727  0 19:42 ?        00:00:00 /usr/sbin/httpd
apache     5733   5727  0 19:42 ?        00:00:00 /usr/sbin/httpd
apache     5734   5727  0 19:42 ?        00:00:00 /usr/sbin/httpd
apache     5735   5727  0 19:42 ?        00:00:00 /usr/sbin/httpd
apache     5736   5727  0 19:42 ?        00:00:00 /usr/sbin/httpd
apache     5737   5727  0 19:42 ?        00:00:00 /usr/sbin/httpd
root       5747   5630  0 19:42 pts/1    00:00:00 grep http

8.创建相关的数据库

create database oozie default charset utf8 collate utf8_general_ci;
create database hive default charset utf8 collate utf8_general_ci;
create database sentry default charset utf8 collate utf8_general_ci;
create database scm default charset utf8 collate utf8_general_ci;
create database monitor default charset utf8 collate utf8_general_ci;
create database metastore default charset utf8 collate utf8_general_ci;
create database amon default charset utf8 collate utf8_general_ci;
create database hue default charset utf8 collate utf8_general_ci;

9.安装cloudera-manager

[root@master soft]# tar -xvf cloudera-manager-el6-cm5.11.2_x86_64.tar -C /opt

[root@master opt]# ls -rlt
total 16
drwxr-xr-x  4 1106 4001 4096 Aug 10  2017 cloudera
drwxr-xr-x  9 1106 4001 4096 Aug 10  2017 cm-5.11.2

上传文件：

[root@master parcel-repo]# pwd
/opt/cloudera/parcel-repo
[root@master parcel-repo]# ls -rlt
total 1478540
-rw-r--r-- 1 root root         41 Aug  2 09:32 CDH-5.11.2-1.cdh5.11.2.p0.4-el6.parcel.sha
-rw-r--r-- 1 root root      72302 Aug  2 09:32 manifest.json
-rw-r--r-- 1 root root 1513940381 Aug  2 09:33 CDH-5.11.2-1.cdh5.11.2.p0.4-el6.parcel

[root@master cloudera-scm-agent]# pwd
/opt/cm-5.11.2/etc/cloudera-scm-agent
[root@master cloudera-scm-agent]# vim config.ini

# Hostname of the CM server.
server_host=master     #主节点所在的主机名

进行同步从节点：

scp -r /opt/cm-5.11.2 root@saver1:/opt
scp -r /opt/cm-5.11.2 root@saver2:/opt

10.所有的节点都建立clouder-scm用户

useradd --system --home=/opt/cm-5.11.2/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

11.主节点初始化配置数据库

把mysql-connector-java-5.1.46.jar 放到这个路径下：/opt/cm-5.11.2/share/cmf/lib/mysql-connector-java-5.1.46.jar   （三个节点都操作）

chmod 777 /opt/cm-5.11.2/share/cmf/lib/mysql-connector-java-5.1.46.jar

[root@master ~]# /opt/cm-5.11.2/share/cmf/schema/scm_prepare_database.sh mysql -hlocalhost -uroot -pmysql --scm-host localhost scm scm scm
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
Verifying that we can write to /opt/cm-5.11.2/etc/cloudera-scm-server
Creating SCM configuration file in /opt/cm-5.11.2/etc/cloudera-scm-server
Executing:  /usr/java/jdk1.7.0_67-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.11.2/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cm-5.11.2/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

12.在主节点上启动CM服务和CM Agent

/opt/cm-5.11.2/etc/init.d/cloudera-scm-server start
/opt/cm-5.11.2/etc/init.d/cloudera-scm-agent  start

13.从节点上启动Agent

/opt/cm-5.11.2/etc/init.d/cloudera-scm-agent  start

14.登入cloudera Manager

http://192.168.1.30:7180/cmf/login

我们看到HDFS是红色的感叹号

原因是设置的副本备份数与DataNode的个数不匹配。

我们在之前理论篇中已经说明了dfs. replication属性默认是3，也就是说副本数---块的备份数默认为3份。

但是我们这里集群只有两个DataNode。

所以导致了达不到目标---副本备份不足。

解决方法

这种情况下的修复有2个步骤，1是设置目标备份数为2，2是通过命令更改当前备份数。

副本不足和副本过多都可以用这2个步骤解决，主要是跟DataNode的个数对应。

设置目标备份数为2

点击集群-HDFS-配置

搜索dfs. replication，设置为2后保存更改

dfs.replication这个参数其实只在文件被写入dfs时起作用，虽然更改了配置文件，但是不会改变之前写入的文件的备份数。所以我们还需要步骤2在master中通过命令更改备份数：

[hdfs@master ~]# hdfs dfs -setrep -R 2 /这里的-R 2的数字2就对应我们的DataNode个数。

需要修改/etc/hadoop/conf.cloudera.hdfs/hdfs-site.xml

加上：

<property>
    <name>dfs.permissions</name>
    <value>false</value>
  </property>

错误就消失了