CentOS 7.3 CDH 5.10.0 Druid0.12.4安装记录

CentOS 7.3 CDH 5.10.0安装记录

0. 集群规划
192.167.1.247 realtime247 realtime+hadoopdata
192.167.1.248 broker248 broker
192.167.1.249 history249 middleware+history+hadoopdata
192.167.1.250 history250 middleware+history+hadoopdata
192.167.1.251 master251 Mysql+coordinator+overlord+hadoopMaster

1. 环境准备

安装第三方依赖包：（所有节点，注意一定要安装)
yum install -y chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/defrag

修改swappiness参数:
vi /etc/sysctl.conf 文件里添加如下参数：
vm.swappiness=0
或者：
echo 'vm.swappiness=0'>> /etc/sysctl.conf

JDK 1.8 RPM包:
http://download.oracle.com/otn-pub/java/jdk/8u172-b11/a58eab1ec242421181065cdc37240b08/jdk-8u172-linux-x64.rpm?AuthParam=1527227488_e083c615628d8b91527508f00008dd6d

下载cloudera-manager，parcel：
https://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz

http://archive.cloudera.com/cdh5/parcels/5.10.0/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel
http://archive.cloudera.com/cdh5/parcels/5.10.0/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1
http://archive.cloudera.com/cdh5/parcels/5.10.0/manifest.json

MYSQL安装包：
https://cdn.mysql.com/archives/mysql-5.6/MySQL-server-5.6.30-1.el7.x86_64.rpm
https://cdn.mysql.com/archives/mysql-5.6/MySQL-client-5.6.30-1.el7.x86_64.rpm
https://cdn.mysql.com/archives/mysql-5.6/MySQL-shared-5.6.30-1.el7.x86_64.rpm
https://cdn.mysql.com/archives/mysql-5.6/MySQL-shared-compat-5.6.30-1.el7.x86_64.rpm
https://cdn.mysql.com/archives/mysql-5.6/MySQL-embedded-5.6.30-1.el7.x86_64.rpm
https://cdn.mysql.com/archives/mysql-5.6/MySQL-devel-5.6.30-1.el7.x86_64.rpm
或者
https://cdn.mysql.com/archives/mysql-5.6/MySQL-5.6.30-1.el7.x86_64.rpm-bundle.tar

jdbc连接包mysql-connector-java.jar:
http://dev.mysql.com/downloads/connector/j/

2. 每台机器安装JDK
2.1 检查是否安装了JDK： rpm -qa | grep jdk
2.2 如果已经安装了JDK，则卸载: yum -y remove [查询出来的包名]
yum -y remove java-1.7.0-openjdk-headless-1.7.0.111-2.6.7.8.el7.x86_64
yum -y remove java-1.8.0-openjdk-1.8.0.102-4.b14.el7.x86_64
yum -y remove java-1.7.0-openjdk-1.7.0.111-2.6.7.8.el7.x86_64
yum -y remove copy-jdk-configs-1.2-1.el7.noarch
yum -y remove java-1.8.0-openjdk-headless-1.8.0.102-4.b14.el7.x86_64

2.3 安装下载下来的JDK RPM包:
rpm -ivh /home/workspace/zipped/jdk-8u172-linux-x64.rpm

2.4 设置JDK环境变量
vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_172-amd64
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

source /etc/profile #环境变量生效

3. 配置网络，修改机器名，关闭防火墙，关闭selinux，安装NTP时间同步
3.1 配置固定IP地址（每台机器，IP每台自己选择变化)
vi /etc/sysconfig/network-scripts/ifcfg-eth0

TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
NAME=eth0
UUID=7ac09286-c35b-4f15-a9ba-701c093832bf
DEVICE=eth0
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_PRIVACY=no
ONBOOT=yes
DNS1=192.168.1.1
IPADDR=192.168.1.246
PREFIX=24
GATEWAY=192.168.1.1

3.2 修改机器名（在各台机器上执行相应的脚本)
192.167.1.247
hostnamectl set-hostname realtime247
hostnamectl --static set-hostname realtime247

192.167.1.248
hostnamectl set-hostname broker248
hostnamectl --static set-hostname broker248

192.167.1.249
hostnamectl set-hostname history249
hostnamectl --static set-hostname history249

192.167.1.250
hostnamectl set-hostname history250
hostnamectl --static set-hostname history250

192.167.1.251
hostnamectl set-hostname master251
hostnamectl --static set-hostname master251

修改hosts文件
vi /etc/hosts

127.0.0.1 localhost
::1 localhost

192.168.1.247 realtime247
192.168.1.248 broker248
192.168.1.249 history249
192.168.1.250 history250
192.168.1.251 master251

3.3 关闭防火墙
systemctl stop firewalld #关闭防火墙
systemctl disable firewalld #禁用防火墙
systemctl status firewalld #查看防火墙状态

3.4 selinux关闭(所有节点官方文档要求)
vi /etc/selinux/config
SELINUX=disabled
#注意： 重启才能生效
#重启后检查
sestatus -v
#SELinux status: disabled

3.5 NTP服务器设置(用于不同节点间实现时间同步)
3.5.1 yum -y install ntp #所有节点安装时间同步服务
3.5.2 master节点更改时间
更改master的节点的时间同步服务器
vi /etc/ntp.conf
# 注释掉所有server *.*.*的指向，新添加一条可连接的ntp服务器(百度一下ntp服务器，我选的是上海交大的)
server ntp.sjtu.edu.cn iburst
restrict default nomodify notrap
driftfile /var/lib/ntp/drift
restrict 127.0.0.1
restrict -6 ::1
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys

3.5.3 在其他节点上把ntp指向master服务器地址即可(/etc/ntp.conf下)
vi /etc/ntp.conf
server master251 iburst

3.5.4 启动并设置开机启动
systemctl start ntpd.service #启动时间同步服务
systemctl enable ntpd.service #设置开机启动

3.5.5 时间同步(主节点不做）
ntpdate -u master251

4. 设置SSH免密登陆
A） 每台机器生成访问秘钥，复制到192.168.1.251:/home/workspace目录下
192.168.1.247：
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys 192.168.1.251:/home/workspace/authorized_keys247
rm -rf ~/.ssh/authorized_keys #删除公钥文件

192.168.1.248：
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys 192.168.1.251:/home/workspace/authorized_keys248
rm -rf ~/.ssh/authorized_keys #删除公钥文件

192.168.1.249：
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys 192.168.1.251:/home/workspace/authorized_keys249
rm -rf ~/.ssh/authorized_keys #删除公钥文件

192.168.1.250：
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys 192.168.1.251:/home/workspace/authorized_keys250
rm -rf ~/.ssh/authorized_keys #删除公钥文件

192.168.1.251：
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
cp ~/.ssh/authorized_keys /home/workspace/authorized_keys251
rm -rf ~/.ssh/authorized_keys #删除公钥文件

B） 在192.168.1.251上将所有的共钥合并成一个公钥文件
cat /home/workspace/authorized_keys247 >> /home/workspace/authorized_keys
cat /home/workspace/authorized_keys248 >> /home/workspace/authorized_keys
cat /home/workspace/authorized_keys249 >> /home/workspace/authorized_keys
cat /home/workspace/authorized_keys250 >> /home/workspace/authorized_keys
cat /home/workspace/authorized_keys251 >> /home/workspace/authorized_keys

C） 将合并后的公钥文件复制到集群中的各个主机中
scp /home/workspace/authorized_keys 192.168.1.247:~/.ssh/
scp /home/workspace/authorized_keys 192.168.1.248:~/.ssh/
scp /home/workspace/authorized_keys 192.168.1.249:~/.ssh/
scp /home/workspace/authorized_keys 192.168.1.250:~/.ssh/
cp /home/workspace/authorized_keys ~/.ssh/ #因为目前在251主机中，所以使用的命令为cp而不是scp

D） 每台机器：
chmod 600 ~/.ssh/authorized_keys #authorized_keys的访问权限
chmod 600 ~/.ssh/

5. 安装MYSQL
A) 查询MYSQL是否安装，如果安装了，则卸载
rpm -qa | grep mariadb
rpm -qa | grep mysql

卸载命令：
# rpm -e [查询出来的MYSQL包名] --nodeps
注意：如果卸载不了就强制卸载，--nodeps --force

yum -y remove [查询出来的MYSQL包名]

rm -rf /var/lib/mysql
rm /etc/my.cnf

B） 安装MYSQL
rpm -ivh MySQL-devel-5.6.30-1.el7.x86_64.rpm
rpm -ivh MySQL-shared-5.6.30-1.el7.x86_64.rpm
rpm -ivh MySQL-server-5.6.30-1.el7.x86_64.rpm
rpm -ivh MySQL-client-5.6.30-1.el7.x86_64.rpm

C) 设置MYSQL
C-1) systemctl start mysql #启动MYSQL ##>>>MYSQL启动后，默认其实己经生成了默认密码，密码保存在/root/.mysql_secret文件中
C-2) chkconfig mysql on #设置开机启动
C-3) 修改MYSQL默认密码
方法1：
mysql -uroot -p[初始化密码] #cat /root/.mysql_secret 即可看到初始化密码
SET PASSWORD=PASSWORD('root'); （重置密码）
方法2：
service mysql stop
mysqld_safe --user=mysql --skip-grant-tables --skip-networking&
mysql -u root mysql
mysql> use mysql;
mysql> UPDATE user SET Password=PASSWORD('root')where USER='root';
mysql> FLUSH PRIVILEGES;
mysql> quit
# service mysql restart
# mysql -u root –p
Enter password: root
mysql>SET PASSWORD=PASSWORD('root'); （重置密码）
mysql> use mysql; (此DB存放MySQL的各种配置信息)
Database changed
mysql> select host,user from user; (查看是否能够查询)

mysqladmin -uroot password '新密码' #设置密码

如果数据库如法启动，使用下面的脚本进行初始化：
mysql_install_db --user=mysql --ldata=/var/lib/mysql/

C-4) 设置远程访问
GRANT ALL PRIVILEGES ON *.* TO '新用户'@'%' IDENTIFIED BY '新用户的密码' WITH GRANT OPTION;
FLUSH PRIVILEGES;

C-5) 复制jdbc mysql 访问包到java lib目录下
如果/usr/share/java/目录不存在，执行
mkdir /usr/share/java/
cp mysql-connector-java-5.1.40-bin.jar /usr/share/java/mysql-connector-java.jar
或
cp mysql/mysql-connector-java-5.1.40-bin.jar /usr/java/jdk1.8.0_172-amd64/lib/mysql-connector-java.jar

C-6） 创建数据库
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
grant all on *.* to root@"%" Identified by "root";

6. 安装Cloudera-Manager
A） 解压cm tar包到指定目录所有服务器都要(或者在主节点解压好，然后通过scp到各个节点同一目录下)
scp /home/workspace/zipped/cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz 192.168.1.247:/home/workspace/zipped/
scp /home/workspace/zipped/cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz 192.168.1.248:/home/workspace/zipped/
scp /home/workspace/zipped/cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz 192.168.1.249:/home/workspace/zipped/
scp /home/workspace/zipped/cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz 192.168.1.250:/home/workspace/zipped/

mkdir /opt/cloudera-manager
tar -axvf /home/workspace/zipped/cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz -C /opt/cloudera-manager

B) 创建cloudera-scm用户（所有节点）
useradd --system --home=/opt/cloudera-manager/cm-5.10.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

C) 在主节点创建cloudera-manager-server的本地元数据保存目录
mkdir /var/cloudera-scm-server
chown cloudera-scm:cloudera-scm /var/cloudera-scm-server
chown cloudera-scm:cloudera-scm /opt/cloudera-manager

D) 配置从节点cloudera-manger-agent指向主节点服务器(master251节点不做此操作)
vi /opt/cloudera-manager/cm-5.10.0/etc/cloudera-scm-agent/config.ini
将server_host改为CMS所在的主机名即master251

E) 主节点中创建parcel-repo仓库目录(只在master251节点操作)
mkdir -p /opt/cloudera/parcel-repo
ln -s /var/www/html/cdh/510/parcels/ parcel-repo
chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
#cp CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1 manifest.json /opt/cloudera/parcel-repo
#mv CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1 CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha #把CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1 后缀要把1去掉

F) 所有节点创建parcels目录(一定要在所有节点操作）
#解释：Clouder-Manager将CDHs从主节点的/opt/cloudera/parcel-repo目录中抽取出来，分发解压激活到各个节点的/opt/cloudera/parcels目录中】
mkdir -p /opt/cloudera/parcels #创建parcels目录
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels #给cloudera-scm授权目录读写权限

G）初始脚本配置数据库scm_prepare_database.sh(在主节点master251上操作)

#mysql -u root -proot #使用root用户登陆mysql
#create database cloudera DEFAULT CHARSET utf8 COLLATE utf8_general_ci; #创建cloudera数据库
grant all on *.* to root@"%" Identified by "root"; #授权

/opt/cloudera-manager/cm-5.10.0/share/cmf/schema/scm_prepare_database.sh mysql -hlocalhost -uroot -proot --scm-host master251 cloudera root root
说明：这个脚本就是用来创建和配置CMS需要的数据库的脚本。各参数是指：
mysql：数据库用的是mysql，如果安装过程中用的oracle，那么该参数就应该改为oracle。
-hlocalhost：数据库建立在localhost主机上面。也就是主节点上面。
-uroot：root身份运行mysql。-123456：mysql的root密码是***。
--scm-host hadoop1：CMS的主机，一般是和mysql安装的主机是在同一个主机上。
最后三个参数是：数据库名，数据库用户名，数据库密码。

H） 主节点上启动cloudera-scm-server
cp /opt/cloudera-manager/cm-5.10.0/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server #将cloudera-scm-server复制到开机启动目录
chkconfig cloudera-scm-server on #设置cloudera-scm-server开机启动

vi /etc/init.d/cloudera-scm-server #修改cloudera-scm-server服务启动位置
#CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}改为
CMF_DEFAULTS=/opt/cloudera-manager/cm-5.10.0/etc/default

service cloudera-scm-server start #启动cloudera-scm-server服务
#同时为了保证在每次服务器重启的时候都能启动cloudera-scm-server，应该在开机启动脚本vi /etc/rc.local中加入命令：
service cloudera-scm-server restart

I） 启动cloudera-scm-agent所有节点
mkdir /opt/cloudera-manager/cm-5.10.0/run/cloudera-scm-agent #创建cloudera-scm-agent运行目录
cp /opt/cloudera-manager/cm-5.10.0/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent #将cloudera-scm-agent复制到开机启动目录
chkconfig cloudera-scm-agent on #设置cloudera-scm-agent开机启动

vi /etc/init.d/cloudera-scm-agent #修改cloudera-scm-agent服务启动位置
#CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}改为
CMF_DEFAULTS=/opt/cloudera-manager/cm-5.10.0/etc/default

service cloudera-scm-agent start #启动cloudera-scm-agent服务
#同时为了保证在每次服务器重启的时候都能启动cloudera-scm-agent，应该在开机启动脚本vi /etc/rc.local中加入命令：
service cloudera-scm-agent restart

Master:
mkdir /var/log/cloudera-scm-server
chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-server
chmod g+w /var/log/cloudera-scm-server

Agent:
mkdir /var/log/cloudera-scm-agent
chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-agent
chmod g+w /var/log/cloudera-scm-agent

7. 搭建本地yum源

A) 安装软件
yum install -y yum-utils createrepo #master机器执行
yum install -y httpd # master机器执行 安装apache2
systemctl start httpd.service #master机器执行 启动apache服务
systemctl enable httpd.service #master机器执行 设置开机启动

systemctl start httpd.service #启动
systemctl stop httpd.service #停止
systemctl restart httpd.service #重启
systemctl enable httpd.service #开机启动
systemctl disable httpd.service #开机不启动

B) 下载CDH rpm包和Parcel包
下载地址：
http://archive.cloudera.com/cm5/repo-as-tarball/5.10.0/cm5.10.0-centos7.tar.gz #CM
http://archive.cloudera.com/cdh5/repo-as-tarball/5.10.0/ #CDH
mkdir -p /var/www/html/cdh/510

tar -zxvf /home/workspace/zipped/cm5.10.0-centos7.tar.gz -C /var/www/html/cdh/510 #解压到本地www站点地址 rpms
mv /opt/cloudera/parcel-repo/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel /var/www/html/cdh/510
mv /opt/cloudera/parcel-repo/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha /var/www/html/cdh/510
mv /opt/cloudera/parcel-repo/manifest.json /var/www/html/cdh/510

chmod -R 755 /var/www/html/* #授权文件夹权限

C) 本地yum源
yum install yum-utils createrepo #安装repo util
mv /home/lenmom/cdh5/ /home/lenmom/cdh5/ # /home/lenmom/cdh5/ contains parcels and rpms.
createrepo /var/www/html/cdh/510/rpms # /var/www/html/cdh/510/rpms rpms.

chmod -R 755 /var/www/html/* #授权文件夹权限

vi /etc/yum.repos.d/cdh510.repo 写入如下内容
[cdh510]
name=repo
baseurl=http://master251/cdh/510/rpms/
enabled=true
gpgcheck=false

分发到所有agent节点:
scp /etc/yum.repos.d/cdh510.repo realtime247:/etc/yum.repos.d/
scp /etc/yum.repos.d/cdh510.repo broker248:/etc/yum.repos.d/
scp /etc/yum.repos.d/cdh510.repo history249:/etc/yum.repos.d/
scp /etc/yum.repos.d/cdh510.repo history250:/etc/yum.repos.d/

保存后，执行以下两条命令
yum clean all #清除yum缓存
rm -Rf /var/cache/yum/x86_64
yum update #更新yum缓存
yum makecache #所有节点执行

8. 安装：

查看端口占用：
netstat -anp|grep 7180 #7180是cloudera-server host的端口号，查看某个端口是否被占用
lsof -i:7180 #7180是cloudera-server host的端口号，查看某个端口是否被占用

浏览器
http://192.168.1.251:7180/ 开始安装过程（等待服务器启动需要一段时间，耐心等待）

远程pacel全部删除，只留一个，地址为：
http://192.168.1.251/cdh/510/parcels/

注意：安装过程中，如果出现“主机运行状况不良”，是因为节点上次安装没有成功，需要删除cm_guid文件才能再次安装。其次要注意master分配的资源是否足够，
有可能是因为master资源不够，心跳无法及时响应，导致报告主机运行状况不良。解决办法如下：
1. rm -f /opt/cloudera-manager/cm-5.10.0/lib/cloudera-scm-agent/cm_guid
service cloudera-scm-agent restart
2. 增加master节点的内存和cpu资源，让他能够及时响应心跳检测服务。