安装hadoop集群-cm 5.14.4

环境 要求：

centos 7

java 1.8

cm 5.14.4

mysql 5.7

4核16G

工具要求：

mysql-connector-java-5.1.39.jar

1、绑定 hosts

10.31.210.15 test-algo-hadoop-01

10.31.210.16 test-algo-hadoop-02

10.31.210.17 test-algo-hadoop-03

2、受信ssh key

test-algo-hadoop-01 自动登录 其它 机器 
test-algo-hadoop-02 自动 登录 其它 的机器 
test-algo-hadoop-01 和test-algo-hadoop-02互登录

3、修改内核并增加到rc.local文件

# 内存参数调整 #sysctl -w vm.swappiness=10 或者 编辑vim /etc/sysctl.conf，

在最后加入： #vm.swappiness = 10

编辑启动项vim /etc/rc.local,最后加入：

#echo never > /sys/kernel/mm/transparent_hugepage/defrag

#echo never > /sys/kernel/mm/transparent_hugepage/enabled

关闭防火墙

#systemctl disable firewalld
#systemctl stop firewalld

设置时间同步

ntpd

4、下载软件包和工具

mysql-connector-java-5.1.39.jar

CDH-5.14.4-1.cdh5.14.4.p0.3-el7.parcel

CDH-5.14.4-1.cdh5.14.4.p0.3-el7.parcel.sha

cloudera-manager-centos7-cm5.14.4_x86_64.tar.gz

manifest.json

解压 ：

#cp /data/soft/cloudera-manager-centos7-cm5.14.4_x86_64.tar.gz /opt/ && cd /opt

#tar zcvf cloudera-manager-centos7-cm5.14.4_x86_64.tar.gz

#cp /data/soft/CDH-5.14.4-1.cdh5.14.4.p0.3-el7.parcel* /opt/cloudera/parcel-repo/

5、安装jdk 1.8 && mysql  （略）

6、配置mysql帐号

grant all on *.* to 'hadoop'@'10.31.%' identified by 'xxxx' with grant option;

flush privileges;

创建数据库

create database amon DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
create database rman DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
create database metastore DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
create database sentry DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
create database nav DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
create database navms DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

安装java（所有机器） &&  mysql jar库

#cp /data/soft/mysql-connector-java-5.1.39.jar /opt/cm-5.14.4/share/cmf/cloudera-navigator-server/jars/
pw
#cp /data/soft/mysql-connector-java-5.1.39.jar /opt/cm-5.14.4/share/cmf/lib

#mkdir -p /usr/share/java && cp /data/soft/mysql-connector-java-5.1.39.jar /usr/share/java/mysql-connector-java.jar

#初始化mysql

/opt/cm-5.14.4/share/cmf/schema/scm_prepare_database.sh mysql cm -htest-algo-hadoop-01 -uhadoop -p --scm-host "10.31.%" scm

yum安装：

yum install libxslt-devel libxslt-python httpd mod_ssl cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-plain chkconfig bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb httpd httpd-tools unzip ntp

cm agent 配置

#vim /opt/cm-5.14.4/etc/cloudera-scm-agent/config.ini

server_host=test-algo-hadoop-01

#scp到其它的服务器

#scp -P 57301 -r /opt/cm-5.14.4 test-algo-hadoop-02:/opt/

#scp -P 57301 -r /opt/cm-5.14.4 test-algo-hadoop-03:/opt/

在所有节点创建cloudera-scm用户

#useradd --system --home=/opt/cm-5.14.4/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

启动cm server 和agent:

/opt/cm-5.14.4/etc/init.d/cloudera-scm-server restart

/opt/cm-5.14.4/etc/init.d/cloudera-scm-agent restart

访问 ：http://服务器IP:7180

安装cloudera 时报错

Error, CM server guid updated, expected 25cf17b3-391a-4368-848a-07118d6f11fb, received b3d4a47a-1476-4ea4-b236-24426b1b8540

rm /opt/cm-5.14.4/lib/cloudera-scm-agent/cm_guid  -f

/opt/cm-5.14.4/etc/init.d/cloudera-scm-agent restart

重新生成guid