这次搭建我使用的机器 os 是 Centos7.4 RH 系的下面以流的方式纪录搭建过程以及注意事项

Step1:

配置域名相关,因为只有三台机器组集群,所以直接使用了 hosts 的方法:

修改主机名
hostnamectl set-hostname ryze-.bigdata.com 然后在 /etc/hosts 文件中配置相关机器的域名 和 域名简写
x.x.x.x ryze-.bigdata.com ryze-
x.x.x.x zed-.bigdata.com zed-
x.x.x.x zed-.bigdata.com zed-
并且测试其能互通 配置 /etc/sysconfig/network
HOSTNAME=foo-.example.com 验证配置
uname -a 需要和 hostname 得到一致的域名

Step2:

关闭防火墙

# 防火墙关闭
iptables-save > /root/firewall.rules
sudo chkconfig iptables off
sudo service iptables stop

Step3:

启动 NTP 服务

yum install ntp

设置一个同步时间服务器
/etc/ntp.conf # 笔者使用的 aliyun 这里应该是可以跳过的,我检查了安装 ntp 之后,里面配置了非常多 aliyun 的相关节点。
server .pool.ntp.org
server .pool.ntp.org
server .pool.ntp.org # 开启 NTP 服务
sudo systemctl start ntpd # 配置 NTP 服务自启动
sudo systemctl enable ntpd # 向某个服务器同步时间
ntpdate -u <ntp_server> # 同步系统时间
hwclock --systohc

所有机器完成上面配置之后,我们开始进入安装的步骤。

Step1:

首先 Cloudera 为用户已经准备好了专用的程序仓库,我们需要将其下载下来

wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/

# Import the repository signing GPG key
sudo rpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera

Step2: 安装 为 CDH 6.0.x 安装 jdk1.8

# yum 安装
sudo yum install oracle-j2sdk1. # 自己下载包安装
tar xvfz /path/to/jdk-8u<update_version>-linux-x64.tar.gz -C /usr/java/
注意不能更换路径 这里如果使用 yum 安装 cloudera 的源的话会安装 1.8u141 并且会补充一个文件
  • The RHEL-compatible and Ubuntu operating systems supported by Cloudera Enterprise 6 all use AES-256 encryption by default for tickets. To support AES-256 bit encryption in JDK versions lower than 1.8u161, you must install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy File on all cluster and Hadoop user machines. Cloudera Manager can automatically install the policy files, or you can install them manually. For JCE Policy File installation instructions, see the README.txt file included in the jce_policy-x.zip file. JDK 1.8u161 and higher enable unlimited strength encryption by default, and do not require policy files.
  • On SLES platforms, do not install or try to use the IBM Java version bundled with the SLES distribution. CDH does not run correctly with that version.
需要注意一下,Java1.8u161 以上的版本似乎不受影响。

Step3:

安装 cloudera manager 到目标机 和 配置 TLS

sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

警告如果没有证书,就不要配这个否则后面会出现 agent 无法上报的问题。
然后配置 auto TSL
sudo JAVA_HOME=/usr/java/jdk1..0_141-cloudera /opt/cloudera/cm-agent/bin/certmanager setup --configure-services That's it! When you start Cloudera Manager Server, it will have TLS enabled, and all hosts that you add to the cluster, as well as any supported services, will automatically have TLS configured and enabled.

Step4: 安装 CM 使用的数据库

# 安装数据库大礼包
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-.noarch.rpm
sudo yum update
sudo yum install mysql-server
sudo systemctl start mysqld # 然后开始数据库配置
sudo systemctl stop mysqld # 拷贝日志文件
Move old InnoDB log files /var/lib/mysql/ib_logfile0 and /var/lib/mysql/ib_logfile1 out of /var/lib/mysql/ to a backup location # 配置数据库文件 /etc/my.conf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size =
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = max_connections =
#expire_logs_days =
#max_binlog_size = 100M #log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log #In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id= binlog_format = mixed read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M # InnoDB settings
innodb_file_per_table =
innodb_flush_log_at_trx_commit =
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency =
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M [mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid sql_mode=STRICT_ALL_TABLES # 添加到自启动项
sudo systemctl enable mysqld # 重新将 MySQL 启动起来
sudo systemctl start mysqld

安装 JDBC 驱动

# Download the MySQL JDBC driver from http://www.mysql.com/downloads/connector/j/5.1.html (in .tar.gz format). As of the time of writing, you can download version 5.1.46 using wget as follows:
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz # Extract the JDBC driver JAR file from the downloaded file. For example:
tar zxvf mysql-connector-java-5.1..tar.gz # Copy the JDBC driver, renamed, to /usr/share/java/. If the target directory does not yet exist, create it. For example:
sudo mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.
sudo cp mysql-connector-java-5.1.-bin.jar /usr/share/java/mysql-connector-java.jar

使用 utf8 编码规则创建新的数据库服务于需要使用到数据库的服务:

Databases for Cloudera Software
Service Database User
Cloudera Manager Server scm scm
Activity Monitor amon amon
Reports Manager rman rman
Hue hue hue
Hive Metastore Server metastore hive
Sentry Server sentry sentry
Cloudera Navigator Audit Server nav nav
Cloudera Navigator Metadata Server navms navms
Oozie oozie oozie

这里我们会使用的服务:

Cloudera Manager Server

Reports Manager

Hue

Hive Metastore Server

Cloudera Navigator Audit Server

Cloudera Navigator Metadata Server

但是我打算全部都建起来,后面再来看

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY 'metastore';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';

Step5: setup database

由于我们的数据库搭建在远端 所以我们使用:

Example : Running the script when MySQL or MariaDB is installed on another host
This example demonstrates how to run the script on the Cloudera Manager Server host (cm01.example.com) and connect to a remote MySQL or MariaDB host (db01.example.com): sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h db01.example.com --scm-host cm01.example.com scm scm
Enter database password:
JAVA_HOME=/usr/java/jdk1..0_141-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1..0_141-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!

Step6: 安装 CDH 和 其他的组件

# start cloudera-scm-server
sudo systemctl start cloudera-scm-server # 通过该命令观察启动流程
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
当看到
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
the Cloudera Manager Admin Console is ready 然后就可以通过 ip: 上去了

之后就是选择版本,选择完了之后进入到集群安装界面:

扫描 hosts 地址使用 FQDN 来标识,填写之后进行扫描。

Range Definition    Matching Hosts
10.1..[-] 10.1.1.1, 10.1.1.2, 10.1.1.3, 10.1.1.4
host[-].example.com host1.example.com, host2.example.com, host3.example.com
host[-].company.com host07.example.com, host08.example.com, host09.example.com, host10.example.com
Note: Unqualified hostnames (short names) must be unique in a Cloudera Manager instance. For example, you cannot have both host01.example.com and host01.standby.example.com managed by the same Cloudera Manager Server.
You can specify multiple addresses and address ranges by separating them with commas, semicolons, tabs, or blank spaces, or by placing them on separate lines. Use this technique to make more specific searches instead of searching overly wide ranges. Only scans that reach hosts running SSH will be selected for inclusion in your cluster by default. You can enter an address range that spans over unused addresses and then clear the nonexistent hosts later in the procedure, but wider ranges require more time to scan.

单机 serach 即可扫描,扫描到节点之后进行下一步。

Select Repository 使用默认选项直接选择下一步。

Accept JDK License 阅读协议☑️选择下一步。

ssh 设置,选择合适的选项下一步。

然后开始安装 agents 下载速度慢可以直接用 proxychain 或者 找快的地方在 https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/ 下好之后上传。

然后开始安装 parcels 因为这个在国内安装下载速度也是超级慢所以推荐直接用网速快的地方去 https://archive.cloudera.com/cdh6/6.0.1/parcels/  下载对应的包。

然后上传到服务器放到 cm 机器的 /opt/cloudera/parcel-repo/ 目录下。要下包 sha1 文件和 manifest 文件。

按照提示解决 warning

已启用透明大页面压缩,可能会导致重大性能问题。请运行“echo never > /sys/kernel/mm/transparent_hugepage/defrag”和
“echo never > /sys/kernel/mm/transparent_hugepage/enabled”以禁用此设置,然后将同一命令添加到 /etc/rc.local 等初始化脚本中,以便在系统重启时予以设置。以下主机将受到影 Starting with CDH 6, PostgreSQL-backed Hue requires the Psycopg2 version to be at least 2.5.4, see the documentation for more information.
This warning can be ignored if hosts will not run CDH 6, or will not run Hue with PostgreSQL. The following hosts have an incompatible Psycopg2 version of '2.5.1': 

Step7: Set Up a Cluster Using the Wizard

选择想要安装的服务

进行角色配置,master 集中在一台机器的话尽量让该机器的配置比较强。

比如我们的 ryze-1 的机器就比较强,所以承担了更多的 master 节点的角色。

设置要安装服务使用的数据库,之前我们 setup 了数据库信息,这里直接填写之前 setup 的信息

审计查看配置参数有没有需要修改的

命令详细信息执行相关操作,启动相关服务

summary 提示信息 -> 服务已安装、配置并在群集中运行 XD。

Reference:

https://www.cloudera.com/documentation/enterprise/6/6.0/topics/installation_reqts.html  Before You Install

https://www.cloudera.com/documentation/enterprise/6/6.0/topics/install_cm_cdh.html  Installing Cloudera Manager, CDH, and Managed Services

http://www.uuboku.com/444.html  mysql配置sql_mode中STRICT_TRANS_TABLES和STRICT_ALL_TABLES 区别

http://www.cnblogs.com/zhoujinyi/p/3179279.html  InnoDB O_DIRECT选项漫谈(一)【转】

https://www.cloudera.com/documentation/enterprise/6/6.0/topics/cm_ig_installing_configuring_dbs.html#cmig_topic_5_1  Required Databases

https://www.oschina.net/question/3964891_2287725  CDH6在安装agent时,提示安装失败 无法接收 Agent 发出的检测信号

https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/  yum 包下载包地址

https://archive.cloudera.com/cdh6/6.0.1/parcels/  parcels 下载包地址

CDH 6.0.1 集群搭建 「Process」的更多相关文章

  1. CDH 6.0.1 集群搭建 「Before install」

    从这一篇文章开始会有三篇文章依次介绍集群搭建 「Before install」 「Process」 「After install」 继上一篇使用 docker 部署单机 CDH 的文章,当我们使用 d ...

  2. CDH 6.0.1 集群搭建 「After install」

    集群搭建完成之后其实还有很多配置工作要做,这里我列举一些我去做的一些. 首先是去把 zk 的角色重新分配一下,不知道是不是我在配置的时候遗漏了什么在启动之后就有报警说目前只能检查到一个节点.去将 zk ...

  3. Redis 5.0.5集群搭建

    Redis 5.0.5集群搭建 一.概述 Redis3.0版本之后支持Cluster. 1.1.redis cluster的现状 目前redis支持的cluster特性: 1):节点自动发现 2):s ...

  4. java_redis3.0.3集群搭建

    redis3.0版本之后支持Cluster,具体介绍redis集群我就不多说,了解请看redis中文简介. 首先,直接访问redis.io官网,下载redis.tar.gz,现在版本3.0.3,我下面 ...

  5. Redis 3.0.2集群搭建以及相关问题汇总

    Redis3 正式支持了 cluster,是为了解决构建redis集群时的诸多不便 (1)像操作单个redis一样操作key,不用操心key在哪个节点上(2)在线动态添加.删除redis节点,不用停止 ...

  6. Hadoop2.0 HA集群搭建步骤

    上一次搭建的Hadoop是一个伪分布式的,这次我们做一个用于个人的Hadoop集群(希望对大家搭建集群有所帮助): 集群节点分配: Park01 Zookeeper NameNode (active) ...

  7. redis4.0.6集群搭建

    文件环境:CentOS7 + redis4.0.6 先去官网下载redis:https://redis.io/,然后上传到你的虚拟机,我上传到了/mysoft 先解压->然后进入主目录-> ...

  8. redis3.0.3集群搭建

    redis3.0版本之后支持Cluster,具体介绍redis集群我就不多说,了解请看redis中文简介. 首先,直接访问redis.io官网,下载redis.tar.gz,现在版本3.0.3,我下面 ...

  9. ubuntu18.04 flink-1.9.0 Standalone集群搭建

    集群规划 Master JobManager Standby JobManager Task Manager Zookeeper flink01 √ √ flink02 √ √ flink03 √ √ ...

随机推荐

  1. 【转】用ffmpeg转多音轨的mkv文件

    命令: ffmpeg -i AmericanCaptain.mkv -map 0:v -vcodec copy -map 0:a:1 -acodec copyAmericanCaptain.mp4 - ...

  2. springboot--bean交给容器

    1.把bean交给springboot管理 springboot也是一个spring容器.把一个bean交给springboot的容器有三种方法,前两种是先把bean交给spring容器再把sprin ...

  3. Python-wxpy继承关系

    聊天对象 通过机器人对象 Bot 的 chats(), friends(),groups(), mps() 方法, 可分别获取到当前机器人的 所有聊天对象.好友.群聊,以及公众号列表. 而获得到的聊天 ...

  4. modbus与rs485的关系_modbus与rs485的区别和联系

    http://www.elecfans.com/tongxin/123/20180103610476.html 经常看到RS485和MODBUS写在一起,它们的区别和联系? RS485是一个物理接口, ...

  5. ogg BR – BOUNDED RECOVERY 测试案例

    首先,我们来看两个OGG同步中可能的问题: l oracle在线日志包含已提交的和未提交的事务,但OGG只会将已提交的事务写入到队列文件.因此,针对未提交的事务,特别是未提交的长事务,OGG会怎样处理 ...

  6. 从 0 到 1 实现 React 系列 —— 2.组件和 state|props

    看源码一个痛处是会陷进理不顺主干的困局中,本系列文章在实现一个 (x)react 的同时理顺 React 框架的主干内容(JSX/虚拟DOM/组件/生命周期/diff算法/setState/ref/. ...

  7. Python全栈开发之路 【第八篇】:面向对象编程设计与开发(2)

    一.继承与派生 什么是继承? 继承指的是类与类之间的关系,是一种什么是什么的关系,继承的功能之一就是用来解决代码重用问题. 继承是一种创建新的类的方式,在python中,新建的类可以继承一个或多个父类 ...

  8. Python学习第三篇——逻辑判定

    request_foods=["tomato","beaf","milk"] for elements in request_foods: ...

  9. eclipse、myeclipse写类时,自动生成注释

    在类的上边/**+enter自动生成注释. 设置方法:Window--Prefences--Java--Code Style--Code Templates--Comments--Types--Edi ...

  10. 大神教你Debian GNU/Linux 9.7 “Stretch” Live和安装镜像开放下载

    Debian项目团队于昨天发布了Debian GNU/Linux 9 "Stretch" 的第7个维护版本更新,重点修复了APT软件管理器中存在的安全漏洞.在敦促每位用户尽快升级系 ...