CentOS7 上搭建 CDH(6.3.0)
这里以四台节点搭建
IP | HostName | OS |
---|---|---|
192.168.8.5 | h5(主) | CentOS 7.5 |
192.168.8.6 | h6(从) | CentOS 7.5 |
192.168.8.7 | h7(从) | CentOS 7.5 |
192.168.8.8 | h8(从) | CentOS 7.5 |
一、节点基本配置
1.yum 源,这里用阿里的(全部节点)
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo
yum makecache
2.编辑 hosts 文件(全部节点)
https://www.cloudera.com/documentation/enterprise/6/6.3/topics/configure_network_names.html
vim /etc/hosts 192.168.8.5 h5
192.168.8.6 h6
192.168.8.7 h7
192.168.8.8 h8
3.关闭防火墙(全部节点)
https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_cdh_disable_iptables.html
systemctl disable firewalld
systemctl stop firewalld
4.关闭 SELinux(全部节点)
https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_cdh_disable_selinux.html
vim /etc/selinux/config SELINUX=disabled # sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config && setenforce
5.安装 NTP 服务(全部节点,其中一台为服务器端,其余为客户端)
https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_cdh_enable_ntp.html
这里以 h5 为 ntp 服务器,其它节点向 h5 同步时间。
具体搭建参考:https://www.cnblogs.com/jhxxb/p/10579816.html
6.安装 Python2.7(使用 HUE 的节点,不使用 HUE 不用安装)
https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_python_27.html
CentOS7 自带的就是 Python2.7,不用安装
yum install python27
source /opt/rh/python27/enable
python --version
二、安装 Cloudera Manager 和 CDH
1.安装 CDH 的 yum 源(全部节点)
https://www.cloudera.com/documentation/enterprise/6/6.3/topics/configure_cm_repo.html
curl -o /etc/yum.repos.d/cloudera-manager.repo https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/cloudera-manager.repo
rpm --import https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPM-GPG-KEY-cloudera
2.安装 JDK(全部节点)
https://www.cloudera.com/documentation/enterprise/6/6.3/topics/cdh_ig_jdk_installation.html
必须安装在 /usr/java/ 目录中,这里使用 CDH 源中带的 JDK
yum install oracle-j2sdk1.
yum 下载安装可能会很慢,可以下载后本地安装
https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPMS/x86_64/
yum install /home/oracle-j2sdk1.-1.8.+update181-.x86_64.rpm -y
3.安装 Cloudera Manager Server(全部节点)
https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_cm_server.html
主节点
yum install cloudera-manager-daemons cloudera-manager-server -y
从节点(在集群部署的时候会自动安装,但下载很慢,可以先提前本地安装)
yum install cloudera-manager-daemons cloudera-manager-agent -y
同样 yum 下载安装可能会很慢,可以下载后本地安装
4.安装数据库,这里选用 MySQL(主节点,也可以放在其它节点)
https://www.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_mysql.html
MySQL 数据库安装参考:https://www.cnblogs.com/jhxxb/p/11156902.html
安装完成后先修改配置
vim /etc/my.cnf
my.cnf 文件
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0 key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1 max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M #log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log #In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1 binlog_format = mixed read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M # InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M [mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid sql_mode=STRICT_ALL_TABLES
然后再启动,新建 CDH 所需要的库:https://www.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_mysql.html#concept_dsg_3mq_bl
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; -- 也可以直接用 root 用户
GRANT ALL ON scm.* TO 'cdh6'@'%' IDENTIFIED BY '';
GRANT ALL ON amon.* TO 'cdh6'@'%' IDENTIFIED BY '';
GRANT ALL ON rman.* TO 'cdh6'@'%' IDENTIFIED BY '';
GRANT ALL ON hue.* TO 'cdh6'@'%' IDENTIFIED BY '';
GRANT ALL ON metastore.* TO 'cdh6'@'%' IDENTIFIED BY '';
GRANT ALL ON sentry.* TO 'cdh6'@'%' IDENTIFIED BY '';
GRANT ALL ON nav.* TO 'cdh6'@'%' IDENTIFIED BY '';
GRANT ALL ON navms.* TO 'cdh6'@'%' IDENTIFIED BY '';
GRANT ALL ON oozie.* TO 'cdh6'@'%' IDENTIFIED BY '';
最后安装 MySQL 驱动
https://www.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_mysql.html#cmig_topic_5_5_3
yum install wget -y
# 下载
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
# 解压
tar -zxf mysql-connector-java-5.1..tar.gz
# 放到指定目录
mkdir -p /usr/share/java/
cp mysql-connector-java-5.1./mysql-connector-java-5.1.-bin.jar /usr/share/java/mysql-connector-java.jar # 删除
rm -rf mysql-connector-java-5.1. mysql-connector-java-5.1..tar.gz
5.设置 Cloudera Manager 数据库(主节点)
https://www.cloudera.com/documentation/enterprise/6/6.3/topics/prepare_cm_database.html
格式:/opt/cloudera/cm/schema/scm_prepare_database.sh <databaseType> <databaseName> <databaseUser>
官方文档推荐的是每个库都有独立的用户,这里为了方便统一用 root 用户。
当 MySQL 或 MariaDB 与 Cloudera Manager Server 位于同一节点时:/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root
当 MySQL 或 MariaDB 在其它节点上时:/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h db01.example.com --scm-host cm01.example.com scm root
6.安装 CDH 和其他软件(主节点)
https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_software_cm_wizard.html
systemctl start cloudera-scm-server
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log # 看到如下日志就可以访问 WEB 端了:http://<server_host>:7180,用户名和密码默认都是 admin
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
子节点启动
systemctl start cloudera-scm-agent
http://wangshubing.com/2019/01/24/CentOS7.5,CDH6%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/
https://mp.weixin.qq.com/s/c7c0wOHb7wPr6Kbg3MTU1A
CentOS7 上搭建 CDH(6.3.0)的更多相关文章
- 《从0到1学习Flink》—— Mac 上搭建 Flink 1.6.0 环境并构建运行简单程序入门
准备工作 1.安装查看 Java 的版本号,推荐使用 Java 8. 安装 Flink 2.在 Mac OS X 上安装 Flink 是非常方便的.推荐通过 homebrew 来安装. brew in ...
- centos7上搭建开源系统jforum
centos7上搭建好tomcat,mysql; 将 jforum-2.6.2.war放到tomcat目录的webapps下: 启动tomcat,./startup.sh ,查看webapp下jfor ...
- 用开源软件TrinityCore在Debian 10上搭建魔兽世界8.3.0.34220的服务器
用开源软件TrinityCore在Debian 10上搭建魔兽世界8.3.0.34220的服务器 TrinityCore是魔兽世界(World of Warcraft)的开源的服务端.目前支持魔兽的3 ...
- Kafka(二)CentOS7.5搭建Kafka2.11-1.1.0集群与简单测试
一.下载 下载地址: http://kafka.apache.org/downloads.html 我这里下载的是Scala 2.11对应的 kafka_2.11-1.1.0.tgz 二.kaf ...
- [深度学习] centos7上搭建基于Anaconda3的caffe+pycaffe环境(python3.6)
本文记录从零开始在CentOS7.x系统上搭建Caffe深度学习平台,并配置pycaffe环境.(由于在虚拟机上搭建,所以为CPU_ONLY模式) 1.选择CentOS7 mini版镜像安装虚拟机 镜 ...
- CentOS7.5搭建Kafka2.11-1.1.0集群与简单测试
一.下载 下载地址: http://kafka.apache.org/downloads.html 我这里下载的是Scala 2.11对应的 kafka_2.11-1.1.0.tgz 二.集群规 ...
- Centos7上搭建redis主从
1. 节点(服务器)数量说明 按照redis官方建议:salve和master的数量按照2n+1台服务器(1台master节点,2n台slave节点) 有兴趣的可以了解下redis的master选举机 ...
- centos7上搭建http服务器以及设置目录访问
参考文献:http://www.jb51.net/article/137596.htm,原文摘抄如下,并根据具体需要作了相应的修改. 步骤: 1. 安装httpd服务 sudo yum install ...
- 在centos7上搭建博客之小白教程~
原理 http使用方法一编译安装,php模块打入方式实现. 软件版本 在本次实验中,我们需要用到的软件版本如下: apr-1.6.2 apr-util-1.6.0 httpd-2.4.28 maria ...
随机推荐
- Notepad++快捷键及使用技巧
常用快捷键: CTRL+Q 注释/取消注释 用Notepad++写代码,要是有一些重复的代码想copy一下,还真不容易,又得动用鼠标,巨烦人....有木有简单的方法呢,确实还是有的不过也不算太好用.主 ...
- redis——搭建
https://blog.csdn.net/sinat_29699167/article/details/79699200 Django使用Redis进行缓存详细最全流程 https://blog.c ...
- GitHub新手使用篇
如何使用GitHub 未完结 目录: ISSUE总汇总: Issue1:GitHub的注册和使用? 答:(1)注册GitHub :https://github.com/.需要填用户名.邮箱.密码,值得 ...
- QTP10破解步骤(亲试有效)
QTP10破解步骤: 1.安装qtp,一路默认下来(一定要默认安装目录),一直到要求输入License的界面 2.拷贝mgn-mqt82.exe到C:\Program Files\Mercury In ...
- (备忘)Java Map 遍历
//最常规的一种遍历方法,最常规就是最常用的,虽然不复杂,但很重要,这是我们最熟悉的,就不多说了!! public static void work(Map<String, Student> ...
- sql 分页查询 (每次6行 )
-- 对比 数据 是否 相同 select * from [dbo].[ProjecrInfo] where Project_state='已审核' -- 查询 已经 审核 有多少数据 -- 每次 ...
- 解决Ubuntu下ssh无法启动
Ubuntu ssh一直无法启动. 通过 systemctl status ssh.service 查看到的错误是 Dec 16 13:35:22 iZm5eckxl2tqyka9eoe7b3Z ...
- GDB调试器教程
启动和退出GDBGDB(GNU Project Debugger)几乎适用于所有类Unix系统,小巧方便且不失功能强大,Linux/Unix程序员经常用它来调试程序. 总的来说有几下几种方法启动GDB ...
- Linux之ln文件创建链接
ln命令用来为文件创建链接,链接类型分为硬链接和软链接(符号链接)两种 1)软连接和Windows系统中的快捷方式有点类似 2)硬链接,相当于多了一个文件名指向同一块内存空间,目录无法创建硬链接,不可 ...
- BZOJ5017 [Snoi2017]炸弹[线段树优化建边+scc缩点+DAG上DP/线性递推]
方法一: 朴素思路:果断建图,每次二分出一个区间然后要向这个区间每个点连有向边,然后一个环的话是可以互相引爆的,缩点之后就是一个DAG,求每个点出发有多少可达点. 然后注意两个问题: 上述建边显然$n ...