Greenplum 集群部署

最近开始接触Greenplum，线上也在使用了，感觉还不错，本次介绍一下集群的部署方法。那么Greenplum的架构如下：

（架构图来源网络）

简单来说GPDB是一个分布式数据库软件，其可以管理和处理分布在多个不同主机上的海量数据。对于GPDB来说，一个DB实例实际上是由多个独立的PostgreSQL实例组成的，它们分布在不同的物理主机上，协同工作，呈现给用户的是一个DB的效果。Master是GPDB系统的访问入口，其负责处理客户端的连接及SQL 命令、协调系统中的其他Instance(Segment)工作，Segment负责管理和处理用户数据。

环境准备：

操作系统：centos 6.3 64位

master 1台（架构图中的主节点），Standby 1台（架构图中的从节点），Segment 2台。共4台服务器。

     ip       主机名
10.69.30.251   mdw
10.36.25.152   smdw
10.36.25.250   sdw1
221.5.32.152   sdw2

1. 修改/etc/hosts文件，添加下面内容（4台服务器相同的配置）

10.69.30.251  mdw
10.36.25.152  smdw
10.36.25.250  sdw1
10.37.237.152  sdw2

2. 服务器关闭selinux，防火墙4台服务器相互开放，测试环境可以直接先关闭防火墙。（4台服务器）

3. 操作系统参数设置

vim /etc/sysctl.conf   （注：4台服务器）

kernel.shmmax =
kernel.shmmni =
kernel.shmall =
kernel.sem =
kernel.sysrq =
kernel.core_uses_pid =
kernel.msgmnb =
kernel.msgmax =
net.ipv4.tcp_syncookies =
net.ipv4.ip_forward =
net.ipv4.conf.default.accept_source_route =
net.ipv4.tcp_tw_recycle =
net.ipv4.tcp_max_syn_backlog =
net.ipv4.conf.all.arp_filter =
net.ipv4.conf.default.arp_filter =
net.core.netdev_max_backlog =
vm.overcommit_memory =
kernel.msgmni =
net.ipv4.ip_local_port_range =  

vim /etc/security/limits.conf  （注：4台服务器）

* soft nofile
* hard nofile
* soft nproc
* hard nproc 

我的操作系统是centos 6.3，所以还有一个文件要修改：那就是/etc/security/limits.d/90-nproc.conf （注：4台服务器都要修改）

修改后的内容如下：

* soft nproc
* hard nproc
* soft nofile
* hard nofile 

磁盘预读参数及 deadline算法修改 （注：4台服务器）

blockdev --setra  /dev/sda
echo deadline > /sys/block/sda/queue/scheduler

软件下载地址：https://network.pivotal.io/products/pivotal-gpdb

在Master主机上安装GP二进制文件，也就是主机名是mdw的服务器。（注：在master上安装即可，后面通过批量的方法安装剩下的服务器）

我下载的文件是greenplum-db-4.3.7.3-build-2-RHEL5-x86_64.zip

文件解压之是一个可执行文件，greenplum-db-4.3.7.3-build-2-RHEL5-x86_64.bin，直接执行安装。

./greenplum-db-4.3.7.3-build--RHEL5-x86_64.bin 

然后有些说明，一直回车，出现如下图按照提示安装即可完成。

在master上添加用户

useradd gpadmin

确保所有的服务器安装了ed软件，否则后面初始化集群就会报错。

yum install ed -y

准备文件，all_hosts ，seg_hosts放到/home/gpadmin。（用于批量安装软件以及后续集群的初始化）

[gpadmin@mdw ~]$ cat seg_hosts
sdw1
sdw2
[gpadmin@mdw ~]$ cat all_hosts
mdw
smdw
sdw1
sdw2
[gpadmin@mdw ~]$ 

建立节点服务器间的信任（我服务器已经打通了信任的，没有打通的会要求输入密码）

[root@mdw gpadmin]# gpssh-exkeys -f all_hosts
[STEP  of ] create local ID and authorize on local host
  ... /root/.ssh/id_rsa file exists ... key generation skipped

[STEP  of ] keyscan all hosts and update known_hosts file

[STEP  of ] authorize current user on remote hosts
  ... send to smdw
  ... send to sdw1
  ... send to sdw2

[STEP  of ] determine common authentication file content

[STEP  of ] copy authentication files to all remote hosts
  ... finished key exchange with smdw
  ... finished key exchange with sdw1
  ... finished key exchange with sdw2

[INFO] completed successfully

批量安装软件（GP）

cd /home/gpadmin/
source /usr/local/greenplum-db/greenplum_path.sh
gpseginstall -f all_hosts -u gpadmin -p gpadmin

最后出现如下提示代表批量安装成功。

:::: gpseginstall:greenplum_master:root-[INFO]:-SUCCESS -- Requested commands completed

检查是否安装完成

su - gpadmin
source /usr/local/greenplum-db/greenplum_path.sh
gpssh -f all_hosts -e ls -l $GPHOME

输出如下代表成功：

[ mdw] ls -l /usr/local/greenplum-db/.
[ mdw] total
[ mdw] drwxr-xr-x  gpadmin gpadmin    Feb    bin
[ mdw] drwxr-xr-x  gpadmin gpadmin    Feb    demo
[ mdw] drwxr-xr-x  gpadmin gpadmin    Feb    docs
[ mdw] drwxr-xr-x  gpadmin gpadmin    Feb    etc
[ mdw] drwxr-xr-x  gpadmin gpadmin    Feb    ext
[ mdw] -rw-r--r--  gpadmin gpadmin   Feb    GPDB-LICENSE.txt
[ mdw] -rw-rw-r--  gpadmin gpadmin     Aug  : greenplum_path.sh
[ mdw] drwxr-xr-x  gpadmin gpadmin    Feb    include
[ mdw] drwxr-xr-x  gpadmin gpadmin    Feb    lib
[ mdw] -rw-r--r--  gpadmin gpadmin  Feb    LICENSE.thirdparty
[ mdw] drwxr-xr-x  gpadmin gpadmin    Aug  : sbin
[ mdw] drwxr-xr-x  gpadmin gpadmin    Feb    share
[sdw1] ls -l /usr/local/greenplum-db/.
[sdw1] total
[sdw1] drwxr-xr-x  gpadmin gpadmin    Feb    bin
[sdw1] drwxr-xr-x  gpadmin gpadmin    Feb    demo
[sdw1] drwxr-xr-x  gpadmin gpadmin    Feb    docs
[sdw1] drwxr-xr-x  gpadmin gpadmin    Feb    etc
[sdw1] drwxr-xr-x  gpadmin gpadmin    Feb    ext
[sdw1] -rw-r--r--  gpadmin gpadmin   Feb    GPDB-LICENSE.txt
[sdw1] -rw-rw-r--  gpadmin gpadmin     Aug  : greenplum_path.sh
[sdw1] drwxr-xr-x  gpadmin gpadmin    Feb    include
[sdw1] drwxr-xr-x  gpadmin gpadmin    Feb    lib
[sdw1] -rw-r--r--  gpadmin gpadmin  Feb    LICENSE.thirdparty
[sdw1] drwxr-xr-x  gpadmin gpadmin    Aug  : sbin
[sdw1] drwxr-xr-x  gpadmin gpadmin    Feb    share
[smdw] ls -l /usr/local/greenplum-db/.
[smdw] total
[smdw] drwxr-xr-x  gpadmin gpadmin    Feb    bin
[smdw] drwxr-xr-x  gpadmin gpadmin    Feb    demo
[smdw] drwxr-xr-x  gpadmin gpadmin    Feb    docs
[smdw] drwxr-xr-x  gpadmin gpadmin    Feb    etc
[smdw] drwxr-xr-x  gpadmin gpadmin    Feb    ext
[smdw] -rw-r--r--  gpadmin gpadmin   Feb    GPDB-LICENSE.txt
[smdw] -rw-rw-r--  gpadmin gpadmin     Aug  : greenplum_path.sh
[smdw] drwxr-xr-x  gpadmin gpadmin    Feb    include
[smdw] drwxr-xr-x  gpadmin gpadmin    Feb    lib
[smdw] -rw-r--r--  gpadmin gpadmin  Feb    LICENSE.thirdparty
[smdw] drwxr-xr-x  gpadmin gpadmin    Aug  : sbin
[smdw] drwxr-xr-x  gpadmin gpadmin    Feb    share
[sdw2] ls -l /usr/local/greenplum-db/.
[sdw2] total
[sdw2] drwxr-xr-x  gpadmin gpadmin    Feb    bin
[sdw2] drwxr-xr-x  gpadmin gpadmin    Feb    demo
[sdw2] drwxr-xr-x  gpadmin gpadmin    Feb    docs
[sdw2] drwxr-xr-x  gpadmin gpadmin    Feb    etc
[sdw2] drwxr-xr-x  gpadmin gpadmin    Feb    ext
[sdw2] -rw-r--r--  gpadmin gpadmin   Feb    GPDB-LICENSE.txt
[sdw2] -rw-rw-r--  gpadmin gpadmin     Aug  : greenplum_path.sh
[sdw2] drwxr-xr-x  gpadmin gpadmin    Feb    include
[sdw2] drwxr-xr-x  gpadmin gpadmin    Feb    lib
[sdw2] -rw-r--r--  gpadmin gpadmin  Feb    LICENSE.thirdparty
[sdw2] drwxr-xr-x  gpadmin gpadmin    Aug  : sbin
[sdw2] drwxr-xr-x  gpadmin gpadmin    Feb    share

创建存储目录

master，standby 操作命令：

mkdir -p /data/master
chown -R gpadmin.gpadmin /data/master/

在master上用下面命令批量操作segment，建立目录，改权限

gpssh -f seg_hosts -e 'mkdir -p /data/primary'
gpssh -f seg_hosts -e 'mkdir -p /data/mirror'
gpssh -f seg_hosts -e 'chown -R gpadmin.gpadmin /data/primary'
gpssh -f seg_hosts -e 'chown -R gpadmin.gpadmin /data/mirror'

设置NTP同步

在每个Segment主机，以root登录编辑/etc/ntp.conf文件。设置第一个server参数指向Master主机，第二个server参数指向Standby主机。如下面：

server mdw prefer
server smdw

在Standby主机，以root登录编辑/etc/ntp.conf文件。设置第一个server参数指向Master主机，第二个参数指向数据中心的时间服务器。

server mdw prefer

在Master主机，使用NTP守护进程同步所有Segment主机的系统时钟。例如，使用gpssh来完成：

gpssh -f all_hosts -v -e 'ntpd'

输出如下代表成功：

[root@mdw gpadmin]# gpssh -f all_hosts -v -e 'ntpd'
[WARN] Reference default values as $MASTER_DATA_DIRECTORY/gpssh.conf could not be found
Using delaybeforesend 0.05 and prompt_validation_timeout 1.0

[Reset ...]
[INFO] login mdw
[INFO] login smdw
[INFO] login sdw1
[INFO] login sdw2
[ mdw] ntpd
[smdw] ntpd
[sdw1] ntpd
[sdw2] ntpd
[INFO] completed successfully

[Cleanup...]

要配置集群自动同步系统时钟，应开启ntpd服务，并设置为开机时自动运行，4台服务器都启动：

 /etc/init.d/ntpd start 

检查环境（我这里检查一下操作系统参数设置是否ok，网络，io，都可以检查的）

[root@mdw gpadmin]# gpcheck -f all_hosts -m mdw
:::: gpcheck:mdw:root-[INFO]:-dedupe hostnames
:::: gpcheck:mdw:root-[INFO]:-Detected platform: Generic Linux Cluster
:::: gpcheck:mdw:root-[INFO]:-generate data on servers
:::: gpcheck:mdw:root-[INFO]:-copy data files from servers
:::: gpcheck:mdw:root-[INFO]:-delete remote tmp files
:::: gpcheck:mdw:root-[INFO]:-Using gpcheck config file: /usr/local/greenplum-db/./etc/gpcheck.cnf
:::: gpcheck:mdw:root-[INFO]:-GPCHECK_NORMAL
:::: gpcheck:mdw:root-[INFO]:-gpcheck completing...

在master上进行下面的操作：

创建Greenplum数据库配置文件，以gpadmin用户登录

su - gpadmin
cp $GPHOME/docs/cli_help/gpconfigs/gpinitsystem_config   /home/gpadmin/gpinitsystem_config
chmod  gpinitsystem_config

设置所有必须的参数

ARRAY_NAME="EMC Greenplum DW"
SEG_PREFIX=gpseg
PORT_BASE=
declare -a DATA_DIRECTORY=(/data/primary)
MASTER_HOSTNAME=mdw
MASTER_DIRECTORY=/data/master
MASTER_PORT=
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=
ENCODING=UNICODE

设置可选参数

MIRROR_PORT_BASE=
REPLICATION_PORT_BASE=
declare -a MIRROR_DATA_DIRECTORY=(/data/mirror)

运行初始化工具初始化数据库

su - gpadmin
source /usr/local/greenplum-db/greenplum_path.sh
gpinitsystem -c gpinitsystem_config -h seg_hosts

正常情况下初始化完成以后最后的日志没有任何报错，而且gp进程也是启动了的。

初始化日志:

:::: gpinitsystem:mdw:gpadmin-[INFO]:-Review options for gpinitstandby
:::: gpinitsystem:mdw:gpadmin-[INFO]:-------------------------------------------------------
:::: gpinitsystem:mdw:gpadmin-[INFO]:-The Master /data/master/gpseg-/pg_hba.conf post gpinitsystem
:::: gpinitsystem:mdw:gpadmin-[INFO]:-has been configured to allow all hosts within this new
:::: gpinitsystem:mdw:gpadmin-[INFO]:-array to intercommunicate. Any hosts external to this
:::: gpinitsystem:mdw:gpadmin-[INFO]:-new array must be explicitly added to this file
:::: gpinitsystem:mdw:gpadmin-[INFO]:-Refer to the Greenplum Admin support guide which is
:::: gpinitsystem:mdw:gpadmin-[INFO]:-located in the /usr/local/greenplum-db/./docs directory
:::: gpinitsystem:mdw:gpadmin-[INFO]:-------------------------------------------------------

启动进程：

[root@mdw gpadmin]# ps -ef | grep postgres
gpadmin          : ?        :: /usr/local/greenplum-db-4.3.7.3/bin/postgres -D /data/master/gpseg- -p  -b  -z  --silent-mode=true -i -M master -C - -x  -E
gpadmin      : ?        :: postgres: port  , master logger process
gpadmin      : ?        :: postgres: port  , stats collector process
gpadmin      : ?        :: postgres: port  , writer process
gpadmin      : ?        :: postgres: port  , checkpoint process
gpadmin      : ?        :: postgres: port  , seqserver process
gpadmin      : ?        :: postgres: port  , ftsprobe process
gpadmin      : ?        :: postgres: port  , sweeper process
root         : pts/    :: grep postgres
[root@mdw gpadmin]# 

现在只有1个master，2个segment，没有standby，那么接下来把standby加入集群。

在master服务器上执行

gpinitstandby -s smdw

输出如下：

[gpadmin@mdw ~]$ gpinitstandby -s smdw
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Validating environment and parameters for standby initialization...
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Checking for filespace directory /data/master/gpseg- on smdw
:::: gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master initialization parameters
:::: gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master hostname               = mdw
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master data directory         = /data/master/gpseg-
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master port                   =
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master hostname       = smdw
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master port           =
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master data directory = /data/master/gpseg-
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum update system catalog         = On
:::: gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
:::: gpinitstandby:mdw:gpadmin-[INFO]:- Filespace locations
:::: gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
:::: gpinitstandby:mdw:gpadmin-[INFO]:-pg_system -> /data/master/gpseg-
Do you want to continue with standby master initialization? Yy|Nn (default=N):
> y
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Syncing Greenplum Database extensions to standby
:::: gpinitstandby:mdw:gpadmin-[INFO]:-The packages on smdw are consistent.
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Adding standby master to catalog...
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Database catalog updated successfully.
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Updating pg_hba.conf file...
:::: gpinitstandby:mdw:gpadmin-[INFO]:-pg_hba.conf files updated successfully.
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Updating filespace flat files...
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Filespace flat file updated successfully.
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Starting standby master
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Checking if standby master is running on host: smdw  in directory: /data/master/gpseg-
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Cleaning up pg_hba.conf backup files...
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Backup files of pg_hba.conf cleaned up successfully.
:::: gpinitstandby:mdw:gpadmin-[INFO]:-Successfully created standby master on smdw

设置环境变量，master，standby都设置。

1.  以gpadmin用户登录：

su - gpadmin

2.  打开用户启动文件(如.bashrc)。比如：

vim ~/.bashrc

3.  添加新行用以加载greenplum_path.sh文件和设置MASTER_DATA_DIRECTORY环境变量。例如：

source /usr/local/greenplum-db/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/data/master/gpseg-

启动和停止数据库测试是否能正常启动和关闭，命令如下

gpstart
gpstop 

到此 greenplum 就部署完成了。下面进行一些简单的测试。

登录数据库：psql -d postgres

建表，插入，查询

postgres=# create table student ( no int primary key,student_name varchar(40),age int);
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "student_pkey" for table "student"
CREATE TABLE
postgres=# insert into student values(1,'yayun',18);
INSERT 0 1
postgres=# select * from student;
 no | student_name | age
----+--------------+-----
  1 | yayun        |  18
(1 row)