gp数据库停止

greenplum是2(master)+7(segment)的集群规模

系统刚准备上线，是用来做统计数据库的，正在帮忙一个hadoop集群核对其数据的准确性，在这个greenplum库中入了清单数据

后检查分析是部分建表语句存在问题，没有指定字段做分布键，也没有指定其是随机分布，导致默认为第一个字段做为分布键导致数据倾斜。

发现数据库非常慢，几乎是不可用，检查greenplum的状态情况

1、检查greenplum数据库的状态

gpadmin@mdw:~> gpstate 20150227:15:20:13:007202 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: 20150227:15:20:13:007202 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.3.1 build 1' 20150227:15:20:13:007202 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.3.1 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Oct 10 2014 14:31:50' 20150227:15:20:13:007202 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20150227:15:20:13:007202 gpstate:mdw:gpadmin-[INFO]:-Gathering data from segments... ...... 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:-Greenplum instance status summary 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Master instance = Active 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Master standby = smdw 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Standby master state = Standby host passive 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total segment instance count from metadata = 56 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Primary Segment Status 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total primary segments = 28 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total primary segment valid (at master) = 24 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[WARNING]:-Total primary segment failures (at master) = 4 <<<<<<<< 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid files missing = 0 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid files found = 28 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 28 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of /tmp lock files missing = 0 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of /tmp lock files found = 28 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[WARNING]:-Total number postmaster processes missing = 4 <<<<<<<< 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number postmaster processes found = 24 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Mirror Segment Status 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total mirror segments = 28 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total mirror segment valid (at master) = 21 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[WARNING]:-Total mirror segment failures (at master) = 7 <<<<<<<< 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid files missing = 0 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid files found = 28 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 28 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of /tmp lock files missing = 0 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of /tmp lock files found = 28 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[WARNING]:-Total number postmaster processes missing = 4 <<<<<<<< 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number postmaster processes found = 24 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[WARNING]:-Total number mirror segments acting as primary segments = 4 <<<<<<<< 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number mirror segments acting as mirror segments = 24 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:-----------------------------------------------------

2、检查数据库服务器的情况

gpadmin@mdw:/> gpssh -h sdw1 -h sdw2 -h sdw3 -h sdw4 -h sdw5 -h sdw6 -h sdw7 "df -hT" [sdw4] df: "/root/.gvfs": 权限不够 [sdw4] 文件系统类型容量已用可用已用% 挂载瀿 [sdw4] /dev/sda2 ext3 99G 5.7G 88G 7% / [sdw4] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw4] tmpfs tmpfs 95G 100K 95G 1% /dev/shm [sdw4] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw4] /dev/sda5 ext3 197G 188M 187G 1% /home [sdw4] /dev/sdb xfs 4.6T 3.8T 865G 82% /data [sdw5] 文件系统类型容量已用可用已用% 挂载瀿 [sdw5] /dev/sda2 ext3 60G 5.5G 51G 10% / [sdw5] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw5] tmpfs tmpfs 32G 88K 32G 1% /dev/shm [sdw5] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw5] /dev/sda5 ext3 785G 197M 745G 1% /home [sdw5] /dev/sdb xfs 4.6T 2.4T 2.2T 53% /data [sdw6] df: "/root/.gvfs": 权限不够 [sdw6] 文件系统类型容量已用可用已用% 挂载瀿 [sdw6] /dev/sda2 ext3 99G 910M 93G 1% / [sdw6] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw6] tmpfs tmpfs 47G 100K 47G 1% /dev/shm [sdw6] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw6] /dev/sda5 ext3 197G 188M 187G 1% /home [sdw6] /dev/sda3 ext3 63G 5.0G 55G 9% /usr [sdw6] /dev/sdb xfs 4.6T 4.5T 93G 99% /data [sdw6] /dev/sr0 iso9660 3.1G 3.1G 0 100% /media/SLES-11-SP2-DVD-x86_6407551 [sdw7] df: "/root/.gvfs": 权限不够 [sdw7] 文件系统类型容量已用可用已用% 挂载瀿 [sdw7] /dev/sda2 ext3 60G 440M 56G 1% / [sdw7] devtmpfs devtmpfs 32G 244K 32G 1% /dev [sdw7] tmpfs tmpfs 95G 112K 95G 1% /dev/shm [sdw7] /dev/sda1 ext3 9.9G 180M 9.2G 2% /boot [sdw7] /dev/sda5 ext3 197G 188M 187G 1% /home [sdw7] /dev/sda6 ext3 9.9G 264M 9.1G 3% /opt [sdw7] /dev/sda8 ext3 9.9G 151M 9.2G 2% /srv [sdw7] /dev/sda7 ext3 9.9G 162M 9.2G 2% /tmp [sdw7] /dev/sda9 ext3 40G 4.8G 33G 13% /usr [sdw7] /dev/sda10 ext3 9.9G 358M 9.0G 4% /var [sdw7] /dev/sdb xfs 4.6T 3.7T 943G 80% /data [sdw1] df: "/root/.gvfs": 权限不够 [sdw1] 文件系统类型容量已用可用已用% 挂载瀿 [sdw1] /dev/sda1 ext3 99G 5.7G 88G 7% / [sdw1] devtmpfs devtmpfs 32G 444K 32G 1% /dev [sdw1] tmpfs tmpfs 95G 100K 95G 1% /dev/shm [sdw1] /dev/sda2 ext3 7.9G 216M 7.3G 3% /boot [sdw1] /dev/sda3 ext3 197G 188M 187G 1% /home [sdw1] /dev/sdb xfs 4.6T 3.6T 1.1T 78% /data [sdw2] df: "/root/.gvfs": 权限不够 [sdw2] 文件系统类型容量已用可用已用% 挂载瀿 [sdw2] /dev/sda1 ext3 99G 5.7G 88G 7% / [sdw2] devtmpfs devtmpfs 32G 444K 32G 1% /dev [sdw2] tmpfs tmpfs 95G 100K 95G 1% /dev/shm [sdw2] /dev/sda2 ext3 7.9G 216M 7.3G 3% /boot [sdw2] /dev/sda3 ext3 197G 188M 187G 1% /home [sdw2] /dev/sdb xfs 4.6T 4.6T 2.0G 100% /data [sdw3] df: "/root/.gvfs": 权限不够 [sdw3] 文件系统类型容量已用可用已用% 挂载瀿 [sdw3] /dev/sda2 ext3 60G 5.5G 51G 10% / [sdw3] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw3] tmpfs tmpfs 32G 100K 32G 1% /dev/shm [sdw3] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw3] /dev/sda5 ext3 785G 197M 745G 1% /home [sdw3] /dev/sdb xfs 4.6T 3.8T 856G 82% /data 发现segment中的sdw2和sdw6的数据空间/data目录的收益率已经达到100%。

3、数据库上删除清单表

用psql连接数据库，执行drop table 表名，执行后，等待大半天也没反应，使用gpssh -f检查服务器的io使用情况，segment服务器的IO没有读写操作，证明数据库已经没办法分发命令下去。

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

4、分别登陆sdw2和sdw6

#cd /data/primary/gpseg4/gp_log

#ls -ltrh

目录下有gpdb-日期.csv文件，直接执行

#rm -rf *.csv

每个gpseg下的gp_log都执行同样的删除操作，执行完后，空间释放得非常少。

=> df -hT [sdw4] df: "/root/.gvfs": 权限不够 [sdw4] 文件系统类型容量已用可用已用% 挂载点 [sdw4] /dev/sda2 ext3 99G 5.7G 88G 7% / [sdw4] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw4] tmpfs tmpfs 95G 100K 95G 1% /dev/shm [sdw4] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw4] /dev/sda5 ext3 197G 188M 187G 1% /home [sdw4] /dev/sdb xfs 4.6T 3.7T 868G 82% /data [sdw5] 文件系统类型容量已用可用已用% 挂载点 [sdw5] /dev/sda2 ext3 60G 5.5G 51G 10% / [sdw5] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw5] tmpfs tmpfs 32G 88K 32G 1% /dev/shm [sdw5] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw5] /dev/sda5 ext3 785G 197M 745G 1% /home [sdw5] /dev/sdb xfs 4.6T 2.4T 2.2T 53% /data [sdw6] df: "/root/.gvfs": 权限不够 [sdw6] 文件系统类型容量已用可用已用% 挂载点 [sdw6] /dev/sda2 ext3 99G 911M 93G 1% / [sdw6] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw6] tmpfs tmpfs 47G 100K 47G 1% /dev/shm [sdw6] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw6] /dev/sda5 ext3 197G 188M 187G 1% /home [sdw6] /dev/sda3 ext3 63G 5.0G 55G 9% /usr [sdw6] /dev/sdb xfs 4.6T 4.5T 96G 98% /data [sdw6] /dev/sr0 iso9660 3.1G 3.1G 0 100% /media/SLES-11-SP2-DVD-x86_6407551 [sdw7] df: "/root/.gvfs": 权限不够 [sdw7] 文件系统类型容量已用可用已用% 挂载点 [sdw7] /dev/sda2 ext3 60G 440M 56G 1% / [sdw7] devtmpfs devtmpfs 32G 244K 32G 1% /dev [sdw7] tmpfs tmpfs 95G 112K 95G 1% /dev/shm [sdw7] /dev/sda1 ext3 9.9G 180M 9.2G 2% /boot [sdw7] /dev/sda5 ext3 197G 188M 187G 1% /home [sdw7] /dev/sda6 ext3 9.9G 264M 9.1G 3% /opt [sdw7] /dev/sda8 ext3 9.9G 151M 9.2G 2% /srv [sdw7] /dev/sda7 ext3 9.9G 162M 9.2G 2% /tmp [sdw7] /dev/sda9 ext3 40G 4.8G 33G 13% /usr [sdw7] /dev/sda10 ext3 9.9G 359M 9.0G 4% /var [sdw7] /dev/sdb xfs 4.6T 3.7T 945G 80% /data [sdw1] df: "/root/.gvfs": 权限不够 [sdw1] 文件系统类型容量已用可用已用% 挂载点 [sdw1] /dev/sda1 ext3 99G 5.7G 88G 7% / [sdw1] devtmpfs devtmpfs 32G 444K 32G 1% /dev [sdw1] tmpfs tmpfs 95G 100K 95G 1% /dev/shm [sdw1] /dev/sda2 ext3 7.9G 216M 7.3G 3% /boot [sdw1] /dev/sda3 ext3 197G 188M 187G 1% /home [sdw1] /dev/sdb xfs 4.6T 3.6T 1.1T 78% /data [sdw2] df: "/root/.gvfs": 权限不够 [sdw2] 文件系统类型容量已用可用已用% 挂载点 [sdw2] /dev/sda1 ext3 99G 5.7G 88G 7% / [sdw2] devtmpfs devtmpfs 32G 444K 32G 1% /dev [sdw2] tmpfs tmpfs 95G 100K 95G 1% /dev/shm [sdw2] /dev/sda2 ext3 7.9G 216M 7.3G 3% /boot [sdw2] /dev/sda3 ext3 197G 188M 187G 1% /home [sdw2] /dev/sdb xfs 4.6T 4.6T 21G 100% /data [sdw3] df: "/root/.gvfs": 权限不够 [sdw3] 文件系统类型容量已用可用已用% 挂载点 [sdw3] /dev/sda2 ext3 60G 5.5G 51G 10% / [sdw3] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw3] tmpfs tmpfs 32G 100K 32G 1% /dev/shm [sdw3] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw3] /dev/sda5 ext3 785G 198M 745G 1% /home [sdw3] /dev/sdb xfs 4.6T 3.8T 859G 82% /data [smdw] df: "/root/.gvfs": 权限不够 [smdw] 文件系统类型容量已用可用已用% 挂载点 [smdw] /dev/sda2 ext3 60G 5.8G 51G 11% / [smdw] devtmpfs devtmpfs 32G 448K 32G 1% /dev [smdw] tmpfs tmpfs 32G 100K 32G 1% /dev/shm [smdw] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [smdw] /dev/sda5 ext3 40G 23G 16G 60% /home [smdw] /dev/sda6 xfs 757G 3.3G 754G 1% /data [smdw] /dev/sr0 iso9660 3.1G 3.1G 0 100% /media/SLES-11-SP2-DVD-x86_6407551 [ mdw] df: "/root/.gvfs": 权限不够 [ mdw] 文件系统类型容量已用可用已用% 挂载点 [ mdw] /dev/sda2 ext3 60G 6.4G 50G 12% / [ mdw] devtmpfs devtmpfs 32G 456K 32G 1% /dev [ mdw] tmpfs tmpfs 32G 100K 32G 1% /dev/shm [ mdw] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [ mdw] /dev/sda5 ext3 40G 281M 38G 1% /home [ mdw] /dev/sda6 xfs 757G 6.4G 751G 1% /data

5、停数据库

执行gpstop

报：-gpstop failed(Reason='FATAL: the database system is shutting down') exiting ...

gpstop停不下来。

6、检查进程

gpssh -h sdw1 -h sdw2 -h sdw3 -h sdw4 -h sdw5 -h sdw6 -h sdw7 -h mdw -h smdw "ps -ef |grep postgres"

数据库segment上进程没有停。下面是一台segment上的进程：

[sdw4] gpadmin 15168 31843 0 Feb26 ? 00:00:14 postgres: port 40000, cems cems 198.168.11.11(52166) con6959 seg12 idle in transaction [sdw4] gpadmin 15170 31838 0 Feb26 ? 00:00:15 postgres: port 40001, cems cems 198.168.11.11(38065) con6959 seg13 idle in transaction [sdw4] gpadmin 15172 31841 0 Feb26 ? 00:00:16 postgres: port 40002, cems cems 198.168.11.11(3175) con6959 seg14 idle in transaction [sdw4] gpadmin 15174 31837 0 Feb26 ? 00:00:16 postgres: port 40003, cems cems 198.168.11.11(31152) con6959 seg15 idle in transaction [sdw4] gpadmin 15176 31843 0 Feb26 ? 00:00:04 postgres: port 40000, cems cems 198.168.11.11(52194) con6959 seg12 idle [sdw4] gpadmin 15178 31838 0 Feb26 ? 00:00:04 postgres: port 40001, cems cems 198.168.11.11(38093) con6959 seg13 idle [sdw4] gpadmin 15180 31841 0 Feb26 ? 00:00:04 postgres: port 40002, cems cems 198.168.11.11(3203) con6959 seg14 idle [sdw4] gpadmin 15182 31837 0 Feb26 ? 00:00:04 postgres: port 40003, cems cems 198.168.11.11(31180) con6959 seg15 idle [sdw4] gpadmin 15204 31843 0 Feb26 ? 00:00:16 postgres: port 40000, cems cems 198.168.11.11(52298) con6949 seg12 idle in transaction [sdw4] gpadmin 15206 31838 0 Feb26 ? 00:00:15 postgres: port 40001, cems cems 198.168.11.11(38197) con6949 seg13 idle in transaction [sdw4] gpadmin 15208 31841 0 Feb26 ? 00:00:16 postgres: port 40002, cems cems 198.168.11.11(3307) con6949 seg14 idle in transaction [sdw4] gpadmin 15210 31837 0 Feb26 ? 00:00:15 postgres: port 40003, cems cems 198.168.11.11(31284) con6949 seg15 idle in transaction [sdw4] gpadmin 31836 1 0 Feb04 ? 00:00:04 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/mirror/gpseg27 -p 50003 -b 57 -z 28 --silent-mode=true -i -M quiescent -C 27 [sdw4] gpadmin 31837 1 0 Feb04 ? 00:10:13 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/primary/gpseg15 -p 40003 -b 17 -z 28 --silent-mode=true -i -M quiescent -C 15 [sdw4] gpadmin 31838 1 0 Feb04 ? 00:10:11 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/primary/gpseg13 -p 40001 -b 15 -z 28 --silent-mode=true -i -M quiescent -C 13 [sdw4] gpadmin 31839 1 0 Feb04 ? 00:00:05 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/mirror/gpseg5 -p 50001 -b 35 -z 28 --silent-mode=true -i -M quiescent -C 5 [sdw4] gpadmin 31840 1 0 Feb04 ? 00:00:03 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/mirror/gpseg2 -p 50002 -b 32 -z 28 --silent-mode=true -i -M quiescent -C 2 [sdw4] gpadmin 31841 1 0 Feb04 ? 00:10:20 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/primary/gpseg14 -p 40002 -b 16 -z 28 --silent-mode=true -i -M quiescent -C 14 [sdw4] gpadmin 31842 1 0 Feb04 ? 00:00:04 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/mirror/gpseg8 -p 50000 -b 38 -z 28 --silent-mode=true -i -M quiescent -C 8 [sdw4] gpadmin 31843 1 0 Feb04 ? 00:10:26 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/primary/gpseg12 -p 40000 -b 14 -z 28 --silent-mode=true -i -M quiescent -C 12 [sdw4] gpadmin 31844 31841 0 Feb04 ? 00:01:15 postgres: port 40002, logger process [sdw4] gpadmin 31845 31838 0 Feb04 ? 00:01:21 postgres: port 40001, logger process [sdw4] gpadmin 31846 31839 0 Feb04 ? 00:01:07 postgres: port 50001, logger process [sdw4] gpadmin 31847 31836 0 Feb04 ? 00:01:13 postgres: port 50003, logger process [sdw4] gpadmin 31848 31843 0 Feb04 ? 00:01:19 postgres: port 40000, logger process [sdw4] gpadmin 31849 31842 0 Feb04 ? 00:01:07 postgres: port 50000, logger process [sdw4] gpadmin 31850 31837 0 Feb04 ? 00:01:25 postgres: port 40003, logger process [sdw4] gpadmin 31851 31840 0 Feb04 ? 00:01:04 postgres: port 50002, logger process [sdw4] gpadmin 31868 31836 0 Feb04 ? 00:08:55 postgres: port 50003, mirror process [sdw4] gpadmin 31871 31837 0 Feb04 ? 00:09:13 postgres: port 40003, primary process [sdw4] gpadmin 31873 31841 0 Feb04 ? 00:08:56 postgres: port 40002, primary process [sdw4] gpadmin 31874 31868 0 Feb04 ? 01:10:10 postgres: port 50003, mirror receiver process [sdw4] gpadmin 31876 31868 0 Feb04 ? 00:34:17 postgres: port 50003, mirror consumer process [sdw4] gpadmin 31877 31868 0 Feb04 ? 00:12:24 postgres: port 50003, mirror consumer writer process [sdw4] gpadmin 31878 31868 0 Feb04 ? 00:36:29 postgres: port 50003, mirror consumer append only process [sdw4] gpadmin 31879 31838 0 Feb04 ? 00:08:55 postgres: port 40001, primary process [sdw4] gpadmin 31881 31868 0 Feb04 ? 00:07:44 postgres: port 50003, mirror sender ack process [sdw4] gpadmin 31882 31868 0 Feb04 ? 00:00:03 postgres: port 50003, mirror verification process [sdw4] gpadmin 31883 31871 0 Feb04 ? 00:07:12 postgres: port 40003, primary receiver ack process [sdw4] gpadmin 31885 31871 0 Feb04 ? 01:15:22 postgres: port 40003, primary sender process [sdw4] gpadmin 31886 31871 0 Feb04 ? 00:07:00 postgres: port 40003, primary consumer ack process [sdw4] gpadmin 31887 31871 0 Feb04 ? 00:15:41 postgres: port 40003, primary recovery process [sdw4] gpadmin 31888 31842 0 Feb04 ? 00:09:02 postgres: port 50000, mirror process [sdw4] gpadmin 31889 31871 0 Feb04 ? 00:01:08 postgres: port 40003, primary verification process [sdw4] gpadmin 31892 31873 0 Feb04 ? 00:07:19 postgres: port 40002, primary receiver ack process [sdw4] gpadmin 31893 31873 0 Feb04 ? 01:10:48 postgres: port 40002, primary sender process [sdw4] gpadmin 31894 31873 0 Feb04 ? 00:06:47 postgres: port 40002, primary consumer ack process [sdw4] gpadmin 31895 31873 0 Feb04 ? 00:15:34 postgres: port 40002, primary recovery process [sdw4] gpadmin 31896 31873 0 Feb04 ? 00:01:13 postgres: port 40002, primary verification process [sdw4] gpadmin 31898 31839 0 Feb04 ? 00:09:08 postgres: port 50001, mirror process [sdw4] gpadmin 31900 31840 0 Feb04 ? 00:09:03 postgres: port 50002, mirror process [sdw4] gpadmin 31901 31879 0 Feb04 ? 00:07:05 postgres: port 40001, primary receiver ack process [sdw4] gpadmin 31902 31879 0 Feb04 ? 01:07:19 postgres: port 40001, primary sender process [sdw4] gpadmin 31903 31879 0 Feb04 ? 00:09:53 postgres: port 40001, primary consumer ack process [sdw4] gpadmin 31904 31879 0 Feb04 ? 00:15:36 postgres: port 40001, primary recovery process [sdw4] gpadmin 31905 31879 0 Feb04 ? 00:01:13 postgres: port 40001, primary verification process [sdw4] gpadmin 31911 31888 0 Feb04 ? 01:11:45 postgres: port 50000, mirror receiver process [sdw4] gpadmin 31912 31888 0 Feb04 ? 00:34:47 postgres: port 50000, mirror consumer process [sdw4] gpadmin 31913 31898 0 Feb04 ? 01:19:23 postgres: port 50001, mirror receiver process [sdw4] gpadmin 31914 31888 0 Feb04 ? 00:13:06 postgres: port 50000, mirror consumer writer process [sdw4] gpadmin 31915 31898 0 Feb04 ? 00:38:56 postgres: port 50001, mirror consumer process [sdw4] gpadmin 31916 31888 0 Feb04 ? 00:36:17 postgres: port 50000, mirror consumer append only process [sdw4] gpadmin 31917 31898 0 Feb04 ? 00:19:13 postgres: port 50001, mirror consumer writer process [sdw4] gpadmin 31919 31898 0 Feb04 ? 00:36:00 postgres: port 50001, mirror consumer append only process [sdw4] gpadmin 31920 31888 0 Feb04 ? 00:07:49 postgres: port 50000, mirror sender ack process [sdw4] gpadmin 31922 31888 0 Feb04 ? 00:00:03 postgres: port 50000, mirror verification process [sdw4] gpadmin 31923 31898 0 Feb04 ? 00:08:11 postgres: port 50001, mirror sender ack process [sdw4] gpadmin 31924 31898 0 Feb04 ? 00:00:03 postgres: port 50001, mirror verification process [sdw4] gpadmin 31925 31900 0 Feb04 ? 01:05:21 postgres: port 50002, mirror receiver process [sdw4] gpadmin 31926 31900 0 Feb04 ? 00:34:34 postgres: port 50002, mirror consumer process [sdw4] gpadmin 31927 31900 0 Feb04 ? 00:12:25 postgres: port 50002, mirror consumer writer process [sdw4] gpadmin 31928 31900 0 Feb04 ? 00:36:15 postgres: port 50002, mirror consumer append only process [sdw4] gpadmin 31930 31900 0 Feb04 ? 00:07:48 postgres: port 50002, mirror sender ack process [sdw4] gpadmin 31931 31900 0 Feb04 ? 00:00:03 postgres: port 50002, mirror verification process [sdw4] gpadmin 31937 31843 0 Feb04 ? 00:00:40 postgres: port 40000, stats collector process [sdw4] gpadmin 31938 31843 0 Feb04 ? 00:07:18 postgres: port 40000, writer process [sdw4] gpadmin 31939 31843 0 Feb04 ? 00:02:02 postgres: port 40000, checkpoint process [sdw4] gpadmin 31940 31843 0 Feb04 ? 00:01:52 postgres: port 40000, sweeper process [sdw4] gpadmin 31944 31837 0 Feb04 ? 00:00:40 postgres: port 40003, stats collector process [sdw4] gpadmin 31945 31837 0 Feb04 ? 00:07:55 postgres: port 40003, writer process [sdw4] gpadmin 31946 31837 0 Feb04 ? 00:03:28 postgres: port 40003, checkpoint process [sdw4] gpadmin 31947 31837 0 Feb04 ? 00:01:50 postgres: port 40003, sweeper process [sdw4] gpadmin 31948 31841 0 Feb04 ? 00:00:36 postgres: port 40002, stats collector process [sdw4] gpadmin 31949 31841 0 Feb04 ? 00:08:00 postgres: port 40002, writer process [sdw4] gpadmin 31950 31841 0 Feb04 ? 00:02:49 postgres: port 40002, checkpoint process [sdw4] gpadmin 31951 31841 0 Feb04 ? 00:01:52 postgres: port 40002, sweeper process [sdw4] gpadmin 31952 31838 0 Feb04 ? 00:00:36 postgres: port 40001, stats collector process [sdw4] gpadmin 31953 31838 0 Feb04 ? 00:07:39 postgres: port 40001, writer process [sdw4] gpadmin 31954 31838 0 Feb04 ? 00:02:36 postgres: port 40001, checkpoint process [sdw4] gpadmin 31955 31838 0 Feb04 ? 00:01:47 postgres: port 40001, sweeper process [sdw4] gpadmin 37706 37670 0 17:52 pts/0 00:00:00 grep postgres [sdw4] gpadmin 42312 31843 0 Feb12 ? 00:02:50 postgres: port 40000, primary process [sdw4] gpadmin 42313 42312 0 Feb12 ? 00:09:58 postgres: port 40000, primary recovery process

7、在master上执行

#gpstop -af

报：-gpstop failed(Reason='FATAL: the database system is shutting down') exiting ...

同样停不了库

gpadmin@mdw:~> gpstop -M fast 20150227:18:42:01:021768 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -M fast 20150227:18:42:01:021768 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20150227:18:42:01:021768 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20150227:18:42:01:021768 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20150227:18:42:01:021768 gpstop:mdw:gpadmin-[CRITICAL]:-gpstop failed. (Reason='FATAL: the database system is shutting down ') exiting...

同样不行。

8、直接kill

gpssh -h sdw1 -h sdw2 -h sdw3 -h sdw4 -h sdw5 -h sdw6 -h sdw7 -h mdw -h smdw "for i in `ps -ef |grep postgres |awk {'print $2'}`;do kill $i; done"

使用上面的检查，segment上的进程都已停止，但是master上的部分进程没有停。

9、停止master进程

mdw -h smdw ps -ef |grep postgres [ mdw] gpadmin 15765 15724 0 17:52 pts/11 00:00:00 grep postgres [ mdw] gpadmin 38876 41830 0 Feb26 ? 00:05:11 postgres: port 5432, cems cems 198.168.11.12(51943) con6959 198.168.11.12(51943) cmd24705 BIND [ mdw] gpadmin 41830 1 0 Feb04 ? 00:00:07 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/master/gpseg-1 -p 5432 -b 1 -z 28 --silent-mode=true -i -M master -C -1 -x 58 -E [ mdw] gpadmin 41838 41830 0 Feb04 ? 00:10:18 postgres: port 5432, master logger process [ mdw] gpadmin 41844 41830 0 Feb04 ? 00:00:18 postgres: port 5432, stats collector process [ mdw] gpadmin 41845 41830 0 Feb04 ? 00:05:02 postgres: port 5432, writer process [ mdw] gpadmin 41846 41830 0 Feb04 ? 00:00:55 postgres: port 5432, checkpoint process [ mdw] gpadmin 41847 41830 0 Feb04 ? 00:00:13 postgres: port 5432, seqserver process [ mdw] gpadmin 41848 41830 0 Feb04 ? 00:04:33 postgres: port 5432, ftsprobe process [ mdw] gpadmin 41851 41830 0 Feb04 ? 00:00:49 postgres: port 5432, sweeper process

执行gkill postgres，没反应。

直接使用kill 41830，没反应，不行。

执行pg_ctl stop -D /data/master/gpseg-1

>pg_ctl stop -D /data/master/gpseg-1

waiting for server to shut down .........................................failed

pg_ctl:server does not shut down

应是新版本中对于进程做了保护，不能直接kill了。

9、使用pg_ctl停库

gpadmin@mdw:~> pg_ctl --help pg_ctl is a utility to start, stop, restart, reload configuration files, report the status of a PostgreSQL server, or signal a PostgreSQL process.

Usage: pg_ctl start [-w] [-t SECS] [-D DATADIR] [-s] [-l FILENAME] [-o "OPTIONS"] pg_ctl stop [-W] [-t SECS] [-D DATADIR] [-s] [-m SHUTDOWN-MODE] pg_ctl restart [-w] [-t SECS] [-D DATADIR] [-s] [-m SHUTDOWN-MODE] [-o "OPTIONS"] pg_ctl reload [-D DATADIR] [-s] pg_ctl status [-D DATADIR] pg_ctl kill SIGNALNAME PID

Common options: -D, --pgdata DATADIR location of the database storage area -s, --silent only print errors, no informational messages -t SECS seconds to wait when using -w option -w wait until operation completes -W do not wait until operation completes --help show this help, then exit --version output version information, then exit --gp-version output Greenplum version information, then exit (The default is to wait for shutdown, but not for start or restart.)

If the -D option is omitted, the environment variable PGDATA is used.

Options for start or restart: -l, --log FILENAME write (or append) server log to FILENAME -o OPTIONS command line options to pass to postgres (PostgreSQL server executable) -p PATH-TO-POSTGRES normally not necessary -c, --core-files allow postgres to produce core files

Options for stop or restart: -m SHUTDOWN-MODE can be "smart", "fast", or "immediate"

Shutdown modes are: smart quit after all clients have disconnected fast quit directly, with proper shutdown immediate quit without complete shutdown; will lead to recovery on restart

Allowed signal names for kill: HUP INT QUIT ABRT TERM USR1 USR2

Report bugs to <pgsql-bugs@postgresql.org>. gpadmin@mdw:~> pg_ctl stop -m immedidate pg_ctl: unrecognized shutdown mode "immedidate" Try "pg_ctl --help" for more information. gpadmin@mdw:~> pg_ctl stop -D /data/master/gpseg-1 -m immedidate pg_ctl: unrecognized shutdown mode "immedidate" Try "pg_ctl --help" for more information. gpadmin@mdw:~> pg_ctl stop -D /data/master/gpseg-1 -m immediate waiting for server to shut down.... done server stopped gpadmin@mdw:~> ps -ef |grep post root 4824 1 0 Feb03 ? 00:00:17 /usr/lib/postfix/master postfix 4857 4824 0 Feb03 ? 00:00:01 qmgr -l -t fifo -u gpadmin 21828 21731 0 18:47 pts/1 00:00:00 grep post

10、启动数据库

gpadmin@mdw:~> gpstart -m 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -m 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.3.3.1 build 1' 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '201310150' 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[WARNING]:-**************************************************************************** 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[WARNING]:-Master-only start requested in a configuration with a standby master. 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[WARNING]:-This is advisable only under the direct supervision of Greenplum support. 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[WARNING]:-This mode of operation is not supported in a production environment and 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[WARNING]:-may lead to a split-brain condition and possible unrecoverable data loss. 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[WARNING]:-****************************************************************************

Continue with master-only startup Yy|Nn (default=N): > y 20150227:18:47:13:021829 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20150227:18:47:15:021829 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20150227:18:47:15:021829 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20150227:18:47:16:021829 gpstart:mdw:gpadmin-[INFO]:-Setting new master era 20150227:18:47:16:021829 gpstart:mdw:gpadmin-[INFO]:-Master Started... gpadmin@mdw:~> gpstop -af 20150227:18:47:28:021878 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -af 20150227:18:47:28:021878 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20150227:18:47:28:021878 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20150227:18:47:28:021878 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 4.3.3.1 build 1' 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-There are 0 connections to the database 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='fast' 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Master host=mdw 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Detected 0 connections to database 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Using standard WAIT mode of 600 seconds 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=fast 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/data/master/gpseg-1 20150227:18:47:30:021878 gpstop:mdw:gpadmin-[INFO]:-Stopping master standby host smdw mode=fast 20150227:18:47:30:021878 gpstop:mdw:gpadmin-[WARNING]:-Error occured while stopping the standby master: ExecutionError: 'non-zero rc: 1' occured. Details: 'ssh -o 'StrictHostKeyChecking no' smdw ". /usr/local/greenplum-db/./greenplum_path.sh; $GPHOME/bin/pg_ctl -D /data/master/gpseg-1 -m fast -w -t 600 stop"' cmd had rc=1 completed=True halted=False stdout='' stderr='pg_ctl: PID file "/data/master/gpseg-1/postmaster.pid" does not exist Is server running? ' 20150227:18:47:30:021878 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown standby process on smdw 20150227:18:47:30:021878 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait... ........................................................................................................................................................................................................................................................................................................................................... ............................................................................................................................................................................... ....................................................................................................... 20150227:18:57:40:021878 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait... .. 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:------------------------------------------------- 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-Failed Segment Stop Information 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:------------------------------------------------- 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:14 FAILED host:'sdw4' datadir:'/data/primary/gpseg12' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:15 FAILED host:'sdw4' datadir:'/data/primary/gpseg13' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:16 FAILED host:'sdw4' datadir:'/data/primary/gpseg14' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:17 FAILED host:'sdw4' datadir:'/data/primary/gpseg15' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:46 FAILED host:'sdw6' datadir:'/data/mirror/gpseg16' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:22 FAILED host:'sdw6' datadir:'/data/primary/gpseg20' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:23 FAILED host:'sdw6' datadir:'/data/primary/gpseg21' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:24 FAILED host:'sdw6' datadir:'/data/primary/gpseg22' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:25 FAILED host:'sdw6' datadir:'/data/primary/gpseg23' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:47 FAILED host:'sdw7' datadir:'/data/mirror/gpseg17' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:26 FAILED host:'sdw7' datadir:'/data/primary/gpseg24' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:27 FAILED host:'sdw7' datadir:'/data/primary/gpseg25' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:28 FAILED host:'sdw7' datadir:'/data/primary/gpseg26' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:29 FAILED host:'sdw7' datadir:'/data/primary/gpseg27' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:10 FAILED host:'sdw3' datadir:'/data/primary/gpseg8' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:11 FAILED host:'sdw3' datadir:'/data/primary/gpseg9' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:12 FAILED host:'sdw3' datadir:'/data/primary/gpseg10' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:13 FAILED host:'sdw3' datadir:'/data/primary/gpseg11' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:2 FAILED host:'sdw1' datadir:'/data/primary/gpseg0' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:3 FAILED host:'sdw1' datadir:'/data/primary/gpseg1' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:4 FAILED host:'sdw1' datadir:'/data/primary/gpseg2' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:5 FAILED host:'sdw1' datadir:'/data/primary/gpseg3' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:48 FAILED host:'sdw1' datadir:'/data/mirror/gpseg18' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:6 FAILED host:'sdw2' datadir:'/data/primary/gpseg4' with reason:'cmd had rc=255 completed=True halted=False stdout='20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Starting gpsegstop.py with args: -D /data/primary/gpseg4:40000 -D /data/primary/gpseg5:40001 -D /data/primary/gpseg6:40002 -D /data/primary/gpseg7:40003 -D /data/mirror/gpseg19:50003 -m fast -t 600 -V postgres (Greenplum Database) 4.3.3.1 build 1 20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Issuing shutdown commands to local segments... 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGSCONT to 37112 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGTERM to 37112 (smart shutdown) 20150227:18:57:36:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGINT to 37112 (fast shutdown) 20150227:18:57:39:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGQUIT to 37112 (immediate shutdown) 20150227:18:57:40:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:- COMMAND RESULTS STATUS--DIR:/data/primary/gpseg4--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg4/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/primary/gpseg5--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg5/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/primary/gpseg6--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: waiting for server to shut down........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... failed stderr: pg_ctl: server does not shut down

STATUS--DIR:/data/primary/gpseg7--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg7/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/mirror/gpseg19--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/mirror/gpseg19/postmaster.pid" does not exist Is server running?

' stderr=''' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:7 FAILED host:'sdw2' datadir:'/data/primary/gpseg5' with reason:'cmd had rc=255 completed=True halted=False stdout='20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Starting gpsegstop.py with args: -D /data/primary/gpseg4:40000 -D /data/primary/gpseg5:40001 -D /data/primary/gpseg6:40002 -D /data/primary/gpseg7:40003 -D /data/mirror/gpseg19:50003 -m fast -t 600 -V postgres (Greenplum Database) 4.3.3.1 build 1 20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Issuing shutdown commands to local segments... 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGSCONT to 37112 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGTERM to 37112 (smart shutdown) 20150227:18:57:36:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGINT to 37112 (fast shutdown) 20150227:18:57:39:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGQUIT to 37112 (immediate shutdown) 20150227:18:57:40:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:- COMMAND RESULTS STATUS--DIR:/data/primary/gpseg4--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg4/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/primary/gpseg5--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg5/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/primary/gpseg7--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg7/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/mirror/gpseg19--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/mirror/gpseg19/postmaster.pid" does not exist Is server running?

' stderr=''' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:8 FAILED host:'sdw2' datadir:'/data/primary/gpseg6' with reason:'cmd had rc=255 completed=True halted=False stdout='20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Starting gpsegstop.py with args: -D /data/primary/gpseg4:40000 -D /data/primary/gpseg5:40001 -D /data/primary/gpseg6:40002 -D /data/primary/gpseg7:40003 -D /data/mirror/gpseg19:50003 -m fast -t 600 -V postgres (Greenplum Database) 4.3.3.1 build 1 20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Issuing shutdown commands to local segments... 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGSCONT to 37112 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGTERM to 37112 (smart shutdown) 20150227:18:57:36:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGINT to 37112 (fast shutdown) 20150227:18:57:39:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGQUIT to 37112 (immediate shutdown) 20150227:18:57:40:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:- COMMAND RESULTS STATUS--DIR:/data/primary/gpseg4--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg4/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/primary/gpseg5--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg5/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/primary/gpseg7--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg7/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/mirror/gpseg19--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/mirror/gpseg19/postmaster.pid" does not exist Is server running?

' stderr=''' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:9 FAILED host:'sdw2' datadir:'/data/primary/gpseg7' with reason:'cmd had rc=255 completed=True halted=False stdout='20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Starting gpsegstop.py with args: -D /data/primary/gpseg4:40000 -D /data/primary/gpseg5:40001 -D /data/primary/gpseg6:40002 -D /data/primary/gpseg7:40003 -D /data/mirror/gpseg19:50003 -m fast -t 600 -V postgres (Greenplum Database) 4.3.3.1 build 1 20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Issuing shutdown commands to local segments... 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGSCONT to 37112 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGTERM to 37112 (smart shutdown) 20150227:18:57:36:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGINT to 37112 (fast shutdown) 20150227:18:57:39:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGQUIT to 37112 (immediate shutdown) 20150227:18:57:40:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:- COMMAND RESULTS STATUS--DIR:/data/primary/gpseg4--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg4/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/primary/gpseg5--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg5/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/primary/gpseg7--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg7/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/mirror/gpseg19--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/mirror/gpseg19/postmaster.pid" does not exist Is server running?

' stderr=''' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:49 FAILED host:'sdw2' datadir:'/data/mirror/gpseg19' with reason:'cmd had rc=255 completed=True halted=False stdout='20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Starting gpsegstop.py with args: -D /data/primary/gpseg4:40000 -D /data/primary/gpseg5:40001 -D /data/primary/gpseg6:40002 -D /data/primary/gpseg7:40003 -D /data/mirror/gpseg19:50003 -m fast -t 600 -V postgres (Greenplum Database) 4.3.3.1 build 1 20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Issuing shutdown commands to local segments... 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGSCONT to 37112 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGTERM to 37112 (smart shutdown) 20150227:18:57:36:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGINT to 37112 (fast shutdown) 20150227:18:57:39:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGQUIT to 37112 (immediate shutdown) 20150227:18:57:40:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:- COMMAND RESULTS STATUS--DIR:/data/primary/gpseg4--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg4/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/primary/gpseg5--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg5/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/primary/gpseg7--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg7/postmaster.pid" does not exist Is server running?

STATUS--DIR:/data/mirror/gpseg19--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/mirror/gpseg19/postmaster.pid" does not exist Is server running?

' stderr=''' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:45 FAILED host:'sdw1' datadir:'/data/mirror/gpseg15' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:51 FAILED host:'sdw1' datadir:'/data/mirror/gpseg21' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:54 FAILED host:'sdw1' datadir:'/data/mirror/gpseg24' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:41 FAILED host:'sdw7' datadir:'/data/mirror/gpseg11' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:44 FAILED host:'sdw7' datadir:'/data/mirror/gpseg14' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:50 FAILED host:'sdw7' datadir:'/data/mirror/gpseg20' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:37 FAILED host:'sdw6' datadir:'/data/mirror/gpseg7' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:40 FAILED host:'sdw6' datadir:'/data/mirror/gpseg10' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:43 FAILED host:'sdw6' datadir:'/data/mirror/gpseg13' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:30 FAILED host:'sdw2' datadir:'/data/mirror/gpseg0' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:52 FAILED host:'sdw2' datadir:'/data/mirror/gpseg22' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:55 FAILED host:'sdw2' datadir:'/data/mirror/gpseg25' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:32 FAILED host:'sdw4' datadir:'/data/mirror/gpseg2' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:35 FAILED host:'sdw4' datadir:'/data/mirror/gpseg5' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:38 FAILED host:'sdw4' datadir:'/data/mirror/gpseg8' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:57 FAILED host:'sdw4' datadir:'/data/mirror/gpseg27' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:31 FAILED host:'sdw3' datadir:'/data/mirror/gpseg1' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:34 FAILED host:'sdw3' datadir:'/data/mirror/gpseg4' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:53 FAILED host:'sdw3' datadir:'/data/mirror/gpseg23' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:56 FAILED host:'sdw3' datadir:'/data/mirror/gpseg26' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:33 FAILED host:'sdw5' datadir:'/data/mirror/gpseg3' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:36 FAILED host:'sdw5' datadir:'/data/mirror/gpseg6' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:39 FAILED host:'sdw5' datadir:'/data/mirror/gpseg9' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:42 FAILED host:'sdw5' datadir:'/data/mirror/gpseg12' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:18 FAILED host:'sdw5' datadir:'/data/primary/gpseg16' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:19 FAILED host:'sdw5' datadir:'/data/primary/gpseg17' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:20 FAILED host:'sdw5' datadir:'/data/primary/gpseg18' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:21 FAILED host:'sdw5' datadir:'/data/primary/gpseg19' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:- Segments stopped successfully = 0 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-Segments with errors during stop = 45 <<<<<<<< 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:- 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-Segments that are currently marked down in configuration = 11 <<<<<<<< 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:- (stop was still attempted on these segments) 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown 0 of 56 segment instances <<<<<<<< 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:------------------------------------------------- 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-Segment instance shutdown failures reported 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-Failed to shutdown 45 of 56 segment instances <<<<< 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-A total of 45 errors were encountered 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-Review logfile /home/gpadmin/gpAdminLogs/gpstop_20150227.log 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-For more details on segment shutdown failure(s) 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:------------------------------------------------- gpadmin@mdw:~> gpadmin@mdw:~> gpadmin@mdw:~> gpstart -a 20150227:18:58:42:022377 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -a 20150227:18:58:42:022377 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20150227:18:58:42:022377 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.3.3.1 build 1' 20150227:18:58:42:022377 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '201310150' 20150227:18:58:42:022377 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20150227:18:58:43:022377 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20150227:18:58:43:022377 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20150227:18:58:44:022377 gpstart:mdw:gpadmin-[INFO]:-Setting new master era 20150227:18:58:44:022377 gpstart:mdw:gpadmin-[INFO]:-Master Started... 20150227:18:58:44:022377 gpstart:mdw:gpadmin-[INFO]:-Shutting down master 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw2 directory /data/mirror/gpseg0 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/mirror/gpseg3 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/mirror/gpseg6 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/mirror/gpseg9 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/mirror/gpseg12 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/primary/gpseg16 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/primary/gpseg17 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/primary/gpseg18 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/primary/gpseg19 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw2 directory /data/mirror/gpseg22 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw2 directory /data/mirror/gpseg25 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-Process results... 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:9 FAILED host:'sdw2' datadir:'/data/primary/gpseg7' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:7 FAILED host:'sdw2' datadir:'/data/primary/gpseg5' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:6 FAILED host:'sdw2' datadir:'/data/primary/gpseg4' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:54 FAILED host:'sdw1' datadir:'/data/mirror/gpseg24' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:51 FAILED host:'sdw1' datadir:'/data/mirror/gpseg21' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:45 FAILED host:'sdw1' datadir:'/data/mirror/gpseg15' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:4 FAILED host:'sdw1' datadir:'/data/primary/gpseg2' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:3 FAILED host:'sdw1' datadir:'/data/primary/gpseg1' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:23 FAILED host:'sdw6' datadir:'/data/primary/gpseg21' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:26 FAILED host:'sdw7' datadir:'/data/primary/gpseg24' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:17 FAILED host:'sdw4' datadir:'/data/primary/gpseg15' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------

20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:- Successful segment starts = 34 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-Failed segment starts, from mirroring connection between primary and mirror = 11 <<<<<<<< 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:- Other failed segment starts = 0 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipped segment starts (segments are marked down in configuration) = 11 <<<<<<<< 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:- 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-Successfully started 34 of 45 segment instances, skipped 11 other segments <<<<<<<< 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-Segment instance startup failures reported 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-Failed start 11 of 45 segment instances <<<<<<<< 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-Review /home/gpadmin/gpAdminLogs/gpstart_20150227.log 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-For more details on segment startup failure(s) 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-Run gpstate -s to review current segment instance status 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-**************************************************************************** 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-There are 11 segment(s) marked down in the database 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-To recover from this current state, review usage of the gprecoverseg 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-management utility which will recover failed segment instance databases. 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-**************************************************************************** 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance mdw directory /data/master/gpseg-1 20150227:19:08:51:022377 gpstart:mdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active 20150227:19:09:05:022377 gpstart:mdw:gpadmin-[INFO]:-Starting standby master 20150227:19:09:05:022377 gpstart:mdw:gpadmin-[INFO]:-Checking if standby master is running on host: smdw in directory: /data/master/gpseg-1 20150227:19:09:08:022377 gpstart:mdw:gpadmin-[WARNING]:-Number of segments which failed to start: 11

12、连接数据库，删除不需要的清单表

gpadmin@mdw:~> psql -d template1 psql (8.2.15) Type "help" for help.

13、gprecoverseg恢复

gprecoverseg Continue with segment recovery procedure Yy|Nn (default=N): > Y 20150227:22:43:00:035973 gprecoverseg:mdw:gpadmin-[INFO]:-15 segment(s) to recover 20150227:22:43:00:035973 gprecoverseg:mdw:gpadmin-[INFO]:-Ensuring 15 failed segment(s) are stopped 20150227:22:43:01:035973 gprecoverseg:mdw:gpadmin-[INFO]:-38594: /data/mirror/gpseg0 20150227:22:43:03:035973 gprecoverseg:mdw:gpadmin-[INFO]:-41988: /data/mirror/gpseg1 20150227:22:43:04:035973 gprecoverseg:mdw:gpadmin-[INFO]:-1525: /data/mirror/gpseg2 20150227:22:43:05:035973 gprecoverseg:mdw:gpadmin-[INFO]:-15788: /data/mirror/gpseg3 20150227:22:43:06:035973 gprecoverseg:mdw:gpadmin-[INFO]:-41987: /data/mirror/gpseg4 20150227:22:43:08:035973 gprecoverseg:mdw:gpadmin-[INFO]:-1524: /data/mirror/gpseg5 20150227:22:43:09:035973 gprecoverseg:mdw:gpadmin-[INFO]:-15787: /data/mirror/gpseg6 20150227:22:43:10:035973 gprecoverseg:mdw:gpadmin-[INFO]:-30986: /data/mirror/gpseg7 20150227:22:43:11:035973 gprecoverseg:mdw:gpadmin-[INFO]:-2730: /data/mirror/gpseg15 20150227:22:43:12:035973 gprecoverseg:mdw:gpadmin-[INFO]:-15786: /data/primary/gpseg18 20150227:22:43:14:035973 gprecoverseg:mdw:gpadmin-[INFO]:-15785: /data/primary/gpseg19 20150227:22:43:15:035973 gprecoverseg:mdw:gpadmin-[INFO]:-2729: /data/mirror/gpseg21 20150227:22:43:16:035973 gprecoverseg:mdw:gpadmin-[INFO]:-38595: /data/mirror/gpseg22 20150227:22:43:17:035973 gprecoverseg:mdw:gpadmin-[INFO]:-2728: /data/mirror/gpseg24 20150227:22:43:19:035973 gprecoverseg:mdw:gpadmin-[INFO]:-38596: /data/mirror/gpseg25 20150227:22:43:23:035973 gprecoverseg:mdw:gpadmin-[INFO]:-Cleaning files from 15 segment(s) ........................................................................................................................................................................................................................................................................................................................................................................................................................................... 20150227:22:50:31:035973 gprecoverseg:mdw:gpadmin-[INFO]:-Building template directory 20150227:22:50:31:035973 gprecoverseg:mdw:gpadmin-[CRITICAL]:-Error occurred: non-zero rc: 1 Command was: 'ssh -o 'StrictHostKeyChecking no' sdw1 ". /usr/local/greenplum-db/./greenplum_path.sh; /usr/bin/scp -o 'StrictHostKeyChecking no' -r /data/primary/gpseg0/postgresql.conf mdw:/data/master/gpbuildingsegment_02272015_35973/schema"' rc=1, stdout='', stderr='ssh: Could not resolve hostname mdw: Name or service not known lost connection

报错，20150227:22:50:31:035973 gprecoverseg:mdw:gpadmin-[CRITICAL]:-Error occurred: non-zero rc: 1 Command was: 'ssh -o 'StrictHostKeyChecking no' sdw1 ". /usr/local/greenplum-db/./greenplum_path.sh; /usr/bin/scp -o 'StrictHostKeyChecking no' -r /data/primary/gpseg0/postgresql.conf mdw:/data/master/gpbuildingsegment_02272015_35973/schema"' rc=1, stdout='', stderr='ssh: Could not resolve hostname mdw: Name or service not known lost connection

经分析，应是hosts有问题了，直接ssh hosts的域名不行，直接ssh ip地址没有问题。

经检查，segment上的hosts有问题，和hadoop集群的hosts一样了。应该hadoop集群是后安装的，当时我要同事帮忙scp修改的hosts，估计把gp的集群也覆盖了。

修改hosts完后，

再执行gprecoverseg命令

gpadmin@mdw:~> gpstate -m 20150228:11:43:19:019954 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -m 20150228:11:43:19:019954 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.3.1 build 1' 20150228:11:43:19:019954 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.3.1 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Oct 10 2014 14:31:50' 20150228:11:43:19:019954 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:--Current GPDB mirror list and status 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:--Type = Spread 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw2 /data/mirror/gpseg0 50000 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw3 /data/mirror/gpseg1 50001 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw4 /data/mirror/gpseg2 50002 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw5 /data/mirror/gpseg3 50003 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw3 /data/mirror/gpseg4 50000 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw4 /data/mirror/gpseg5 50001 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw5 /data/mirror/gpseg6 50002 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw6 /data/mirror/gpseg7 50003 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw4 /data/mirror/gpseg8 50000 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw5 /data/mirror/gpseg9 50001 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw6 /data/mirror/gpseg10 50002 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw7 /data/mirror/gpseg11 50003 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw5 /data/mirror/gpseg12 50000 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw6 /data/mirror/gpseg13 50001 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw7 /data/mirror/gpseg14 50002 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw1 /data/mirror/gpseg15 50003 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw6 /data/mirror/gpseg16 50000 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw7 /data/mirror/gpseg17 50001 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw1 /data/mirror/gpseg18 50002 Acting as Primary Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw2 /data/mirror/gpseg19 50003 Acting as Primary Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw7 /data/mirror/gpseg20 50000 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw1 /data/mirror/gpseg21 50001 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw2 /data/mirror/gpseg22 50002 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw3 /data/mirror/gpseg23 50003 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw1 /data/mirror/gpseg24 50000 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw2 /data/mirror/gpseg25 50001 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw3 /data/mirror/gpseg26 50002 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw4 /data/mirror/gpseg27 50003 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[WARNING]:-2 segment(s) configured as mirror(s) are acting as primaries

恢复正常

gp数据库停止的更多相关文章

GP数据库常用SQL语句
GP数据库常用SQL语句 --1,查看列名以及类型 select upper(column_name) ,data_type from information_schema.columns wher ...
【gp数据库】你一定会用到的运维语句宝典
-- 查询未空闲的进程信息 select * from pg_stat_activity where current_query<>'<IDLE>'; 结果可查看数据库名,进程 ...
【gp数据库】OLTP和OLAP区别详解
原来一直使用Oracle,新公司使用greenplum后发现系统的并发性差很多,后来才了解因为Oracle属于OLTP类型,而gp数据库属于OLAP类型的.具体了解如下: 数据库系统一般分为两种类型, ...
【gp数据库】查询系统表看模式下所有表的分布键信息
Greenplum是关系型的分布式数据库,需要存储的数据库在进入数据库时,将先进行数据分布的处理工作,讲一个表的数据平均分不到每个节点上,并为每个表指定一个分发列(distribute Column) ...
使用sql查询mysql/oracle/sql server/gp数据库中指定表的字段信息(字段名/字段类型/字段长度/是否是主键/是否为空)
1,根据数据库类型拼接不同URL /** * 根据类型不同拼接连接的URL * @param dbType 1:mysql.2:oracle.3:sql server.4:gp * @param ip ...
gp数据库运维
最近需要将一份db2导出的历史数据入库gp集群,然后把每天的增量数据导出成txt文件和对应的log日志,再ftp传输给另外一台机器.其中陆续碰到一些坑,在此记录历史文件数据清洗列分隔符的选择碰到 ...
[原创]Greenplum数据库集群实践
GreenPlum实践 ============================================== 目录: 一.安装环境准备二.GP数据库安装三.集群添加standby节点四. ...
【SQLServer】【恢复挂起的解决方案】附加文件时候的提示“无法重新生成日志，原因是数据库关闭时存在打开的事务/用户，该数据库没有检查点或者该数据库是只读的。 ”【数据库恢复】
汇总篇:http://www.cnblogs.com/dunitian/p/4822808.html#tsql 先贴错误: 吐槽一下: 进入正题: 新建一个同名数据库停止MSSQL服务替换数据库文 ...
Mariadb数据库设置及操作一主多从备份还原（实测笔记）
环境: 系统硬件:vmware vsphere (CPU:2*4核,内存2G,双网卡) 系统版本:CentOS-7-x86_64-Minimal-1611.iso 数据库版本信息 : 10.1.20- ...

随机推荐

中国移动CMPP协议错误码
中国移动CMPP协议错误码状态码说明出现次数高 DELIVRD 消息发送成功用户成功接收到短信 REJECTD 消息因为某些原因被拒绝不 ...
[日常] 编写HTTP接口文档
一.什么是接口文档?在项目开发中,web项目的前后端分离开发,APP开发,需要由前后端工程师共同定义接口,编写接口文档,之后大家都根据这个接口文档进行开发,到项目结束前都要一直维护.二.为什么要写接口 ...
2 springboot多模块项目
一般来说创建一个springboot工程基本就可以了,但是有的时候可能需要将业务模块逻辑划分,每块业务模块都是一个工程,下边演示下多模块进行开发目录结构 ...somefun ......somef ...
Java 中 List 和数组之间的转换
前言:在 java 开发过程中,经常会遇到数组和 list 互转的情况,这里记录一下,免得以后又去到处百度. 一.数组转为 LIST 这里转自 https://www.cnblogs.com/lius ...
mysql行转列，列转行
行转列,列转行是我们在开发过程中经常碰到的问题.行转列一般通过CASE WHEN 语句来实现,也可以通过 SQL SERVER 2005 新增的运算符PIVOT来实现.用传统的方法,比较好理解.层次清 ...
一文总结学习 Python 的 14 张思维导图
本文主要涵盖了 Python 编程的核心知识(暂不包括标准库及第三方库,后续会发布相应专题的文章). 首先,按顺序依次展示了以下内容的一系列思维导图:基础知识,数据类型(数字,字符串,列表,元组,字典 ...
Spring 配置数据源之一三兄弟
前期的准备工作,我们是使用的是maven,我们下载节点即可... 节点如下: <dependency> <groupId>org.springframework</gro ...
朴素贝叶斯文本分类实现 python cherry分类器
贝叶斯模型在机器学习以及人工智能中都有出现,cherry分类器使用了朴素贝叶斯模型算法,经过简单的优化,使用1000个训练数据就能得到97.5%的准确率.虽然现在主流的框架都带有朴素贝叶斯模型算法,大 ...
needPrint 不显示打印按钮
客户问题: 客户用的是needPrint 来显示打印按钮,现在访问不能显示后台提示有错误打开控制台显示: java.security.AccessControlException: ...
180217_JAVA学习_TreeSet中存放含多个String的类并设置排序规则
有Person类如下: class Person { String name; int age; String address; } 有main如下: import java.util.TreeSet ...

gp数据库停止

gp数据库停止的更多相关文章

随机推荐

热门专题