Pivotal Knowledge Base

Follow

gpinitstandby fails after updating port

Environment

Pivotal Greenplum: 4.3.x

Symptom

If you attempt to change the port in the master using the port GUC and PGPORT environment variable, but fail to update the necessary catalog entry in gp_segment_configuration, gpinitstandby will throw the following error:

20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:-Validating environment and parameters for standby initialization... 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:-Checking for filespace directory /db/gp-data/master/gpseg-1 on smdw 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:------------------------------------------------------ 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:-Greenplum standby master initialization parameters 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:------------------------------------------------------ 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:-Greenplum master hostname = mdw 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:-Greenplum master data directory = /db/gp-data/master/gpseg-1 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:-Greenplum master port = 5332 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:-Greenplum standby master hostname = smdw 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:-Greenplum standby master port = 5332 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:-Greenplum standby master data directory = /db/gp-data/master/gpseg-1
 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:-Greenplum update system catalog = On 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:------------------------------------------------------ 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:- Filespace locations 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:------------------------------------------------------ 
20170829:13:35:23:012395 gpinitstandby:mdw:gpmadmin-[INFO]:-pg_system -> /db/gp-data/master/gpseg-1
20170829:13:45:49:012634 gpinitstandby:mdw:gpmadmin-[INFO]:-Updating pg_hba.conf file... 
20170829:13:45:55:012634 gpinitstandby:mdw:gpmadmin-[INFO]:-pg_hba.conf files updated successfully. 
20170829:13:45:55:012634 gpinitstandby:mdw:gpmadmin-[ERROR]:-Failed to copy filespace directories from master to standby. 
20170829:13:45:55:012634 gpinitstandby:mdw:gpmadmin-[ERROR]:-Failed to create standby 
20170829:13:45:55:012634 gpinitstandby:mdw:gpmadmin-[WARNING]:-Trying to rollback changes that have been made... 
20170829:13:45:55:012634 gpinitstandby:mdw:gpmadmin-[INFO]:-Rolling back catalog change... 
20170829:13:45:55:012634 gpinitstandby:mdw:gpmadmin-[INFO]:-Removing standby master from catalog... 
20170829:13:45:55:012634 gpinitstandby:mdw:gpmadmin-[INFO]:-Database catalog updated successfully. 
20170829:13:45:55:012634 gpinitstandby:mdw:gpmadmin-[INFO]:-Restoring pg_hba.conf file... 
20170829:13:46:01:012634 gpinitstandby:mdw:gpmadmin-[INFO]:-Saving standby filespace locations 
20170829:13:46:01:012634 gpinitstandby:mdw:gpmadmin-[INFO]:-List of standby filespace directories saved to /tmp/standby_filespaces_only.txt on standby host 
20170829:13:46:01:012634 gpinitstandby:mdw:gpmadmin-[WARNING]:-Please make cleanup for standby host only! 
20170829:13:46:01:012634 gpinitstandby:mdw:gpmadmin-[INFO]:-Cleaning up pg_hba.conf backup files... 
20170829:13:46:06:012634 gpinitstandby:mdw:gpmadmin-[INFO]:-Backup files of pg_hba.conf cleaned up successfully. 
20170829:13:46:06:012634 gpinitstandby:mdw:gpmadmin-[ERROR]:-Error initializing standby master: ExecutionError: 'non-zero rc: 1' occured. Details: 'ssh -o 'StrictHostKeyChecking no' smdw ". /db/greenplum-db/./greenplum_path.sh; pg_basebackup -x -R -E ./pg_log -E ./db_dumps -E ./gpperfmon/data -E ./gpperfmon/logs -D /db/gp-data/master/gpseg-1 -h mdw -p 5332"' cmd had rc=1 completed=True halted=False 
stdout='' 
stderr='pg_basebackup: could not connect to server: could not connect to server: Connection refused 
Is the server running on host "mdw" (xxx.xxx.xxx.xxx) and accepting 
TCP/IP connections on port 5332? 

Cause

gpinitstandby -s <standby_host> checks gp_segment_configuration to determine the port number to use for the standby master, whereas gpstart checks postgresql.conf and/or PGPORT when starting up the database. 

$ echo $PGPORT
5333

gpmadmin=# show port;
port
------
5333
$ ps -ef | grep postgres
gpmadmin 10814 1 0 12:40 ? 00:00:00 /db/greenplum-db-4.3.13.0/bin/postgres -D /db/gp-data/master/gpseg-1 -p 5333 -b 1 -z 3 --silent-mode=true -i -M master -C -1 -x 0 -E
gpmadmin 10815 10814 0 12:40 ? 00:00:00 postgres: port 5333, master logger process
gpmadmin 10819 10814 0 12:40 ? 00:00:00 postgres: port 5333, stats collector process
gpmadmin 10820 10814 0 12:40 ? 00:00:01 postgres: port 5333, writer process
gpmadmin 10821 10814 0 12:40 ? 00:00:00 postgres: port 5333, checkpoint process
gpmadmin 10822 10814 0 12:40 ? 00:00:00 postgres: port 5333, seqserver process
gpmadmin 10823 10814 0 12:40 ? 00:00:00 postgres: port 5333, ftsprobe proces
gpmadmin 10825 10814 0 12:40 ? 00:00:00 postgres: port 5333, sweeper process
gpmadmin 10826 10814 0 12:40 ? 00:00:00 postgres: port 5333, stats sender process

gpmadmin=# select dbid,content,hostname,port from gp_segment_configuration where content = -1;
dbid | content | hostname | port
------+---------+----------+------
1 | -1 | mdw | 5332

Resolution

To avoid making changes to the catalog, there are a couple of options:

1. Revert the PGPORT and postgresql.conf configuration and restart the database. Then run gpinitstandby again.

2. If you need to change the port, you can run gpinitstandby with the -P flag which will create a standby master with a new port number:

gpinitstandby -s smdw -P 5333

Failover to the standby master using gpactivatestandby and recreate the new master from the activated standby master. Details of reactivating the master can be found in the GPDB documentation.

Comments

Powered by Zendesk