Pivotal Knowledge Base

Follow

gpstart fails after gpdb upgrade on DCA to 4.3.6.1 and 4.3.6.2

Environment 

Product Version
Pivotal Greenplum (GPDB) 4.3.6.1, 4.3.6.2

Symptom

After successful upgrade of gpdb from 4.3.x to 4.3.6.1 and 4.3.6.2, gpstart fails with the below error message.

Error Message:

20151208:11:26:38:008704 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode
20151208:11:26:40:008704 gpstart:mdw:gpadmin-[CRITICAL]:-Failed to start Master instance in admin mode
20151208:11:26:40:008704 gpstart:mdw:gpadmin-[CRITICAL]:-Error occurred: non-zero rc: 1
Command was: 'env GPSESSID=0000000000 GPERA=None $GPHOME/bin/pg_ctl -D /data/master/gpseg-1 -l /data/master/gpseg-1/pg_log/startup.log -w -t 600 -o " -p 5432 -b 1 -z 0 --silent-mode=true -i -M master -C -1 -x 290 -c gp_role=utility " start'
rc=1, stdout='waiting for server to start...... stopped waiting
', stderr='pg_ctl: postmaster pid 8715 not running
pg_ctl: could not start server
Examine the log output.

Error as shown in master logs -

2015-12-08 11:26:38.543881 MST,,,p8715,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","TransitionToMasterOrMirrorless: initializing XLog startup",,,,,,,0,,"primary_mirror_mode.c",1887,
2015-12-08 11:26:38.544028 MST,,,p8715,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","TransitionToMasterOrMirrorless: starting database",,,,,,,0,,"primary_mirror_mode.c",1901,
2015-12-08 11:26:38.546498 MST,,,p8717,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","database system was shut down at 2015-12-08 11:25:12 MST",,,,,,,0,,"xlog.c",6372,
2015-12-08 11:26:38.547080 MST,,,p8717,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","checkpoint record is at E3/CB89820",,,,,,,0,,"xlog.c",6560,
2015-12-08 11:26:38.547184 MST,,,p8717,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","redo record is at E3/CB89820; undo record is at 0/0; shutdown TRUE",,,,,,,0,,"xlog.c",6617,
2015-12-08 11:26:38.547220 MST,,,p8717,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","next transaction ID: 0/92951226; next OID: 334541614",,,,,,,0,,"xlog.c",6621,
2015-12-08 11:26:38.547248 MST,,,p8717,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","next MultiXactId: 2620; next MultiXactOffset: 6202",,,,,,,0,,"xlog.c",6624,
2015-12-08 11:26:38.547308 MST,,,p8717,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","end of transaction log location is E3/CB898C8",,,,,,,0,,"xlog.c",6988,
2015-12-08 11:26:39.090242 MST,,,p8718,th23175760,,,2015-12-08 11:26:39 MST,0,,,seg-1,,,,,"LOG","00000","connection received: host=[local]",,,,,,,0,,"postmaster.c",6827,
2015-12-08 11:26:39.090405 MST,"gpadmin","postgres",p8718,th23175760,"[local]",,2015-12-08 11:26:39 MST,0,,,seg-1,,,,,"FATAL","57P03","the database system is starting up",,,,,,,0,,"postmaster.c",2955,
2015-12-08 11:26:39.090780 MST,"gpadmin","postgres",p8718,th23175760,"[local]",,2015-12-08 11:26:39 MST,0,,,seg-1,,,,,"LOG","00000","Send dummy packet failed, sendto failed: Invalid argument",,,,,,,0,,"ic_common.c",1099,
2015-12-08 11:26:39.102798 MST,,,p8717,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","Oldest active transaction from prepared transactions 92951226",,,,,,,0,,"xlog.c",6033,
2015-12-08 11:26:39.105168 MST,,,p8717,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","database system is ready",,,,,,,0,,"xlog.c",6084,
2015-12-08 11:26:39.105209 MST,,,p8717,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","PostgreSQL 8.2.15 (Greenplum Database 4.3.6.1 build 2) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Oct 1 2015 15:10:42",,,,,,,0,,"xlog.c",6095,
2015-12-08 11:26:39.121352 MST,,,p8717,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","Finished normal startup for clean shutdown case",,,,,,,0,,"xlog.c",7202,
2015-12-08 11:26:39.121572 MST,,,p8717,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","Send dummy packet failed, sendto failed: Invalid argument",,,,,,,0,,"ic_common.c",1099,
2015-12-08 11:26:39.144729 MST,,,p8717,th0,,,,0,,,seg-1,,,,,"PANIC","XX000","Unexpected internal error: Segment process received signal SIGSEGV",,,,,,,0,,,,"1 0x3b7bc0eca0 libpthread.so.0 <symbol not found> + 0x7bc0eca0
2 0x95c5f5 postgres proc_exit + 0x95
"
2015-12-08 11:26:39.511348 MST,,,p8715,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","startup process (PID 8717) was terminated by signal 11: Segmentation fault",,,,,,,0,,"postmaster.c",5876,
2015-12-08 11:26:39.511397 MST,,,p8715,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","aborting startup due to startup process failure",,,,,,,0,,"postmaster.c",4706,
2015-12-08 11:26:39.511602 MST,,,p8715,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","Send dummy packet failed, sendto failed: Invalid argument",,,,,,,0,,"ic_common.c",1099,
2015-12-08 11:26:40.015580 MST,,,p8716,th23175760,,,,0,,,seg-1,,,,,"LOG","00000","Send dummy packet failed, sendto failed: Invalid argument",,,,,,,,"SendDummyPacket","ic_common.c",1099,

Cause

gp_interconnect_type in postgresql.conf is set to UDP. UDP implementation for interconnect is deprecated now. Please refer deprecated features in Greenplum Database 4.3.6.1 Release Notes.

Resolution

Change gp_interconnect_type to udpifc. Use the gpconfig utility to set a parameter in all postgresql.conf files. For example:

gpconfig -c gp_interconnect_type -v UDPIFC

If the database is down and would not come up please refer to Scripts - Add / Remove parameters from postgresql.conf.

UDPIFC is an entirely new implementation of interconnect and does not depend on UDP implementation.

Comments

Powered by Zendesk