Pivotal Knowledge Base

Follow

Postmaster Reload may cause Fatal Exception with valid Postgresql.conf changes

Environment

Product Version
Pivotal Greenplum (GPDB) 4.x
OS ALL

Symptom

When a user is attempting to implement multiple changes to the postgresql.conf file through direct editing or gpconfig, be cautious to process conflicting statements in the correct order if issuing a reload (gpstop -u).

Consider the following changes —

statement_mem=2500MB
max_statement_mem=3000MB (Note: Default 2000MB)

If the new value for statement_mem is greater than the current value for max_statement_mem, when the postmaster-reload is issued (gpstop -u) the changes are implemented IN ORDER.

In this example, issuing a reload command (gpstop -u) will cause a FATAL exception and stop any postmaster which implements the change.

Error Message:

2016-07-09 23:22:35.215363 BST,,,p6037,th-1881245296,,,,0,,,seg-1,,,,,"FATAL","XX000","Invalid input for statement_mem. Must be less than max_statement_mem (10485760 kB). (cdbvars.c:1258)",,,,,,,0,,"cdbvars.c",1258,"Stack trace:
1    0xa6dcaa postgres  (elog.c:468)
2    0xa6fd58 postgres elog_finish (elog.c:1403)
3    0xc304db postgres gpvars_assign_statement_mem (cdbvars.c:1258)
4    0xaa01ba postgres set_config_option (guc.c:8518)
5    0xaa183d postgres ProcessConfigFile (guc-file.l:163)
6    0x870fce postgres  (postmaster.c:6294)
7    0x37d84302d0 libc.so.6  (??:0)
8    0x37d84ce3b3 libc.so.6 __select (??:0)
9    0x87a63b postgres PostmasterMain (postmaster.c:2240)
10   0x77ff3a postgres main (main.c:212)
11   0x37d841d9c4 libc.so.6 __libc_start_main (??:0)
12   0x47c449 postgres  (??:0)
"
2016-07-09 23:22:35.281873 BST,,,p6045,th-1881245296,,,,0,con2,,seg-1,,,,,"LOG","00000","exiting because postmaster has died",,,,,,,0,,"walsendserver.c",293,

Cause 

Reload and Start process postgresql.conf files differently.
Reload implements changes in order, Start implements all parameters simultaneously. 

Resolution

Recommend to change parameter conflicting with each other in a sequential manner. For the above Example: 

1. First, change the value of 'max_statement_mem' to 3000MB and then change the value of 'statement_mem' to 2000MB. [Since statement_mem is dependent on max_statment_mem]
 
In order to come out of the FATAL message
[Master has crashed and is failing to start after the 'gpstop -u']: 
 
2. Reorder the parameters in the postgresql.conf file to the correct sequence:
** You may have to use gpconfig to change it across all segments.
max_statement_mem=3000MB
statement_mem=2500MB

3. Start the master [Since the problematic 'gpstop -u' previously issued stops the master]

gpstart -m

4. Cleanly shutdown and restart the cluster: 

gpstop -af
gpstart -a

 

Internal Comments

This section is strictly for Pivotal Support.

Notes: MPP-26472

 

Comments

Powered by Zendesk