Pivotal Knowledge Base

Follow

gpexpand fails with the error message "Primary must have mirror when mirroring is enabled"

Environment

Product Version
Pivotal Greenplum (GPDB) 4.3.x
OS RHEL 6.x

Symptom

When attempting to expand the existing GPDB cluster with primary segments only, gpexpand fails with the following error message:

Primary must have mirror when mirroring enabled

Error Message:

[gpadmin@hitw-mck-mdw-01 ~]$ gpexpand -i gpexpand_inputfile_20160624_162205 -D gpexpand 
20160624:16:26:20:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1'
20160624:16:26:20:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_6
4-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40'
20160624:16:26:20:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Querying gpexpand schema for current expansion state
20160624:16:26:21:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Readying Greenplum Database for a new expansion
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database gpadmin for unalterable tables...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database postgres for unalterable tables...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database richard for unalterable tables...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database gpexpand for unalterable tables...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database template1 for unalterable tables...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database gpperfmon for unalterable tables...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database gpadmin for tables with unique indexes...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database postgres for tables with unique indexes...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database richard for tables with unique indexes... 20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database gpexpand for tables with unique indexes...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database template1 for tables with unique indexes...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database gpperfmon for tables with unique indexes...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Syncing Greenplum Database extensions
20160624:16:26:43:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-The packages on hitw-mck-sdw-05 are consistent.
20160624:16:26:43:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-The packages on hitw-mck-sdw-04 are consistent.
20160624:16:26:43:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-The packages on hitw-mck-sdw-06 are consistent.
20160624:16:26:44:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-The packages on hitw-mck-sdw-01 are consistent.
20160624:16:26:44:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-The packages on hitw-mck-sdw-03 are consistent.
20160624:16:26:45:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-The packages on hitw-mck-sdw-02 are consistent.
20160624:16:26:45:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Creating segment template
20160624:16:26:45:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-VACUUM FULL on the catalog tables
20160624:16:26:48:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Starting copy of segment dbid 1 to location /data/gpdb_master/gpexpand_06242016_1037
20160624:16:26:50:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Copying postgresql.conf from existing segment into template
20160624:16:26:50:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Copying pg_hba.conf from existing segment into template
20160624:16:26:50:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Adding new segments into template pg_hba.conf
20160624:16:26:50:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Creating schema tar file
20160624:16:26:52:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Distributing template tar file to new hosts
20160624:16:28:10:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Configuring new segments (primary)
20160624:16:28:13:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Configuring new segments (mirror)
20160624:16:28:13:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Backing up pg_hba.conf file on original segments
20160624:16:28:14:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Copying new pg_hba.conf file to original segments
20160624:16:28:14:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Configuring original segments
20160624:16:28:14:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Cleaning up temporary template files
20160624:16:28:16:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Starting Greenplum Database in restricted mode
20160624:16:28:30:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[ERROR]:-gpexpand failed: Primary must have mirror when mirroring enabled: 2|0|p|p|s|u|hitw-mck-sd
w-01|hitw-mck-sdw-01-1|40000|None|/data/gpdb_p1/gp0||/data/gpdb_p1/gp0/base/3803222,/data/gpdb_p1/gp0/base/10899,/data/gpdb_p1/gp0/base/1,/data/gpdb_p1/gp0/
base/102776162,/data/gpdb_p1/gp0/base/102732589,/data/gpdb_p1/gp0/base/10900,/data/gpdb_p1/gp0/base/16992 Exiting...
20160624:16:28:30:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[ERROR]:-Please run 'gpexpand -r -D gpexpand' to rollback to the original state.
20160624:16:28:30:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Shutting down gpexpand...

[gpadmin@hitw-mck-mdw-01 ~]$ cat gpexpand_inputfile_20160627_113214
hitw-mck-sdw-01:hitw-mck-sdw-01-1:40004:/data/gpdb_p5/gp24:27:24:p
hitw-mck-sdw-01:hitw-mck-sdw-01-2:40005:/data/gpdb_p6/gp25:28:25:p
hitw-mck-sdw-01:hitw-mck-sdw-01-3:40006:/data/gpdb_p7/gp26:29:26:p
hitw-mck-sdw-01:hitw-mck-sdw-01-4:40007:/data/gpdb_p8/gp27:30:27:p
hitw-mck-sdw-02:hitw-mck-sdw-02-1:40004:/data/gpdb_p5/gp28:31:28:p
hitw-mck-sdw-02:hitw-mck-sdw-02-2:40005:/data/gpdb_p6/gp29:32:29:p
hitw-mck-sdw-02:hitw-mck-sdw-02-3:40006:/data/gpdb_p7/gp30:33:30:p
hitw-mck-sdw-02:hitw-mck-sdw-02-4:40007:/data/gpdb_p8/gp31:34:31:p
hitw-mck-sdw-03:hitw-mck-sdw-03-1:40004:/data/gpdb_p5/gp32:35:32:p
hitw-mck-sdw-03:hitw-mck-sdw-03-2:40005:/data/gpdb_p6/gp33:36:33:p
hitw-mck-sdw-03:hitw-mck-sdw-03-3:40006:/data/gpdb_p7/gp34:37:34:p
hitw-mck-sdw-03:hitw-mck-sdw-03-4:40007:/data/gpdb_p8/gp35:38:35:p
hitw-mck-sdw-04:hitw-mck-sdw-04-1:40004:/data/gpdb_p5/gp36:39:36:p
hitw-mck-sdw-04:hitw-mck-sdw-04-2:40005:/data/gpdb_p6/gp37:40:37:p
hitw-mck-sdw-04:hitw-mck-sdw-04-3:40006:/data/gpdb_p7/gp38:41:38:p
hitw-mck-sdw-04:hitw-mck-sdw-04-4:40007:/data/gpdb_p8/gp39:42:39:p
hitw-mck-sdw-05:hitw-mck-sdw-05-1:40004:/data/gpdb_p5/gp40:43:40:p
hitw-mck-sdw-05:hitw-mck-sdw-05-2:40005:/data/gpdb_p6/gp41:44:41:p
hitw-mck-sdw-05:hitw-mck-sdw-05-3:40006:/data/gpdb_p7/gp42:45:42:p
hitw-mck-sdw-05:hitw-mck-sdw-05-4:40007:/data/gpdb_p8/gp43:46:43:p
hitw-mck-sdw-06:hitw-mck-sdw-06-1:40004:/data/gpdb_p5/gp44:47:44:p
hitw-mck-sdw-06:hitw-mck-sdw-06-2:40005:/data/gpdb_p6/gp45:48:45:p
hitw-mck-sdw-06:hitw-mck-sdw-06-3:40006:/data/gpdb_p7/gp46:49:46:p
hitw-mck-sdw-06:hitw-mck-sdw-06-4:40007:/data/gpdb_p8/gp47:50:47:p 

Cause

Once gpexpand is executed, it will first check if the GPDB system is configured with mirror segments by checking any entries in the system table gp_segment_configuration with role='m'.  In this case, there are no mirrors configured for primary segments; however, the standby master is provisioned, which also has role='m' in  gp_segment_configuration.  

gpadmin=# select * from gp_segment_configuration ;
dbid | content | role | preferred_role | mode | status | port | hostname | address | replication_port | san_mounts
------+---------+------+----------------+------+--------+-------+-----------------+-------------------+------------------+------------
2 | 0 | p | p | s | u | 40000 | hitw-mck-sdw-01 | hitw-mck-sdw-01-1 | |
3 | 1 | p | p | s | u | 40001 | hitw-mck-sdw-01 | hitw-mck-sdw-01-2 | |
7 | 5 | p | p | s | u | 40001 | hitw-mck-sdw-02 | hitw-mck-sdw-02-2 | |
11 | 9 | p | p | s | u | 40001 | hitw-mck-sdw-03 | hitw-mck-sdw-03-2 | |
15 | 13 | p | p | s | u | 40001 | hitw-mck-sdw-04 | hitw-mck-sdw-04-2 | |
19 | 17 | p | p | s | u | 40001 | hitw-mck-sdw-05 | hitw-mck-sdw-05-2 | |
23 | 21 | p | p | s | u | 40001 | hitw-mck-sdw-06 | hitw-mck-sdw-06-2 | |
4 | 2 | p | p | s | u | 40002 | hitw-mck-sdw-01 | hitw-mck-sdw-01-3 | |
8 | 6 | p | p | s | u | 40002 | hitw-mck-sdw-02 | hitw-mck-sdw-02-3 | |
12 | 10 | p | p | s | u | 40002 | hitw-mck-sdw-03 | hitw-mck-sdw-03-3 | |
16 | 14 | p | p | s | u | 40002 | hitw-mck-sdw-04 | hitw-mck-sdw-04-3 | |
20 | 18 | p | p | s | u | 40002 | hitw-mck-sdw-05 | hitw-mck-sdw-05-3 | |
24 | 22 | p | p | s | u | 40002 | hitw-mck-sdw-06 | hitw-mck-sdw-06-3 | |
5 | 3 | p | p | s | u | 40003 | hitw-mck-sdw-01 | hitw-mck-sdw-01-4 | |
9 | 7 | p | p | s | u | 40003 | hitw-mck-sdw-02 | hitw-mck-sdw-02-4 | |
13 | 11 | p | p | s | u | 40003 | hitw-mck-sdw-03 | hitw-mck-sdw-03-4 | |
17 | 15 | p | p | s | u | 40003 | hitw-mck-sdw-04 | hitw-mck-sdw-04-4 | |
21 | 19 | p | p | s | u | 40003 | hitw-mck-sdw-05 | hitw-mck-sdw-05-4 | |
25 | 23 | p | p | s | u | 40003 | hitw-mck-sdw-06 | hitw-mck-sdw-06-4 | |
6 | 4 | p | p | s | u | 40000 | hitw-mck-sdw-02 | hitw-mck-sdw-02-1 | |
10 | 8 | p | p | s | u | 40000 | hitw-mck-sdw-03 | hitw-mck-sdw-03-1 | |
14 | 12 | p | p | s | u | 40000 | hitw-mck-sdw-04 | hitw-mck-sdw-04-1 | |
18 | 16 | p | p | s | u | 40000 | hitw-mck-sdw-05 | hitw-mck-sdw-05-1 | |
22 | 20 | p | p | s | u | 40000 | hitw-mck-sdw-06 | hitw-mck-sdw-06-1 | |
26 | -1 | m | m | s | u | 5432 | hitw-mck-mdw-02 | hitw-mck-mdw-02 | |
1 | -1 | p | p | s | u | 5432 | hitw-mck-mdw-01 | hitw-mck-mdw-01 | |

Resolution

The issue has been reported to engineering with internal JIRA MPP-26457. Until the issue is fixed, please use the workaround below:

  1. Remove the standby master with "gpinitstandby -r".
  2. Run gpexpand to expand cluster.
  3. After gpexpand completes, run "gpinitstandby -s <standby hostname>" to reinitialize standby master.

Note: Services will not be impacted with remove/reinitialize standby master if GPDB system is on 4.3.x release. If it's on 4.2.x release, then schedule a downtime to implement the workaround.

 

 

 

Comments

Powered by Zendesk