Pivotal Knowledge Base

Follow

gpexpand Fails with the Error "Primary must have mirror when mirroring is enabled"

Environment

  • Pivotal Greenplum Database (GPDB) 4.3.x
  • Operating System- Red Hat Enterprise Linux 6.x

Symptom

When attempting to expand the existing GPDB cluster with primary segments only, gpexpand fails with the following error message:

Primary must have mirror when mirroring enabled

Error Message:

[gpadmin@hitw-mck-mdw-01 ~]$ gpexpand -i gpexpand_inputfile_20160624_162205 -D gpexpand 
20160624:16:26:20:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1'
20160624:16:26:20:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_6
4-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40'
20160624:16:26:20:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Querying gpexpand schema for current expansion state
20160624:16:26:21:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Readying Greenplum Database for a new expansion
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database gpadmin for unalterable tables...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database postgres for unalterable tables...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database richard for unalterable tables...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database gpexpand for unalterable tables...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database template1 for unalterable tables...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database gpperfmon for unalterable tables...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database gpadmin for tables with unique indexes...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database postgres for tables with unique indexes...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database richard for tables with unique indexes... 20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database gpexpand for tables with unique indexes...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database template1 for tables with unique indexes...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Checking database gpperfmon for tables with unique indexes...
20160624:16:26:42:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Syncing Greenplum Database extensions
20160624:16:26:43:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-The packages on hitw-mck-sdw-05 are consistent.
20160624:16:26:43:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-The packages on hitw-mck-sdw-04 are consistent.
20160624:16:26:43:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-The packages on hitw-mck-sdw-06 are consistent.
20160624:16:26:44:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-The packages on hitw-mck-sdw-01 are consistent.
20160624:16:26:44:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-The packages on hitw-mck-sdw-03 are consistent.
20160624:16:26:45:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-The packages on hitw-mck-sdw-02 are consistent.
20160624:16:26:45:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Creating segment template
20160624:16:26:45:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-VACUUM FULL on the catalog tables
20160624:16:26:48:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Starting copy of segment dbid 1 to location /data/gpdb_master/gpexpand_06242016_1037
20160624:16:26:50:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Copying postgresql.conf from existing segment into template
20160624:16:26:50:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Copying pg_hba.conf from existing segment into template
20160624:16:26:50:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Adding new segments into template pg_hba.conf
20160624:16:26:50:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Creating schema tar file
20160624:16:26:52:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Distributing template tar file to new hosts
20160624:16:28:10:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Configuring new segments (primary)
20160624:16:28:13:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Configuring new segments (mirror)
20160624:16:28:13:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Backing up pg_hba.conf file on original segments
20160624:16:28:14:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Copying new pg_hba.conf file to original segments
20160624:16:28:14:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Configuring original segments
20160624:16:28:14:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Cleaning up temporary template files
20160624:16:28:16:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Starting Greenplum Database in restricted mode
20160624:16:28:30:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[ERROR]:-gpexpand failed: Primary must have mirror when mirroring enabled: 2|0|p|p|s|u|hitw-mck-sd
w-01|hitw-mck-sdw-01-1|40000|None|/data/gpdb_p1/gp0||/data/gpdb_p1/gp0/base/3803222,/data/gpdb_p1/gp0/base/10899,/data/gpdb_p1/gp0/base/1,/data/gpdb_p1/gp0/
base/102776162,/data/gpdb_p1/gp0/base/102732589,/data/gpdb_p1/gp0/base/10900,/data/gpdb_p1/gp0/base/16992 Exiting...
20160624:16:28:30:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[ERROR]:-Please run 'gpexpand -r -D gpexpand' to rollback to the original state.
20160624:16:28:30:001037 gpexpand:hitw-mck-mdw-01:gpadmin-[INFO]:-Shutting down gpexpand...

[gpadmin@hitw-mck-mdw-01 ~]$ cat gpexpand_inputfile_20160627_113214
hitw-mck-sdw-01:hitw-mck-sdw-01-1:40004:/data/gpdb_p5/gp24:27:24:p
hitw-mck-sdw-01:hitw-mck-sdw-01-2:40005:/data/gpdb_p6/gp25:28:25:p
hitw-mck-sdw-01:hitw-mck-sdw-01-3:40006:/data/gpdb_p7/gp26:29:26:p
hitw-mck-sdw-01:hitw-mck-sdw-01-4:40007:/data/gpdb_p8/gp27:30:27:p
hitw-mck-sdw-02:hitw-mck-sdw-02-1:40004:/data/gpdb_p5/gp28:31:28:p
hitw-mck-sdw-02:hitw-mck-sdw-02-2:40005:/data/gpdb_p6/gp29:32:29:p
hitw-mck-sdw-02:hitw-mck-sdw-02-3:40006:/data/gpdb_p7/gp30:33:30:p
hitw-mck-sdw-02:hitw-mck-sdw-02-4:40007:/data/gpdb_p8/gp31:34:31:p
hitw-mck-sdw-03:hitw-mck-sdw-03-1:40004:/data/gpdb_p5/gp32:35:32:p
hitw-mck-sdw-03:hitw-mck-sdw-03-2:40005:/data/gpdb_p6/gp33:36:33:p
hitw-mck-sdw-03:hitw-mck-sdw-03-3:40006:/data/gpdb_p7/gp34:37:34:p
hitw-mck-sdw-03:hitw-mck-sdw-03-4:40007:/data/gpdb_p8/gp35:38:35:p
hitw-mck-sdw-04:hitw-mck-sdw-04-1:40004:/data/gpdb_p5/gp36:39:36:p
hitw-mck-sdw-04:hitw-mck-sdw-04-2:40005:/data/gpdb_p6/gp37:40:37:p
hitw-mck-sdw-04:hitw-mck-sdw-04-3:40006:/data/gpdb_p7/gp38:41:38:p
hitw-mck-sdw-04:hitw-mck-sdw-04-4:40007:/data/gpdb_p8/gp39:42:39:p
hitw-mck-sdw-05:hitw-mck-sdw-05-1:40004:/data/gpdb_p5/gp40:43:40:p
hitw-mck-sdw-05:hitw-mck-sdw-05-2:40005:/data/gpdb_p6/gp41:44:41:p
hitw-mck-sdw-05:hitw-mck-sdw-05-3:40006:/data/gpdb_p7/gp42:45:42:p
hitw-mck-sdw-05:hitw-mck-sdw-05-4:40007:/data/gpdb_p8/gp43:46:43:p
hitw-mck-sdw-06:hitw-mck-sdw-06-1:40004:/data/gpdb_p5/gp44:47:44:p
hitw-mck-sdw-06:hitw-mck-sdw-06-2:40005:/data/gpdb_p6/gp45:48:45:p
hitw-mck-sdw-06:hitw-mck-sdw-06-3:40006:/data/gpdb_p7/gp46:49:46:p
hitw-mck-sdw-06:hitw-mck-sdw-06-4:40007:/data/gpdb_p8/gp47:50:47:p 

Cause

Once gpexpand is executed, it will first check if the GPDB system is configured with mirror segments by checking any entries in the system table gp_segment_configuration with role='m'.  In this case, there are no mirrors configured for primary segments; however, the standby master is provisioned, which also has role='m' in  gp_segment_configuration.  

gpadmin=# select * from gp_segment_configuration ;
dbid | content | role | preferred_role | mode | status | port | hostname | address | replication_port | san_mounts
------+---------+------+----------------+------+--------+-------+-----------------+-------------------+------------------+------------
2 | 0 | p | p | s | u | 40000 | hitw-mck-sdw-01 | hitw-mck-sdw-01-1 | |
3 | 1 | p | p | s | u | 40001 | hitw-mck-sdw-01 | hitw-mck-sdw-01-2 | |
7 | 5 | p | p | s | u | 40001 | hitw-mck-sdw-02 | hitw-mck-sdw-02-2 | |
11 | 9 | p | p | s | u | 40001 | hitw-mck-sdw-03 | hitw-mck-sdw-03-2 | |
15 | 13 | p | p | s | u | 40001 | hitw-mck-sdw-04 | hitw-mck-sdw-04-2 | |
19 | 17 | p | p | s | u | 40001 | hitw-mck-sdw-05 | hitw-mck-sdw-05-2 | |
23 | 21 | p | p | s | u | 40001 | hitw-mck-sdw-06 | hitw-mck-sdw-06-2 | |
4 | 2 | p | p | s | u | 40002 | hitw-mck-sdw-01 | hitw-mck-sdw-01-3 | |
8 | 6 | p | p | s | u | 40002 | hitw-mck-sdw-02 | hitw-mck-sdw-02-3 | |
12 | 10 | p | p | s | u | 40002 | hitw-mck-sdw-03 | hitw-mck-sdw-03-3 | |
16 | 14 | p | p | s | u | 40002 | hitw-mck-sdw-04 | hitw-mck-sdw-04-3 | |
20 | 18 | p | p | s | u | 40002 | hitw-mck-sdw-05 | hitw-mck-sdw-05-3 | |
24 | 22 | p | p | s | u | 40002 | hitw-mck-sdw-06 | hitw-mck-sdw-06-3 | |
5 | 3 | p | p | s | u | 40003 | hitw-mck-sdw-01 | hitw-mck-sdw-01-4 | |
9 | 7 | p | p | s | u | 40003 | hitw-mck-sdw-02 | hitw-mck-sdw-02-4 | |
13 | 11 | p | p | s | u | 40003 | hitw-mck-sdw-03 | hitw-mck-sdw-03-4 | |
17 | 15 | p | p | s | u | 40003 | hitw-mck-sdw-04 | hitw-mck-sdw-04-4 | |
21 | 19 | p | p | s | u | 40003 | hitw-mck-sdw-05 | hitw-mck-sdw-05-4 | |
25 | 23 | p | p | s | u | 40003 | hitw-mck-sdw-06 | hitw-mck-sdw-06-4 | |
6 | 4 | p | p | s | u | 40000 | hitw-mck-sdw-02 | hitw-mck-sdw-02-1 | |
10 | 8 | p | p | s | u | 40000 | hitw-mck-sdw-03 | hitw-mck-sdw-03-1 | |
14 | 12 | p | p | s | u | 40000 | hitw-mck-sdw-04 | hitw-mck-sdw-04-1 | |
18 | 16 | p | p | s | u | 40000 | hitw-mck-sdw-05 | hitw-mck-sdw-05-1 | |
22 | 20 | p | p | s | u | 40000 | hitw-mck-sdw-06 | hitw-mck-sdw-06-1 | |
26 | -1 | m | m | s | u | 5432 | hitw-mck-mdw-02 | hitw-mck-mdw-02 | |
1 | -1 | p | p | s | u | 5432 | hitw-mck-mdw-01 | hitw-mck-mdw-01 | |

Resolution

The issue has been reported to engineering with internal JIRA MPP-26457. Until the issue is fixed, please use the workaround below:

  1. Remove the standby master with "gpinitstandby -r".
  2. Run gpexpand to expand cluster.
  3. After gpexpand completes, run "gpinitstandby -s <standby hostname>" to reinitialize standby master.

Note- Services will not be impacted by "remove/reinitialize" standby master if GPDB system is on 4.3.x release. If it's on 4.2.x release, then schedule a downtime to implement the workaround.

 

Comments

Powered by Zendesk