Pivotal Knowledge Base

Follow

Incremental gprecoverseg Fails with the Error "Persistent Mirroring State is Disabled"

Environment

  • Pivotal Greenplum Database (GPDB) 4.3.x
  • Operating System- Red Hat Enterprise Linux 6.x

Symptom 

The article outlines and describes the steps to follow when gprecoverseg (incremental) fails with error "persistent mirroring state is disabled" for a particular segment.

Error message:

Incremental recovery for versions >= 4.3.10.0 is failing and returning the following error: 

[gpadmin@mdw cluster]$ gprecoverseg -a
20170210:15:42:25:020543 gprecoverseg:mdw:gpadmin-[INFO]:-Starting gprecoverseg with args: -a
20170210:15:42:25:020543 gprecoverseg:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.10.0 build commit: f413ff3b006655f14b6b9aa217495ec94da5c96c'
20170210:15:42:25:020543 gprecoverseg:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.10.0 build commit: f413ff3b006655f14b6b9aa217495ec94da5c96c) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Oct 21 2016 19:36:26'
20170210:15:42:25:020543 gprecoverseg:mdw:gpadmin-[INFO]:-Checking if segments are ready to connect
20170210:15:42:25:020543 gprecoverseg:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20170210:15:42:25:020543 gprecoverseg:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20170210:15:42:27:020543 gprecoverseg:mdw:gpadmin-[WARNING]:-Segments with dbid 4, 5 not recovered; persistent mirroring state is disabled.
20170210:15:42:27:020543 gprecoverseg:mdw:gpadmin-[INFO]:-No segments to recover

Cause

This error is caused due to a new test for mirroring that was introduced in 4.3.10.0 which checks for inconsistent mirroring state. 

Reference: Readme 4.3.10.0 (gpcheckcat Utility: New and Enhanced Tests).

Check:

Run the following query in PSQL to find out the content IDs affected by this issue: 

SELECT gp_segment_id,mirror_existence_state,count(*) from gp_dist_random('gp_persistent_relation_node') where mirror_existence_state=1 group by 1,2;

Resolution

If a recovery is immediately needed, run full recovery to overcome this error for the segments affected. For any other segments that are marked down (not affected by this error), a separate incremental recovery can be done using this method.

Full Recovery:

gprecoverseg -F  

In order to avoid running a full recovery everytime the affected segment is marked down, run a full gpcheckcat and open a new Severity 2 (S2) Support ticket with Pivotal Support

Comments

Powered by Zendesk