Pivotal Knowledge Base

Follow

Incremental GPRECOVERSEG Fails with Error: "Persistent Mirroring State is Disabled"

Environment

 Product  Version
 Pivotal Greenplum  4.3.10.0 and above
 OS  RHEL 6.x

Summary 

The article outlines and describes the steps to follow when gprecoverseg (incremental) fails with error "persistent mirroring state is disabled" for a particular segment.

Error message:

Incremental recovery for versions >= 4.3.10.0 is failing and returning the following error: 

[gpadmin@mdw cluster]$ gprecoverseg -a
20170210:15:42:25:020543 gprecoverseg:mdw:gpadmin-[INFO]:-Starting gprecoverseg with args: -a
20170210:15:42:25:020543 gprecoverseg:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.10.0 build commit: f413ff3b006655f14b6b9aa217495ec94da5c96c'
20170210:15:42:25:020543 gprecoverseg:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.10.0 build commit: f413ff3b006655f14b6b9aa217495ec94da5c96c) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Oct 21 2016 19:36:26'
20170210:15:42:25:020543 gprecoverseg:mdw:gpadmin-[INFO]:-Checking if segments are ready to connect
20170210:15:42:25:020543 gprecoverseg:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20170210:15:42:25:020543 gprecoverseg:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20170210:15:42:27:020543 gprecoverseg:mdw:gpadmin-[WARNING]:-Segments with dbid 4, 5 not recovered; persistent mirroring state is disabled.
20170210:15:42:27:020543 gprecoverseg:mdw:gpadmin-[INFO]:-No segments to recover

Cause

This error is caused due to a new test for mirroring that was introduced in 4.3.10.0 which checks for inconsistent mirroring state. 

Reference: Readme 4.3.10.0 (gpcheckcat Utility: New and Enhanced Tests).

Check:

Run the following query in PSQL to find out the content IDs affected by this issue: 

SELECT gp_segment_id,mirror_existence_state,count(*) from gp_dist_random('gp_persistent_relation_node') where mirror_existence_state=1 group by 1,2;

Resolution

If a recovery is immediately needed, run full recovery to overcome this error for the segments affected. For any other segments that are marked down (not affected by this error), a separate incremental recovery can be done using this method.

Full Recovery:

gprecoverseg -F  

In order to avoid running a full recovery everytime the affected segment is marked down, run a full gpcheckcat and open a new Severity 2 (S2) Support ticket with Pivotal Support

Comments

Powered by Zendesk