Pivotal Knowledge Base

Follow

[ERROR]:-gpdbrestore error: Dump file '....../gp_cdatabase_1_1_.......' does not exist on Master

Environment

Product Version
Pivotal Greenplum (GPDB) 4.3.x
OS RHEL 6.x
Others  

Symptom

gpdbrestore fails when open files limit (ulimit) has been reached with DDBOOST as an option. 

Error Message:

[ERROR]:-gpdbrestore error: Dump file '...../gp_cdatabase_1_1_......' does not exist on Master

Cause 

Old backup files stored in Data Domain that consumes all the open file limit while restoring. Once this limit has been reached, gpdbrestore fails and no new files from latest backup will be restored.

RCA

Currently, there is no provision in the system where we are logging ulimit error for gpdbrestore. Engineering has been informed about this. 

So to verify this issue store the output of strace for gpdbrestore process

The following message is reported by strace 

open("....../gp_segment_config_files_0_18........", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 65535
open("....../gp_segment_config_files_0_19........", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EMFILE (Too many open files)
write(2, "ddp_open_dir(......"..., 36) = 36

Above error message reported the file number last open is 65535, so if the ulimit is set at 65536, gpdbrestore fails.

You can verify the ulimit on your system by issuing below command on Linux server

ulimit -n  

After confirming the above message , please proceed on Resolution.

Note: To store strace output please make sure system has enough space as it may take GB of data depending on number of backup files in Data Domain

Resolution

Purge old backups in Data Domain and then try gpdbrestore.

 

Comments

Powered by Zendesk