- Pivotal Greenplum (GPDB) 4.3.x
- OS RHEL 6.x
gpdbrestore fails when open files limit (ulimit) has been reached with DDBOOST as an option.
[ERROR]:-gpdbrestore error: Dump file '...../gp_cdatabase_1_1_......' does not exist on Master
Old backup files stored in Data Domain that consumes all the open file limit while restoring. Once this limit has been reached, gpdbrestore fails and no new files from latest backup will be restored.
Currently, there is no provision in the system where we are logging ulimit error for gpdbrestore. Engineering has been informed about this.
So to verify this issue store the output of strace for gpdbrestore process
The following message is reported by strace
open("....../gp_segment_config_files_0_18........", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 65535
open("....../gp_segment_config_files_0_19........", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EMFILE (Too many open files)
write(2, "ddp_open_dir(......"..., 36) = 36
Above error message reported the file number last open is 65535, so if the ulimit is set at 65536, gpdbrestore fails.
You can verify the ulimit on your system by issuing below command on Linux server
After confirming the above message, please proceed with Resolution.
Note- To store strace output please make sure system has enough space as it may take GB of data depending on the number of backup files in Data Domain.
Purge old backups in Data Domain and then try gpdbrestore.