Pivotal Greenplum (GPDB) below 126.96.36.199
gpcrondump when trying to backup a database/table etc, errors out at step "Ensuring remaining free disk" with "Too many open files" or "Cannot allocate memory".
[.....] 20150401:11:06:38:011302 gpcrondump:mdw:gpadmin-[INFO]:-Vacuum mode type = Off 20150401:11:06:38:011302 gpcrondump:mdw:gpadmin-[INFO]:-Ensuring remaining free disk > 10 20150401:11:08:37:011302 gpcrondump:mdw:gpadmin-[CRITICAL]:-gpcrondump failed. (Reason='error 'ERROR: could not create temporary file base/17227/pgsql_tmp/workfile_set_HashJoin_Slice-1.XXXXWeXZe6/spillfile_f129609:Too many open files ERROR: could not seek in temporary file: Bad file descriptor ' in ' SELECT ALLTABLES.oid, ALLTABLES.schemaname, ALLTABLES.tablename FROM [....] [....]
File "/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line 421, in execute stdout=subprocess.PIPE, close_fds=True) File "/usr/local/greenplum-db/ext/python/lib/python2.6/subprocess.py", line 595, in __init__ errread, errwrite) File "/usr/local/greenplum-db/ext/python/lib/python2.6/subprocess.py", line 1009, in _execute_child self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory
This is caused due to the bug 24591 which is a consequence of python issue with multiple processes allocating too much memory or over-allocating file descriptors.
- Workaround: use "-b" option along with the gpcrondump command so that it can skip the disk space check functionality.
- Fix: Upgrade to 188.8.131.52 which fixes the issue.