Pivotal Knowledge Base

Follow

gpcrondump fails with "OSError: [Errno 12] Cannot allocate memory"

Environment

GPDB < 4.3.5.0

Problem

gpcrondump when trying to backup a database / table etc , errors out at step "Ensuring remaining free disk" with "Too many open files" or "Cannot allocate memory".

Examples:

[.....]
20150401:11:06:38:011302 gpcrondump:mdw:gpadmin-[INFO]:-Vacuum mode type                     = Off
20150401:11:06:38:011302 gpcrondump:mdw:gpadmin-[INFO]:-Ensuring remaining free disk         > 10
20150401:11:08:37:011302 gpcrondump:mdw:gpadmin-[CRITICAL]:-gpcrondump failed. (Reason='error 'ERROR:  could not create temporary file base/17227/pgsql_tmp/workfile_set_HashJoin_Slice-1.XXXXWeXZe6/spillfile_f129609:Too many open files
ERROR:  could not seek in temporary file: Bad file descriptor
' in '
    
SELECT ALLTABLES.oid, ALLTABLES.schemaname, ALLTABLES.tablename FROM 

[....]
[....]
File "/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line 421, in execute stdout=subprocess.PIPE, close_fds=True) File "/usr/local/greenplum-db/ext/python/lib/python2.6/subprocess.py", line 595, in __init__ errread, errwrite) File "/usr/local/greenplum-db/ext/python/lib/python2.6/subprocess.py", line 1009, in _execute_child self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory

Cause

This is caused due to the bug 24591 which is a consequence of python issue with multiple processes allocating too much memory or over allocating file descriptors.

Solution

  • Workaround: use "-b" option along with the gpcrondump command so that it can skip the disk space check functionality.
  • Fix: Upgrade to 4.3.5.0 which fixes the issue.

Comments

Powered by Zendesk