Pivotal Knowledge Base

Follow

dca_setup fails with python error "Error! pty.fork() failed: [Errno 12] Cannot allocate memory"

Environment

DCA V2 2.0.1.0 (Not limited to a specific DCA/version)

Problem

When running dca_setup from master on a V2/V1 system, the setup prematurely terminates failing at the cluster expansion level. This was observed while trying to add 160 nodes to an existing cluster of 32 nodes i.e., expanding from 2 racks to 10 racks.

Here is how the output of the command appears when it errors out, in dca_setup log file which is located under /home/gapdmin/gpAdminLogs directory:

20140415:16:47:45:110287 dca_setup:mdw:root-[INFO]:-Generate the SSH Keys for any new hosts and switches Action: Starting to run Generate the SSH Keys for any new ho
sts and switches Action
20140415:16:48:39:110287 dca_setup:mdw:root-[INFO]:-bash << "CMD_END_TEXT"
chown gpadmin:gpadmin /tmp/tmpOMkeNk
CMD_END_TEXT
20140415:16:49:55:110287 dca_setup:mdw:root-[ERROR]:-ActionRunException('Generate the SSH Keys for any new hosts and switches Action: run: Got exception for i-sw-11: Error! pty.fork() failed: [Errno 12] Cannot allocate memory',)
Traceback (most recent call last):
File "/opt/dca/lib/dca_setup/ui_action_base.py", line 105, in run
question_list = myExec.execute_actions(answer_map = answer_map)
File "/opt/dca/lib/dca_setup/setup_execute.py", line 54, in execute_actions
question_list = self._check_prereqs(answer_map = answer_map)
File "/opt/dca/lib/dca_setup/setup_execute.py", line 131, in _check_prereqs
question_list = self._check_prereqs_connectivity_main(action_list, answer_map)
File "/opt/dca/lib/dca_setup/setup_execute.py", line 304, in _check_prereqs_connectivity_main
question_list = self._run_regenerate_action(GenerateSSHKeysNewHostsAction, action_list, answer_map)
File "/opt/dca/lib/dca_setup/setup_execute.py", line 346, in _run_regenerate_action
self.generate_keys_action.execute()
File "/opt/dca/lib/dca_setup/action_base.py", line 207, in execute
self._run()
File "/opt/dca/lib/dca_setup/action_base.py", line 120, in _run
raise ActionRunException("%s: run: %s" % (self.name, e))
ActionRunException: Generate the SSH Keys for any new hosts and switches Action: run: Got exception for i-sw-11: Error! pty.fork() failed: [Errno 12] Cannot allocate memory
20140415:16:50:30:110287 dca_setup:mdw:root-[INFO]:-
Application terminated by user.

20140415:16:57:53:117140 dca_setup:mdw:root-[INFO]:-ping -c 1 -w 4 smdw
20140415:16:57:54:117140 dca_setup:mdw:root-[WARNING]:-The -a/--auto option has not affect without the -d/--dev flag
20140415:16:57:54:117140 dca_setup:mdw:root-[INFO]:-bash << "CMD_END_TEXT"
ipmitool -H mdw-sp -U root -P sephiroth lan print 1

Cause

Python which tries to preallocate memory based on the value determined from "ulimit -s"

Solution

The problem was resolved by reducing the "ulimit -s"/stack size on master host. Once the value on a DCA V2 was reduced from "10240" to "5120", cluster expansion was successful.

The error can easily be reproduced by increasing the value at session level (as root) and trying a gpssh, which preallocates memory before launching the threads.

[root@mdw sam]# ulimit -s 2097152
[root@mdw sam]# gpssh -f /data/sam/hostfile 
20140416:18:45:52:018641 gpssh:default-[ERROR]:-[Errno 12] Cannot allocate memory
Traceback (most recent call last):
  File "/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line 202, in run
    self.cmd.run()
  File "/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line 687, in run
    self.exec_context.execute(self)
  File "/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line 640, in execute
    LocalExecutionContext.execute(self,cmd)
  File "/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line 421, in execute
    stdout=subprocess.PIPE, close_fds=True)
  File "/usr/local/greenplum-db/ext/python/lib/python2.6/subprocess.py", line 595, in __init__
    errread, errwrite)
  File "/usr/local/greenplum-db/ext/python/lib/python2.6/subprocess.py", line 1009, in _execute_child
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory 

Comments

Powered by Zendesk