Pivotal Knowledge Base

Follow

gpexpend Fails with "gpexpand_schema.tar: No space left on device"

Environment

  • Pivotal Greenplum Database (GPDB) 4.3.x
  • Operating System- Red Hat Enterprise Linux 6.x

Symptom

While running gpexpand fails with the error "No space left on device" because of the lack of space on the segment server.

Error Message:

20151103:10:45:49:182997 gpexpand:mdw:gpadmin-[INFO]:-Adding new segments into template pg_hba.conf
20151103:10:45:49:182997 gpexpand:mdw:gpadmin-[INFO]:-Creating schema tar file
20151103:10:51:00:182997 gpexpand:mdw:gpadmin-[INFO]:-Distributing template tar file to new hosts
20151103:10:56:08:182997 gpexpand:mdw:gpadmin-[ERROR]:-gpexpand failed: ExecutionError: 'Error Executing Command: ' occured. Details: 'GPSTART_INTERNAL_MASTER_ONLY=1 /usr/bin/scp -o 'StrictHostKeyChecking no' -r gpexpand_schema.tar sdw6.gphd.local:.' cmd had rc=1 completed=True halted=False
stdout=''
stderr='scp: ./gpexpand_schema.tar: No space left on device
'
Exiting...
20151103:10:56:08:182997 gpexpand:mdw:gpadmin-[ERROR]:-Please run 'gpexpand -r -D dca_expansion_db' to rollback to the original state.
20151103:10:56:08:182997 gpexpand:mdw:gpadmin-[INFO]:-Shutting down gpexpand... 

Cause 

gpexpand tar the  $MASTER_DATA_DIRECTORY. Usually, this includes a backup directory db_dumps. This creates a massive file that includes backup. The gpexpand tries to copy over this file to a segment server /home directory but often, there is not enough space on a segment server.

Resolution

Move the backup directory from MASTER_DATA_DIRECTORY. Usually /data has enough space so you can move it one level up to remove it from the tar file.

Comments

Powered by Zendesk