Pivotal Knowledge Base

Follow

gpexpend fails with gpexpand_schema.tar: No space left on device

Environment

Product Version
Pivotal Greenplum (GPDB) 4.x.x
OS RHEL 6.x

Symptom

While running gpexpand fails with the error "No space left on device" because of the lack of space on the segment server.

Error Message:

20151103:10:45:49:182997 gpexpand:mdw:gpadmin-[INFO]:-Adding new segments into template pg_hba.conf
20151103:10:45:49:182997 gpexpand:mdw:gpadmin-[INFO]:-Creating schema tar file
20151103:10:51:00:182997 gpexpand:mdw:gpadmin-[INFO]:-Distributing template tar file to new hosts
20151103:10:56:08:182997 gpexpand:mdw:gpadmin-[ERROR]:-gpexpand failed: ExecutionError: 'Error Executing Command: ' occured. Details: 'GPSTART_INTERNAL_MASTER_ONLY=1 /usr/bin/scp -o 'StrictHostKeyChecking no' -r gpexpand_schema.tar sdw6.gphd.local:.' cmd had rc=1 completed=True halted=False
stdout=''
stderr='scp: ./gpexpand_schema.tar: No space left on device
'
Exiting...
20151103:10:56:08:182997 gpexpand:mdw:gpadmin-[ERROR]:-Please run 'gpexpand -r -D dca_expansion_db' to rollback to the original state.
20151103:10:56:08:182997 gpexpand:mdw:gpadmin-[INFO]:-Shutting down gpexpand... 

Cause 

gpexpand tar the  $MASTER_DATA_DIRECTORY. Usually this includes a backup directory db_dumps. This creates a massive file that includes backup. The gpexpand tries to copy over this file to a segment server /home directory but usually there is not enough space on a segment server.

Resolution

Move the backup directory from MASTER_DATA_DIRECTORY. Usually /data has enough space so you can move it one level up to remove it from the tar file.

Comments

Powered by Zendesk