Pivotal Knowledge Base

Follow

gpcrondump backup to ddboost fails with the error message "Creating path xx/xx/xx on ddboost failed. Err 5034"

Environment

Product Version
Pivotal Greenplum (GPDB) 4.3.x
OS RHEL 6.x
Data Domain  5.x

Symptom

Backup to Data Domain via ddboost fails with the error message "[ERROR]:-Creating path gp_production2/20160229/gp_dump_20160229160828_ao_state_file on ddboost failed. Err 5034"

The complete error message looks like the following:

[gpadmin@smdw brian]$ gpcrondump -x test --ddboost
20160229:16:08:24:019500 gpcrondump:smdw:gpadmin-[INFO]:-Starting gpcrondump with args: -x test --ddboost
20160229:16:08:24:019500 gpcrondump:smdw:gpadmin-[INFO]:-Bypassing disk space checks due to DDBoost parameters
20160229:16:08:25|ddboost-[DEBUG]:-Libraries were loaded successfully
20160229:16:08:25|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-The specified DDBoost credentials are OK
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:----------------------------------------------------
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-Master Greenplum Instance dump parameters
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:----------------------------------------------------
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-Dump type                            = Full database
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-Database to be dumped                = test
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-Master port                          = 5432
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-Master data directory                = /data/master/boc_4361_-1
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-Run post dump program                = Off
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-Rollback dumps                       = Off
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-Dump file compression                = On
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-Clear old dump files                 = Off
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-Update history table                 = Off
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-Secure config files                  = Off
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-Dump global objects                  = Off
20160229:16:08:28:019500 gpcrondump:smdw:gpadmin-[INFO]:-Vacuum mode type                     = Off

Continue with Greenplum dump Yy|Nn (default=N):
> y
20160229:16:08:30:019500 gpcrondump:smdw:gpadmin-[INFO]:-Directory /data/master/boc_4361_-1/gp_production2/20160229 exists
20160229:16:08:30:019500 gpcrondump:smdw:gpadmin-[INFO]:-Checked /data/master/boc_4361_-1 on master
20160229:16:08:31:019500 gpcrondump:smdw:gpadmin-[INFO]:-Configuring for single database dump
20160229:16:08:31:019500 gpcrondump:smdw:gpadmin-[INFO]:-Adding compression parameter
20160229:16:08:31:019500 gpcrondump:smdw:gpadmin-[INFO]:-Adding --no-expand-children
20160229:16:08:31:019500 gpcrondump:smdw:gpadmin-[INFO]:-Dump process command line gp_dump -p 5432 -U gpadmin --gp-d=gp_production2/20160229 --gp-r=/data/master/boc_4361_-1/gp_production2/20160229 --gp-s=p --gp-k=20160229160828 --no-lock --ddboost --gp-c --no-expand-children test
20160229:16:08:31:019500 gpcrondump:smdw:gpadmin-[INFO]:-Starting Dump process
20160229:16:18:39:019500 gpcrondump:smdw:gpadmin-[WARNING]:-Dump process returned exit code 1
20160229:16:18:39:019500 gpcrondump:smdw:gpadmin-[INFO]:-Timestamp key = 20160229160828
20160229:16:18:40:019500 gpcrondump:smdw:gpadmin-[INFO]:-Releasing pg_class lock
20160229:16:18:43:019500 gpcrondump:smdw:gpadmin-[CRITICAL]:-Error occurred: non-zero rc: 1
 Command was: 'gpddboost --copyToDDBoost    --from-file=/data/master/boc_4361_-1/gp_production2/20160229/gp_dump_20160229160828_ao_state_file --to-file=gp_production2/20160229/gp_dump_20160229160828_ao_state_file'
rc=1, stdout='', stderr='20160229:16:18:40|ddboost-[DEBUG]:-Libraries were loaded successfully
20160229:16:18:40|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
20160229:16:18:40|gpddboost-[ERROR]:-mkdir failed. err 5034
20160229:16:18:40|gpddboost-[ERROR]:-Creating path gp_production2/20160229/gp_dump_20160229160828_ao_state_file on ddboost failed. Err 5034

Attempting to list the files in the directory also results in the "Error Code: 5034" as shown below:

[gpadmin@smdw brian]$ gpddboost --ls /
20160229:16:08:07|ddboost-[DEBUG]:-Libraries were loaded successfully
20160229:16:08:07|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
20160229:16:08:07|gpddboost-[ERROR]:-Opening directory / on ddboost failed. Err 5034
[gpadmin@smdw brian]$

The file libDDBoost.log (located in the same directory from where the command was called) shows the following error message, "Permission denied":

02/29 17:08:07: [4C1B:83EC10] ddp_open_dir() failed, Path GPDB//, Err: 5034-nfs lookup failed (nfs: Permission denied)
02/29 17:18:40: [5066:3999CF0] ddp_mkdir() failed, Path GPDB/gp_production2, mode 493, Err: 5034-nfs lookup failed (nfs: Permission denied)
02/29 17:18:40: [5066:3999CF0] ddp_unlink() failed, Path GPDB/gp_production2/20160229/gp_dump_20160229160828_ao_state_file, Err: 5034-nfs lookup failed (nfs: Permission denied)

The configuration of ddboost on the client side is shown here:

[gpadmin@smdw brian]$ gpddboost --show-config
20160229:16:07:54|ddboost-[DEBUG]:-Libraries were loaded successfully
20160229:16:07:54|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
Data Domain Hostname:dd_1.dca
Data Domain Boost Username:gpdbbooster
Default Backup Directory:gp_production2
Data Domain default log level:WARNING

Cause

The cause of the issue is that the storage unit where gpcrondump is trying to write does not have the necessary permission to write.

From the DDBoost server, the mtree storage unit "GPDB" which hosts the sub-directory "gp_production2" is owned by "sysadmin" and not the user "gpdbbooster", which is the user that the gpcrondump uses to write to the storage-unit.

sysadmin@dd_1# ddboost storage-unit show
Name   Pre-Comp (GiB)   Status   User
----   --------------   ------   --------
GPDB              0.8   RW       sysadmin
----   --------------   ------   --------
 Q    : Quota Defined
 RO   : Read Only
 RW   : Read Write
 RD   : Replication Destination

Resolution

On the Data Domain , connect as superuser (i.e. sysadmin) and change the storage-unit owner to the user that should have the permission to write to the storage-unit.

ddboost storage-unit modify <storage-unit> user <ddboost-user>  

You may refer to the example shown below:

sysadmin@dd_1# ddboost storage-unit modify GPDB user gpdbbooster
Storage-unit "GPDB" modified for user "gpdbbooster".

And try to backup to ddboost again.

Comments

Powered by Zendesk