Pivotal Knowledge Base

Follow

How to delete data and re-claim space from Data Domain System using gpddboost and nfs exports

Environment

Product Version
Pivotal Greenplum (GPDB) 4.3
Data Domain  

Symptom

When a user tries to delete data from Data Domain system and re-claim cleanable space using the command "gpcrondump -o --ddboost", following error occurs:

20130827:15:26:50|ddboost-[ERROR]:-ddboost create storage unit failed. Err = 5040
20130827:15:26:50|gpddboost-[ERROR]:-Could not connect to DD_host with DD_user and the DD_password.
20130827:15:26:53:008209 gpcrondump:dca1_mastr1:gpadmin-[ERROR]:-gpcrondump error: Failed to connect to DD_host with DD_user and the DD_password

Cause

The message indicating failed to connect to Data Domain with the username / password is misleading, the actual problem here is the error message "[ERROR]:-ddboost create storage unit failed", which gives us a hint that it is not able to create the needed storage unit. So a quick check on the Data Domain server shows that the filesystem is 100%. See the explaination below:

gpddb_user@DD640XIAE> filesys show space
Active Tier:
Resource           Size GiB   Used GiB   Avail GiB   Use%   Cleanable GiB
----------------   --------   --------   ---------   ----   -------------
/data: pre-comp           -   290616.1           -      -               -
/data: post-comp     7580.7     7580.7         0.0   100%             0.0
/ddvar                 78.7        2.2        72.6     3%               -
----------------   --------   --------   ---------   ----   -------------

Resolution

There are two ways to resolve this issue and reclaim space on Data Domian:

  • Free space on the Data Domain (unless more disk could be added)
  • Schedule cleanup activities on Data Domain

From DCA or GPDB software using "gpddboost" clear up some space:

  • See what is the default backup directory using: gpddboost --show-config
[gpadmin@dca1_mastr1 ~]$ gpddboost --show-config
20130827:18:47:52|ddboost-[DEBUG]:-Libraries were loaded successfully
20130827:18:47:52|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
Data Domain Hostname:dd640_1_1
Data Domain Boost Username:gpddb_user
Default Backup Directory:db_dumps
Data Domain default log level:WARNING
  • List the directory under db_dumps using: gpddboost --listDirectory --dir=db_dumps
[gpadmin@dca1_mastr1 ~]$ gpddboost --listDirectory  --dir=db_dumps
20130827:18:48:07|ddboost-[DEBUG]:-Libraries were loaded successfully
20130827:18:48:07|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
20130827:18:48:07|gpddboost-[DEBUG]:-ddp_open_dir(db_dumps)
20130227
20130301
20130311
20130312
[.....]
20130825
  • Use the command to delete the directory
gpddboost --del-dir=db_dumps/20130618

Note: Delete only the directories which are safe to delete.

  • Verify Cleanable Gib space.
sysadmin@DD640XIAE# filesys show space
Active Tier:
Resource           Size GiB   Used GiB   Avail GiB   Use%   Cleanable GiB
----------------   --------   --------   ---------   ----   -------------
/data: pre-comp           -   154789.5           -      -               -
/data: post-comp     7580.7     7511.2        69.5    99%          2038.5
/ddvar                 78.7        2.2        72.6     3%               -
----------------   --------   --------   ---------   ----   -------------

Note: Cleanable GiB indicates the space which could be reclaimed via Garbage Collection Process.

  • After deleting the directories, run the command after logging on DD as sysadmin to make the free space available:
'filesys clean start nowait' 

Note: This initiates Garbage Collection Process

  • Watch the status using: filesys clean watch or filesys clean status (It would take around 4-5 hours, but depends on the size)
sysadmin@DD640XIAE# filesys clean status
Cleaning started at 2013/08/27 06:00:02: phase 9 of 10 (copy)
 67.1% complete,     0 GiB free; time: phase  0:09:11, total  3:32:50
  • You could view the current space usage using after file sys clean is completed:
sysadmin@DD640XIAE# filesys show space

Login to Data Domain, export the nfs directory and mount the backup directory of DD on DCA and then delete the directories:

  • Verify the "mtree" list after logging on to DCA.
sysadmin@DD640XIAE# mtree list
Name                Pre-Comp (GiB)   Status
-----------------   --------------   ------
/data/col1/GPDB           133508.8   RW
/data/col1/backup              0.0   RW
-----------------   --------------   ------
  • Setup nfs export on Data Domain for mtree /data/col1/GPDB
sysadmin@DD640XIAE# nfs add /data/col1/GPDB   *
Hint: * is to allow all client, but you could mention IP like 172.10.10.0/24 as well.
sysadmin@DD640XIAE# nfs show clients
path              client          options
---------------   -------------   ----------------------------------------
/data/col1/GPDB   *               (rw,no_root_squash,no_all_squash,secure)

Note: Perform this steps if its have not been done already in 2.3. Mount the nfs export on DD on DCA master

[root@dca1_mastr1 ~]#  
mkdir /bkp_pivGPDB; 
mount -t nfs -o hard,intr,nfsvers=3,tcp,rsize=32768,wsize=32768 dd640_1_1:/data/col1/GPDB /bkp_pivGPDB
  • You could use Linux commands now on the shared filesystem for deleting data on Data Domain.(rm -rf /bkp_pivGPDB/db_dumps/20130620)
[root@dca1_mastr1 ~]# ls /bkp_pivGPDB/
db_dumps
[root@dca1_mastr1 ~]# ls /bkp_pivGPDB/db_dumps/
20130620  20130624  20130628  20130713  20130717  20130721  20130725  20130729  20130802  
  • After deleting the directories, run the command after logging on DD as sysadmin to make the free space available:
filesys clean start nowait 
  • Watch the status using the commands on Data Domain; watch the status using:
filesys clean status or filesys clean status

Note: Garbage collection process could take around 4-5 hours for around 10 TB of total disk space depending on the size.

Comments

Powered by Zendesk