Pivotal Knowledge Base

Follow

gpdbrestore using --truncate option combined with --table-file option fails with error message "CRITICAL:-gpdbrestore failed."

Environment

Product Version
Pivotal Greenplum (GPDB) 4.3.5.x

Symptom

While restoring from the backup using --truncate option combined with --table-file option, if the dump file contains a new table that is not present in the target database, and the new table's name is present in the file, the process aborts with an error.

" CRITICAL:-gpdbrestore failed. (Reason=''str' object is not callable') exiting..."

Cause

Important parameters involved in the restore command - 

--truncate - Truncate table data before restoring data to the table from the backup. If this option is not specified, existing table data is not removed before data is restored to the table.This option is supported only when restoring a set of tables with the option -T or --table-file.This option is not supported with the -e option.

--table-file - Specify a file that contains a list of table names to restore. The file contains any number of table names, listed one per line.

When --truncate option is specified, a truncate statement is generated for each table listed in the --table-file. When these statements are executed, if a table specified in the --table-file does not exist currently in the destination database, an error is generated.
 
Example:
 
File "tables.txt" contains:
public.t1
public.t2
public.t2
 
Table public.t2 does not exist in destination database (the database where data is restored).
 
Command "gpdbrestore -t <key> --truncate --table-file tables.txt" generates truncate commands and fails when the statement "TRUNCATE TABLE public.t2;" is attempted to be executed.

Reproduction Steps

1. Create Database

template1=# create database fpl_test;
CREATE DATABASE

2. Connect to it and check the version.

template1=# \c fpl_test 
You are now connected to database "fpl_test" as user "gpadmin".
fpl_test=# select version();
version
------------------------------------------------------------------------------------------------------------------------------------------------------
PostgreSQL 8.2.15 (Greenplum Database 4.3.5.1 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 14 2015 14:07:14
(1 row)

3. Create Schema

fpl_test=# create schema fpl;
CREATE SCHEMA

4. Create test tables to be backed up

fpl_test=# create table fpl.x1(a int);
NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'a' as the Greenplum Database data distribution key for this table.
HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE

fpl_test=# create table fpl.x2(a int);
NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'a' as the Greenplum Database data distribution key for this table.
HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE
fpl_test=# create table fpl.x3(a int);
NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'a' as the Greenplum Database data distribution key for this table.
HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE

fpl_test=# create table fpl.x4(a int);
NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'a' as the Greenplum Database data distribution key for this table.
HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE
fpl_test=# \dt
List of relations
Schema | Name | Type | Owner | Storage
--------+------+-------+---------+---------
public | x1 | table | gpadmin | heap
public | x2 | table | gpadmin | heap
public | x3 | table | gpadmin | heap
public | x4 | table | gpadmin | heap
(4 rows)
fpl_test=# \q

5. Take full database backup 

[gpadmin@mdw ~]$ gpcrondump -x fpl_test
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Starting gpcrondump with args: -x fpl_test
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:----------------------------------------------------
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Master Greenplum Instance dump parameters
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:----------------------------------------------------
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump type = Full database
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Database to be dumped = fpl_test
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Master port = 8004
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Master data directory = /data/master/eddie_4351-1/
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Run post dump program = Off
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Rollback dumps = Off
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump file compression = On
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Clear old dump files = Off
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Update history table = Off
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Secure config files = Off
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump global objects = Off
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Vacuum mode type = Off
20151028:11:33:51:006542 gpcrondump:mdw:gpadmin-[INFO]:-Ensuring remaining free disk > 10

Continue with Greenplum dump Yy|Nn (default=N):
> y
20151028:11:33:56:006542 gpcrondump:mdw:gpadmin-[INFO]:-Directory /data/master/eddie_4351-1/db_dumps/20151028 not found, will try to create
20151028:11:33:56:006542 gpcrondump:mdw:gpadmin-[INFO]:-Created /data/master/eddie_4351-1/db_dumps/20151028
20151028:11:33:56:006542 gpcrondump:mdw:gpadmin-[INFO]:-Checked /data/master/eddie_4351-1/ on master
20151028:11:33:57:006542 gpcrondump:mdw:gpadmin-[INFO]:-Configuring for single database dump
20151028:11:33:57:006542 gpcrondump:mdw:gpadmin-[INFO]:-Validating disk space
20151028:11:33:58:006542 gpcrondump:mdw:gpadmin-[INFO]:-Adding compression parameter
20151028:11:33:58:006542 gpcrondump:mdw:gpadmin-[INFO]:-Adding --no-expand-children
20151028:11:33:58:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump process command line gp_dump -p 8004 -U gpadmin --gp-d=db_dumps/20151028 --gp-r=/data/master/eddie_4351-1/db_dumps/20151028 --gp-s=p --gp-k=20151028113351 --no-lock --gp-c --no-expand-children fpl_test
20151028:11:33:58:006542 gpcrondump:mdw:gpadmin-[INFO]:-Starting Dump process
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump process returned exit code 0
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Timestamp key = 20151028113351
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Checked master status file and master dump file.
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump status report
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:----------------------------------------------------
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Target database = fpl_test
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump subdirectory = 20151028
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump type = Full database
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Clear old dump directories = Off
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump start time = 11:33:51
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump end time = 11:34:00
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Status = COMPLETED
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump key = 20151028113351
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Dump file compression = On
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Vacuum mode type = Off
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-Exit code zero, no warnings generated
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:----------------------------------------------------
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[WARNING]:-Found neither /usr/local/GP-4.3.5.1/bin/mail_contacts nor /home/gpadmin/mail_contacts
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[WARNING]:-Unable to send dump email notification
20151028:11:34:00:006542 gpcrondump:mdw:gpadmin-[INFO]:-To enable email notification, create /usr/local/GP-4.3.5.1/bin/mail_contacts or /home/gpadmin/mail_contacts containing required email addresses

6. Create another test table

create table fpl.x5(a int);

fpl_test=# \dt fpl.*
List of relations
Schema | Name | Type | Owner | Storage
--------+------+-------+---------+---------
fpl | x1 | table | gpadmin | heap
fpl | x2 | table | gpadmin | heap
fpl | x3 | table | gpadmin | heap
fpl | x4 | table | gpadmin | heap
fpl | x5 | table | gpadmin | heap
(5 rows)

7. Take incremental Backup to add x5

[gpadmin@mdw ~]$ gpcrondump --incremental -x fpl_test
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Starting gpcrondump with args: --incremental -x fpl_test
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Bypassing disk space checks for incremental backup
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:----------------------------------------------------
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Master Greenplum Instance dump parameters
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:----------------------------------------------------
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump type = Incremental
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Database to be dumped = fpl_test
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Master port = 8004
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Master data directory = /data/master/eddie_4351-1/
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Run post dump program = Off
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Rollback dumps = Off
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump file compression = On
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Clear old dump files = Off
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Update history table = Off
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Secure config files = Off
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump global objects = Off
20151028:11:38:15:008524 gpcrondump:mdw:gpadmin-[INFO]:-Vacuum mode type = Off
Continue with Greenplum dump Yy|Nn (default=N):
> y
20151028:11:38:21:008524 gpcrondump:mdw:gpadmin-[INFO]:-Directory /data/master/eddie_4351-1/db_dumps/20151028 exists
20151028:11:38:21:008524 gpcrondump:mdw:gpadmin-[INFO]:-Checked /data/master/eddie_4351-1/ on master
20151028:11:38:22:008524 gpcrondump:mdw:gpadmin-[INFO]:-Adding compression parameter
20151028:11:38:22:008524 gpcrondump:mdw:gpadmin-[INFO]:-Adding --incremental
20151028:11:38:22:008524 gpcrondump:mdw:gpadmin-[INFO]:-Adding --no-expand-children
20151028:11:38:22:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump process command line gp_dump -p 8004 -U gpadmin --gp-d=db_dumps/20151028 --gp-r=/data/master/eddie_4351-1/db_dumps/20151028 --gp-s=p --gp-k=20151028113815 --no-lock --gp-c --incremental --no-expand-children fpl_test --table-file=/tmp/dirty_backup_list__SsEIq
20151028:11:38:22:008524 gpcrondump:mdw:gpadmin-[INFO]:-Starting Dump process
20151028:11:38:29:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump process returned exit code 0
20151028:11:38:29:008524 gpcrondump:mdw:gpadmin-[INFO]:-Timestamp key = 20151028113815
20151028:11:38:29:008524 gpcrondump:mdw:gpadmin-[INFO]:-Checked master status file and master dump file.
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Releasing pg_class lock
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump status report
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:----------------------------------------------------
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Target database = fpl_test
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump subdirectory = 20151028
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump type = Incremental
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Clear old dump directories = Off
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump start time = 11:38:15
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump end time = 11:38:29
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Status = COMPLETED
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump key = 20151028113815
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Dump file compression = On
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Vacuum mode type = Off
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-Exit code zero, no warnings generated
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:----------------------------------------------------
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[WARNING]:-Found neither /usr/local/GP-4.3.5.1/bin/mail_contacts nor /home/gpadmin/mail_contacts
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[WARNING]:-Unable to send dump email notification
20151028:11:38:30:008524 gpcrondump:mdw:gpadmin-[INFO]:-To enable email notification, create /usr/local/GP-4.3.5.1/bin/mail_contacts or /home/gpadmin/mail_contacts containing required email addresses
Backup completes successfully.

8. Drop the newly created table

fpl_test=# drop table fpl.x5;
DROP TABLE
fpl_test=# \dt fpl.*;
List of relations
Schema | Name | Type | Owner | Storage
--------+------+-------+---------+---------
fpl | x1 | table | gpadmin | heap
fpl | x2 | table | gpadmin | heap
fpl | x3 | table | gpadmin | heap
fpl | x4 | table | gpadmin | heap
(4 rows)

9. Below are the contents of file that needs to be restored, please note we have added the table in the list which we have already dropped.

fpl.x1
fpl.x2
fpl.x3
fpl.x4
fpl.x5

10. Try restoring using the incremental backup.

[gpadmin@mdw ~]$ gpdbrestore --noanalyze -t 20151028113815 -a --truncate --table-file shubh/restore_dd
20151028:11:40:57:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Starting gpdbrestore with args: --noanalyze -t 20151028113815 -a --truncate --table-file shubh/restore_dd
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-------------------------------------------
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Greenplum database restore parameters
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-------------------------------------------
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Restore type = Incremental Table Restore
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Database name = fpl_test
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-------------------------------------------
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Table restore list
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-------------------------------------------
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Table = fpl.x1
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Table = fpl.x2
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Table = fpl.x3
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Table = fpl.x4
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Table = fpl.x5
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Restore method = Specific table restore
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Restore timestamp = 20151028113815
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Restore compressed dump = On
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Restore global objects = Off
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-Array fault tolerance = f
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[INFO]:-------------------------------------------
20151028:11:40:58:009817 gpdbrestore:mdw:gpadmin-[CRITICAL]:-gpdbrestore failed. (Reason=''str' object is not callable') exiting...

Note: The restoration fails since the table is not present in the destination database, but it's name is present in the file list supplied. 

Resolution

The only solution is to create the extra table present in the file list at the destination database.

Comments

  • Avatar
    Archana Sundararam

    Also another workaround is to manually truncate all the tables that need to be restored with the IF EXISTS option and then try the restore only command.

Powered by Zendesk