Pivotal Knowledge Base

Follow

gpinitstandby Fails With Error "pg_basebackup: could not read COPY data: server closed the connect" due to permission issue with files in the directory

Problem

Adding a standby is failing with the following error:  

20150605:16:04:28:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master initialization parameters
20150605:16:04:28:209741 gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
20150605:16:04:28:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master hostname = mdw.gphd.local
20150605:16:04:28:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master data directory = /data/master/gpseg-1
20150605:16:04:28:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum master port = 5432
20150605:16:04:28:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master hostname = smdw
20150605:16:04:28:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master port = 5432
20150605:16:04:28:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum standby master data directory = /data/master/gpseg-1
20150605:16:04:28:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Greenplum update system catalog = On
20150605:16:04:28:209741 gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
20150605:16:04:28:209741 gpinitstandby:mdw:gpadmin-[INFO]:- Filespace locations
20150605:16:04:28:209741 gpinitstandby:mdw:gpadmin-[INFO]:------------------------------------------------------
20150605:16:04:28:209741 gpinitstandby:mdw:gpadmin-[INFO]:-pg_system -> /data/master/gpseg-1
20150605:16:04:30:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Syncing Greenplum Database extensions to standby
20150605:16:04:30:209741 gpinitstandby:mdw:gpadmin-[INFO]:-The packages on smdw are consistent.
20150605:16:04:30:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Updating pg_hba.conf file...
20150605:16:04:30:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Updating pg_hba.conf file on segments...
20150605:16:04:36:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Adding standby master to catalog...
20150605:16:04:36:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Database catalog updated successfully.
20150605:16:06:16:209741 gpinitstandby:mdw:gpadmin-[ERROR]:-Failed to copy filespace directories from master to standby.
20150605:16:06:16:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Trying to rollback changes that have been made...
20150605:16:06:16:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Backing up pg_hba.conf file...
20150605:16:06:16:209741 gpinitstandby:mdw:gpadmin-[ERROR]:-Failed to remove standby from master catalog.
20150605:16:06:16:209741 gpinitstandby:mdw:gpadmin-[INFO]:-Removing filespace directories on standby master...
20150605:16:06:22:209741 gpinitstandby:mdw:gpadmin-[ERROR]:-Error initializing standby master: ExecutionError: 'non-zero rc: 1' occured. Details: 'ssh -o 'StrictHostKeyChecking no' smdw ". /usr/local/greenplum-db/./greenplum_path.sh; pg_basebackup -x -R -E ./pg_log -E ./gp_dumps -E ./gpperfmon/data -D /data/master/gpseg-1 -h mdw.gphd.local -p 5432"' cmd had rc=1 completed=True halted=False
stdout=''
stderr='WARNING: skipping special file "./db_dumps"
pg_basebackup: could not read COPY data: server closed the connect

Cause

There was a file owned by root in the $MASTER_DATA_DIRECTORY

[gpadmin@gpdbsne gpsne-1]$ find $MASTER_DATA_DIRECTORY -user root
/data/master/gpsne-1/pg_hba.conf.bak

Solution

Correct the permission issue so that the gpadmin user is the owner and is able to copy the file.

Comments

  • Avatar
    Fernando

    Hi Krzysztof,

    Another recommendation would be to look for any other file with a big size. I have had a similar issue and the permissions of the files were fine, but there was a different directory in $MASTER_DATA_DIRECTORY, a sql file from client with almost 500MB.

    Try to run also the command below to find the 10 biggest files in the directory:

    find $MASTER_DATA_DIRECTORY -printf '%s %p\n'| sort -nr | head -10

    Another thing that can be done to troubleshoot is run the pg_basebackup command failing on the standby master in verbose mode:

    Example:

    pg_basebackup -v -x -R -E ./pg_log -E ./gp_dumps -E ./gpperfmon/data -D /data/master/gpseg-1 -h mdw.gphd.local -p 5432

    And then compare what's missing in the directory created in the standby master data directory with the master data directory.

  • Avatar
    Gowri Kothandaraman

    Hi Team,

    I had same issue .upon checking there was Two huge file which caused problem for me.In my case it is gpmmon log file

    [gpadmin@acmpmrkd101]-/home/gpadmin >find $MASTER_DATA_DIRECTORY -printf '%s %p\n'| sort -nr | head -10
    30404871609 /data/master/gpseg-1/gpperfmon/logs/gpmmon.2015.06.11_012042.log
    1566053727 /data/master/gpseg-1/gpperfmon/logs/gpmmon.2015.03.11_111428.log
    1073741824 /data/master/gpseg-1/base/22212/2658
    886702080 /data/master/gpseg-1/base/22212/1249
    591953920 /data/master/gpseg-1/base/1065190/23521583
    495026176 /data/master/gpseg-1/base/22212/2619
    474939392 /data/master/gpseg-1/base/1065190/2658
    397901824 /data/master/gpseg-1/base/1065190/1249
    377946112 /data/master/gpseg-1/base/1065190/2619
    347144192 /data/master/gpseg-1/base/22212/2659

    once i moved those file gpinitstandby work perfectly .

    [gpadmin@acmpmrkd101]-/home/gpadmin >find $MASTER_DATA_DIRECTORY -printf '%s %p\n'| sort -nr | head -10
    1073741824 /data/master/gpseg-1/base/22212/2658
    886702080 /data/master/gpseg-1/base/22212/1249
    591953920 /data/master/gpseg-1/base/1065190/23521583
    495091712 /data/master/gpseg-1/base/22212/2619
    474939392 /data/master/gpseg-1/base/1065190/2658
    397901824 /data/master/gpseg-1/base/1065190/1249
    377946112 /data/master/gpseg-1/base/1065190/2619
    347144192 /data/master/gpseg-1/base/22212/2659

Powered by Zendesk