Pivotal Knowledge Base

Follow

How to recover failed/unreachable segment host to a new backup host?

Question

Segment server host can get physically damaged and could be left unrecoverable. How can we recover the failed segments on this damaged host to a new backup host? 

If you wish to move the segment directory to a new location or change the port then you may use the steps indicated on the article.

Solution

First perform the following checks

  1. Run 'gpstate -e' and confirm all segments that are located on the failed physical host are marked down. They should be in mirror mode. 
  2. In addition to gpstate user can also check segments status from psql prompt by running the following query
    select * from gp_segment_configuration where hostname = "<hostname>";
  3. Make sure the associated acting primary segments of the failed segments are up and in change-tracking. In other words, make sure the failed segments have successfully failed over to their counterparts.
    => gpstate -e
  4. If you can ssh into the failed host, make sure there are no orphaned segment processes
    ps -ef | grep -i postgres | grep -i silent
  5. If there are any orphaned segment processes, kill them
    kill <process_id> 

Please make sure that the new replacement host has everything configured exactly as the other segment servers such as;  OS version, locales, gpadmin user account, data directory locations created, ssh keys exchanged, number of network interfaces, network interface naming convention etc. 

Important

Please confirm that only primary/mirrors on the failed host are down. If segments on other hosts are down as well, you may need to first recover them before proceeding. 

After the above checks are completed, run the following command on the master: 

gprecoverseg -p <new_backup_host> 

Once command completes, run 'gpstate -e' to confirm that the new segments have been created and synced on the new backup host. 

 

Comments

Powered by Zendesk