Segment server host can get physically damaged and could be left unrecoverable. How can we recover the failed segments on this damaged host to a new backup host?
If you wish to move the segment directory to a new location or change the port then you may use the steps indicated in the article.
First, perform the following checks
- Run 'gpstate -e' and confirm all segments that are located on the failed physical host are marked down. They should be in mirror mode.
- In addition to gpstate user can also check segments status from psql prompt by running the following query
select * from gp_segment_configuration where hostname = "<hostname>";
- Make sure the associated acting primary segments of the failed segments are up and in change-tracking. In other words, make sure the failed segments have successfully failed over to their counterparts.
=> gpstate -e
- If you can ssh into the failed host, make sure there are no orphaned segment processes
ps -ef | grep -i postgres | grep -i silent
- If there are any orphaned segment processes, kill them
Please make sure that the new replacement host has everything configured exactly as the other segment servers such as; OS version, locales, gpadmin user account, data directory locations created, ssh keys exchanged, number of network interfaces, network interface naming convention etc.
Please confirm that only primary/mirrors on the failed host are down. If segments on other hosts are down as well, you may need to first recover them before proceeding.
After the above checks are completed, run the following command on the master:
gprecoverseg -p <new_backup_host>
Once the command completes, run 'gpstate -e' to confirm that the new segments have been created and synced with the new backup host.