Pivotal Knowledge Base

Follow

How to recover gptext when a node has been replaced

Issue:

Node replaced on a cluster that runs gptext and the disks have been reformatted. In this case gptext-recover will not recover the failed segments.

Error:

index is corrupted in both primary and mirror. Cannot recover.

Fix:

One check that can be done is to use gptext-recover with force option. Caveat is that the corrupted index needs to be recreated in this case: 

Command: 

gptext-recover -f -c <contentid> -r <role of the solr instance to be recovered>

If above does not work then try the following:

1) Replace failed hardware and recover all Greenplum database segments using full recovery. Also run gprecoverseg -r to correct the roles.
2) run 'gptext-stop' to stop gptext so that we could copy clean indexes from other machines
3) copy <segment_dir>/solr directory from corresponding machine which has same gpdb content id.
4) modify solr setting
 4.1) modify solr.xml:  set correct "enable.slave" and "enable.master" value according to segment role.
 4.2) change lib/log4j.properties to use correct directory, mostly replace 'mirror' with 'primary' or replace 'primary' with 'mirror' in the file
 4.3) change jetty.conf file
  -) PORT: use correct port according to segment role. Look at gpdb-solr.properties file for port value
  -) SOLR_HOME: point to correct directory according to segment role
5) double check the cluster, whether there is orphan solr processes by 'ps -ef|grep solr'. kill them if there is any
6) run 'gptext-start'
delete $GPTEXT-HOME/gptext-start.pid and $segdatadir/solr/jetty.pid if it report error.
 
All of these need to be taken care of by gptext-recover command. But there is a bug in that command and hence need to be done manually.

 

Comments

  • Avatar
    Michael Nemesh

    Thanks for documenting this. One error I found,m at least with respect to greenplum-text-1.1.0.0B1, in step 4.2 "logs/log4j.properties" should read "lib/log4j.properties"

Powered by Zendesk