Node replaced on a cluster that runs gptext and the disks have been reformatted. In this case gptext-recover will not recover the failed segments.
index is corrupted in both primary and mirror. Cannot recover.
One check that can be done is to use gptext-recover with force option. Caveat is that the corrupted index needs to be recreated in this case:
gptext-recover -f -c <contentid> -r <role of the solr instance to be recovered>
If above does not work then try the following:
1) Replace failed hardware and recover all Greenplum database segments using full recovery. Also run gprecoverseg -r to correct the roles.
2) run 'gptext-stop' to stop gptext so that we could copy clean indexes from other machines
3) copy <segment_dir>/solr directory from corresponding machine which has same gpdb content id.
4) modify solr setting
4.1) modify solr.xml: set correct "enable.slave" and "enable.master" value according to segment role.
4.2) change lib/log4j.properties to use correct directory, mostly replace 'mirror' with 'primary' or replace 'primary' with 'mirror' in the file
4.3) change jetty.conf file
-) PORT: use correct port according to segment role. Look at gpdb-solr.properties file for port value
-) SOLR_HOME: point to correct directory according to segment role
5) double check the cluster, whether there is orphan solr processes by 'ps -ef|grep solr'. kill them if there is any
6) run 'gptext-start'
delete $GPTEXT-HOME/gptext-start.pid and $segdatadir/solr/jetty.pid if it report error.
All of these need to be taken care of by gptext-recover command. But there is a bug in that command and hence need to be done manually.