Pivotal Knowledge Base

Follow

Pivotal Cloud Foundry® Redis Instance Alive after Successful De-provisioning

Environment

 Product  Version
 Pivotal Cloud Foundry® (PCF) Redis  All 

Symptom

A 'shared-vm' Redis instance that was de-provisioned is still accessible. When running a 'ps aux | grep redis-serve[r]' on the 'cf-redis-broker' VM, it would list an additional Redis instance that failed to de-provision despite performing the method successfully. When the user ran 'cf delete-service servicename'. 'cf services', it does not report the service instance. This issue is extremely uncommon.

Cause

The problem occurs when the process that performs the de-provision (DP) races with the Redis process watcher (RPW). RPW performs a periodic check to ensure Redis instances are healthy and resurrects them if it finds them down.

When a de-provision is requested, DP places a lock in the instance’s data directory to notify RPW that the instance is about to be deleted. It is possible that the DP starts immediately after the RPW starts. In the rare occasion that the RPW takes a while, it might end up picking up some of the effects of the de-provision. It will thus declare the instance unhealthy and resurrect it, canceling out the de-provision.

Resolution

  1. SSH onto the OpsManager VM
  2. 'bosh ssh' onto the cf-redis-broker VM
  3. Change to root by running 'sudo su'
  4. Run 'monit stop all'
  5. Change to the pidfile directory 'cd /var/vcap/sys/run/shared-instance-pidfiles' and delete any pidfiles located inside (may be empty)
  6. Change to the broker directory 'cd /var/vcap/store/cf-redis-broker/redis-data' and delete the directory that matches the GUID of the Redis instance that should have been de-provisioned. See below if unsure what the GUID is.
  7. Run 'monit start all'

Additional Information

Obtaining the GUID:

If you are not sure what the GUID of the instance being de-provisioned is, you will have to get a list of all the GUIDs related to your Redis service instances. Running 'cf services' will give you a list of all services. From here for each 'shared-vm redis' service, run 'cf the-service-name --guid' to obtain the GUID of the instance. Compare the list of GUIDs to those on the disks to see the GUID that does not have a matching service instance.

Comments

Powered by Zendesk