Pivotal Cloud Foundry® Redis 1.4.22, 1.4.23, 1.4.24, 1.5.11, 1.5.12, 1.5.13, 1.5.14
OpsManager upgrade or delete operations are hanging while updating the cf-redis-broker VM.
A possible indication that you have encountered this issue is on the redis-broker VM in the /var/vcap/sys/log/cf-redis-broker there should be a drain.log file.
This will show logging similar to the following:
Wed May 11 17:15:07 2016 - Starting drain Wed May 11 17:15:07 2016 - Disabling process watcher Wed May 11 17:15:08 2016 - Waiting for redis-server shutdown ...repeats indefinitely...
In the affected tiles, running a bosh recreate on the cf-redis-broker VM (or any other process that involves running the release drain script). The recreated redis-server processes may ignore SIGTERM signals while being otherwise responsive. This is due to a bug in Golang 1.5. As a result, further bosh recreate operations or OpsManager tile upgrades fail because the Redis processes cannot be taken offline gracefully.
- SSH onto the affected OpsManager VM
- bosh ssh onto the cf-redis-broker VM of the affected Redis deployment
- As root, run monit summary and ensure all processes are reporting not monitored
- As root, run
pkill -9 redis-server
to send a SIGKILL to all running Redis instances. This will cause all Redis server instances to stop running immediately.
- The stalled upgrade will now complete as normal
This issue is fixed in
- 1.4.25 and later
- 1.5.15 and later