Pivotal Knowledge Base

Follow

Pivotal Cloud Foundry® Redis tile upgrade issue

Environment

Product Version
Pivotal Cloud Foundry® Redis

1.4.22, 1.4.23, 1.4.24

1.5.11, 1.5.12, 1.5.13, 1.5.14

Symptom

OpsManager upgrade or delete operations are hanging while updating the cf-redis-broker VM.

A possible indication that you have encountered this issue is on the redis-broker VM in the /var/vcap/sys/log/cf-redis-broker there should be a drain.log file.
This will show logging similar to the following:

Wed May 11 17:15:07 2016 - Starting drain
Wed May 11 17:15:07 2016 - Disabling process watcher
Wed May 11 17:15:08 2016 - Waiting for redis-server shutdown
...repeats indefinitely...

Cause

In the affected tiles, running a bosh recreate on the cf-redis-broker VM (or any other process that involves running the release drain script). The recreated redis-server processes may ignore SIGTERM signals while being otherwise responsive. This is due to a bug in Golang 1.5. As a result, further bosh recreate operations or OpsManager tile upgrades fail because the Redis processes cannot be taken offline gracefully.

Resolution

  • SSH onto the affected OpsManager VM
  • bosh ssh onto the cf-redis-broker VM of the affected Redis deployment
  • As root, run monit summary and ensure all processes are reporting not monitored
  • As root, run

            pkill -9 redis-server

        to send a SIGKILL to all running Redis instances. This will cause all Redis      server instances to stop running immediately.

  • The stalled upgrade will now complete as normal

This issue is fixed in

  • 1.4.25 and later
  • 1.5.15 and later

Comments

Powered by Zendesk