Pivotal Cloud Foundry® (PCF) Redis all versions
On rare occasions, Redis deployment fails to complete and hangs during an OpsManager redeploy.
On the "cf-redis-broker" node, process-watcher doesn’t shut down despite reporting to the Monit, causing it to continue to operate while the Redis instances are being drained. It can be confirmed by checking the following:
- Run "monit summary" as root user and seeing that non-system jobs are not monitored.
- Check that "processmonitor" is still alive by running "ps aux | grep processmonitor"
- Check the drain logs with "tail -f /var/vcap/sys/log/cf-redis-broker/drain.log" and see multiple attempts at shutting down Redis instances.
Wed May 11 17:15:07 2016 - Starting drain
Wed May 11 17:15:07 2016 - Disabling process watcher
Wed May 11 17:15:08 2016 - Waiting for redis-server shutdown
Run "pkill processmonitor" on the "cf-redis-broker" node and confirm that the process was killed with "ps aux | grep processmonitor/p".
This specific issue will be logged in "/var/vcap/sys/log/monit/process-watcher.err.log" for version 1.7.0 and above.