Pivotal Cloud Foundry® (PCF) Redis All Version
On rare occasions, Redis deployment fails to complete and hangs during an OpsManager redeploy.
On the 'cf-redis-broker' node, process-watcher doesn’t shut down despite reporting to the Monit, causing it to continue to operate while the Redis instances are being drained. It can be confirmed by checking the following:
- Run 'monit summary' as root user and seeing that non-system jobs are not monitored.
- Check that 'processmonitor' is still alive by running 'ps aux | grep processmonitor'
- Check the drain logs with 'tail -f /var/vcap/sys/log/cf-redis-broker/drain.log' and see multiple attempts at shutting down Redis instances.
Wed May 11 17:15:07 2016 - Starting drain
Wed May 11 17:15:07 2016 - Disabling process watcher
Wed May 11 17:15:08 2016 - Waiting for redis-server shutdown
Run 'pkill processmonitor' on the 'cf-redis-broker' node and confirm that the process was killed with 'ps aux | grep processmonitor'
This specific issue will be logged in '/var/vcap/sys/log/monit/process-watcher.err.log' for version 1.7.0 and later.