|Pivotal Cloud Foundry® (PCF) Redis||All|
On rare occasions, Redis deployment fails to complete and hangs during an OpsManager redeploy.
On the 'cf-redis-broker' node, process-watcher doesn’t shut down despite reporting to the monit that it has, causing it to continue to operate while the Redis instances are being drained. It can be confirmed by checking the following:
- Run 'monit summary' as root user and seeing that non-system jobs are not monitored.
- Check that 'processmonitor' is still alive by running 'ps aux | grep processmonitor'
- Check the drain logs with 'tail -f /var/vcap/sys/log/cf-redis-broker/drain.log' and see multiple attempts at shutting down Redis instances. Example below.
Wed May 11 17:15:07 2016 - Starting drain
Wed May 11 17:15:07 2016 - Disabling process watcher
Wed May 11 17:15:08 2016 - Waiting for redis-server shutdown
Run 'pkill processmonitor' on the 'cf-redis-broker' node and confirm that the process was killed with 'ps aux | grep processmonitor'
This specific issue will be logged in '/var/vcap/sys/log/monit/process-watcher.err.log' for version 1.7.0 and later.