Pivotal Knowledge Base

Follow

Pivotal Cloud Foundry® Redis Deploy Fails to Complete

Environment

Pivotal Cloud Foundry® (PCF) Redis All Version

Symptom

On rare occasions, Redis deployment fails to complete and hangs during an OpsManager redeploy. 

Cause

On the 'cf-redis-broker' node, process-watcher doesn’t shut down despite reporting to the Monit, causing it to continue to operate while the Redis instances are being drained. It can be confirmed by checking the following:

  • Run 'monit summary' as root user and seeing that non-system jobs are not monitored.
  • Check that 'processmonitor' is still alive by running 'ps aux | grep processmonitor'
  • Check the drain logs with 'tail -f /var/vcap/sys/log/cf-redis-broker/drain.log' and see multiple attempts at shutting down Redis instances. 

Example

Wed May 11 17:15:07 2016 - Starting drain
Wed May 11 17:15:07 2016 - Disabling process watcher
Wed May 11 17:15:08 2016 - Waiting for redis-server shutdown
...repeats indefinitely...
 

Resolution

Run 'pkill processmonitor' on the 'cf-redis-broker' node and confirm that the process was killed with 'ps aux | grep processmonitor'  

Additional Information

This specific issue will be logged in '/var/vcap/sys/log/monit/process-watcher.err.log' for version 1.7.0 and later.

Comments

Powered by Zendesk