Pivotal Knowledge Base

Follow

Pivotal Cloud Foundry® Redis Deploy Fails to Complete

Environment

 Product  Version
 Pivotal Cloud Foundry® (PCF) Redis  All 

Symptom

On rare occasions, Redis deployment fails to complete and hangs during an OpsManager redeploy. 

Cause

On the 'cf-redis-broker' node, process-watcher doesn’t shut down despite reporting to the monit that it has, causing it to continue to operate while the Redis instances are being drained. It can be confirmed by checking the following:

  • Run 'monit summary' as root user and seeing that non-system jobs are not monitored.
  • Check that 'processmonitor' is still alive by running 'ps aux | grep processmonitor'
  • Check the drain logs with 'tail -f /var/vcap/sys/log/cf-redis-broker/drain.log' and see multiple attempts at shutting down Redis instances. Example below.

Wed May 11 17:15:07 2016 - Starting drain
Wed May 11 17:15:07 2016 - Disabling process watcher
Wed May 11 17:15:08 2016 - Waiting for redis-server shutdown
...repeats indefinitely...
 

Resolution

Run 'pkill processmonitor' on the 'cf-redis-broker' node and confirm that the process was killed with 'ps aux | grep processmonitor'  

Additional Information

This specific issue will be logged in '/var/vcap/sys/log/monit/process-watcher.err.log' for version 1.7.0 and later.

Comments

Powered by Zendesk