Pivotal Knowledge Base

Follow

CF Logging is unavailable and running "cf logs <app>" returns an empty result

Environment

Pivotal Cloud Foundry (PCF) 1.7 to 1.10

Symptom

Going to the "logs" tab in the Pivotal Apps Manager for an <app> or running "cf logs <app>" returns an empty result.

Cause 

There could be multiple reasons for this issue, one of them could be caused by network glitch causing ETCD for leader election from PCF 1.7 to 1.9. During this election and transition, the metron agents could be crashing, then hanging as they cannot connect to the leader. The logging issue could be the result of the chain of events.

Resolution

To work around this issue, try restarting the consul_agent first and then metron_agent on all CF VMs.

We can run a simple code like below that automates this process:

bosh vms cf-manifest | grep running | awk '{print $2}' | while read job; do echo '=========' 
$job; bosh ssh $job sudo /var/vcap/bosh/bin/monit restart consul_agent; sleep 20; done
bosh vms cf-manifest | grep running | awk '{print $2}' | while read job; do echo '=========' 
$job; bosh ssh $job sudo /var/vcap/bosh/bin/monit restart metron_agent; sleep 20; done

Comments

Powered by Zendesk