Pivotal Knowledge Base

Follow

No Application Logs or Metrics Available from Apps Manager or CLI

Environment

 Product  Version
 Pivotal Cloud Foundry® Elastic  Runtime

 All

Symptom

No logs or metrics can be seen from apps manager or the CF CLI for applications.

Cause

The reason for this issue could be that ETCD experienced a networking issue that affected the Dopplers and caused the metron agent to lose track of the available Dopplers. Another known issue is that the Dopplers are unable to refresh their TTL (Time To Live) on their endpoints in ETCD. 

Looking at the logs for metron agent on a Diego cell. You may see the following error:

panic: No udp enabled dopplers available, check your manifest to make sure you have dopplers listening for udp

or

{"timestamp":1476530710.878901958,"process_id":8089,"source":"metron",
"log_level":"error","message":"DopplerForwarder: failed to pick a client",
"data":{"error":"loggregator client pool is empty"},
"file":"/var/vcap/data/compile/metron_agent/loggregator/src/metron/writers/dopplerforwarder/doppler_forwarder.go",
"line":38,"method":"metron/writers/dopplerforwarder.(*DopplerForwarder).Write"}

Resolution

Areas to investigate are:

  • Check that ETCD is not experiencing a split brain (see [1] section "Failed Deploys, Upgrades, Split-Brain Scenarios, etc." to recover from this).
  • Check info in ETCD. It’s possible that the data available about the Dopplers is incorrect [2]. 
  • Check ETCD nodes for any clock drift i.e. does the time on each node match.
  • Ensure that Dopplers and Traffic Controllers are working as expected by reviewing logs and monit processes.
  • If the VM from the metron is reporting the panic: "No UDP enabled Dopplers available," a restart of the process from the monit should fix the logging and the metric collection.  

[1] https://github.com/cloudfoundry-incubator/etcd-release

[2] https://github.com/cloudfoundry/loggregator/wiki/FAQ#q-how-can-i-check-the-health-of-my-etcd-cluster

 

 

 

 

Comments

Powered by Zendesk