Pivotal Cloud Foundry® (PCF) Elastic Runtime all versions
This article discusses how the Loggregator works in Pivotal Cloud Foundry.
Loggregator simply transports log messages from the stdout/stderr of an app to the endpoint of Cloud Foundry logging, namely the firehose. Persistence of the logs is the responsibility of whatever consumes the logs. For example aggregators such as ELK stacks, Splunk or simply cf logs.
Log messages not immediately extracted and persisted, are discarded. The exception is the small number of logs stored in a buffer and are available through cf logs --recent.
The Loggregator Components
Loggregator transports log messages via a chain of components from applications to either the firehose, or a Doppler syslog drain. Various components of the chain are scaled horizontally as necessary for load. These components fulfill the following functions:
Provides the entry point to the Loggregator chain on each VM that has either logs or metrics to send through Loggregator. It accepts messages from clients, groups them into full buffers for efficient transport, and load balances by randomly selecting a different Doppler for each buffer transmission.
The doppler is the primary transport mechanism. It provides load balancing store and forwarding of log messages. it supports outputting to either:
- A Syslog drain on a per-app basis from all Dopplers in parallel via a custom provisioned service binding or
- An aggregating Traffic Controller
It provides a small amount of per-app buffering, typically 100 messages, to support cf logs --recent
“Re-aggregates” the parallel log streams transported by the Dopplers. it provides endpoints for clients (Nozzles or the CLI) to request output of either
- All logs and metrics - this is the Firehose.
- All logs and metrics for a single app - sometimes called the Garden hose
On request, the Traffic Controller opens parallel web sockets to all Dopplers requesting either the logs of a single app or the complete log stream. For each request, the Traffic Controller maintains the separate aggregated streams until the client releases the connection.
Provide the mechanism for transporting log and metric data from the Firehose to the downstream aggregating system of choice (such as ELK, Splunk, Datadog, etc.). They read in messages from the Firehose in the custom Loggregator dropsonde protocol, and convert messages into the appropriate downstream format, such as syslog. They can also perform other functions such as hydrating the log message with App ID, based on the included app guid. There are several solutions in the Cloud Foundry OSS community for targets such as Syslog, Graphite, and Kafka.
Loggregator interacts with other components to support the above chain:
Please note that the Loggregator can lose messages depending on the number of messages it handles.
Please see this article for more information.
"Log messages" is used here as a proxy term for all messages transported by Loggregator, including metrics.