Pivotal Knowledge Base


How to Calculate the Loggregators Message Throughput


Pivotal Cloud Foundry® (PCF) 1.6.x and later


The purpose of this document is to help in calculating the total number of messages that the Loggregator produces over a given amount of time. This is useful when trying to scale your Loggregator environment to handle the number of messages you are producing.


Software requited:

  • cf cli [1] 
  • Cloud Foundry Firehose plugin

Note- The following commands should be run from the Ops Manager VM.

How to install the Cloud Foundry Firehose plugin

Installing the plugin is really simple.

1. Log on to the CF CLI with your admin credentials:

cf api <your_cf_api_url>

cf login

2. Add the plugin repository:

cf add-plugin-repo CF-Community http://plugins.cloudfoundry.org/

3. Install the plugin:

cf install-plugin "Firehose Plugin-r CF-Community 

4. Install Pipe Viewer (PV):

apt-get install pv 


Please run the following commands from the Ops Manager VM.

     1.    Connect via Single SHell (SSH) into the Ops Manager VM  

   2.    Log on as admin into the cf cli

   3.    cf nozzle -n | pv -l -i10 -r >/dev/null

[460.4 /s] 

i.e. Loggregator is processing 460.4 messages a second. The output will refresh every 10 seconds. 

Scaling Loggregator[2]

Rule of thumb

1. Dopplers

The number of Dopplers should be approximately the total number of messages per second divided by 2000.

(Dopplers =1 per msgs/sec /2,000)

2. Traffic Controllers

You should have one Traffic Controller for every four Dopplers.

(Traffic Controllers = 1 for every 4 Dopplers)

3. Nozzles 

For a Nozzle such as the firehose-to-syslog [3] Nozzle, you will require one every four Dopplers.

(Nozzles = 1 for every 4 Dopplers) 

For Scaling Nozzles in PCF 1.12 please click here [5]

4. Diego Cell

Doppler can handle about 16k / envelopes per second so you should avoid exceeding 8-10k per second into metron.

(Diego Cell = 1 per 1250 msgs/sec )

Log Message Size Constraints: When a Diego Cell emits app logs to Metron, Diego breaks up log messages greater than approximately 60 KiB into multiple envelopes.

Additional Information

[1] Installing cf cli: https://docs.run.pivotal.io/cf-cli/install-go-cli.html.

[2] Scaling Loggregator: http://docs.pivotal.io/pivotalcf/1-12/loggregator/log-ops-guide.html#scaling

[3] Firehose-to-syslog: https://github.com/cloudfoundry-community/firehose-to-syslog

[4] Loggregator Message Throughput and Reliability: https://docs.pivotal.io/pivotalcf/1-12/loggregator/log-ops-guide.html#throughput-reliability

[5] Scaling Nozzles: https://docs-pcf-staging.cfapps.io/pivotalcf/1-12/loggregator/log-ops-guide.html#scaling-nozzles


Powered by Zendesk