Pivotal Knowledge Base

Follow

How to verify metrics are reaching autoscaler

Environment

PCF Autoscaler

Gorouter

Purpose

Identify the cause why the autoscaler not scaling when expected.

The first thing we need to ascertain is whether the relevant metrics (HttpStartStop, originating from gorouter) are reaching the firehose, where the Autoscaler collects them. In order to do that, we'd like to nozzle the firehose of the test environment for `HttpStartStop` and then siege the tested application.

Cause 

Metrics may not be passed to autoscaler if loggregator system is undersized or metrics are being lost.

Procedure

Follow these steps to validate autoscaler functionality:

  1. `ssh` to the affected system

  2. Install the cf-nozzle plugin located at: `https://github.com/cloudfoundry-attic/firehose-plugin`

  3. Run `cf app-nozzle <app_name> -f HttpStartStop`

  4. In a separate window, run `watch cf apps`:

    Screen_Shot_2017-09-20_at_10.55.20_AM.png
  5. Run the test: ab -n 3000 -c 10 <app URL>/entries

    Screen_Shot_2017-09-20_at_10.57.26_AM.png
  6. Many messages scroll through your cf app-nozzle window (if metrics are being received)

  7. You should observe application scaling.

Additional Information

If metrics are not reaching autoscaler, then it is recommended to review the loggregator system which passes metrics as log messages. Oftentimes missing metrics are indicative of the problem in the loggregator system. See the following articles:

[1] Why Loggregator may lose logs: https://discuss.pivotal.io/hc/en-us/articles/223207207-Why-Loggregator-may-lose-logs

[2] Scaling Loggregator: http://docs.pivotal.io/pivotalcf/1-12/loggregator/log-ops-guide.html#scaling

[3] Firehose-to-syslog: https://github.com/cloudfoundry-community/firehose-to-syslog

[4] Loggregator Message Throughput and Reliability: https://docs.pivotal.io/pivotalcf/1-12/loggregator/log-ops-guide.html#throughput-reliability

[5] Scaling Nozzles: https://docs-pcf-staging.cfapps.io/pivotalcf/1-12/loggregator/log-ops-guide.html#scaling-nozzles

[6] Calculate Loggregator Throughput: https://discuss.pivotal.io/hc/en-us/articles/225564028-How-to-Calculate-the-Loggregators-Message-Throughput

[7] Installing cf cli: https://docs.run.pivotal.io/cf-cli/install-go-cli.html.

Comments

Powered by Zendesk