Pivotal Knowledge Base

Follow

How to Verify Metrics are Reaching the Autoscaler

Environment

  • Pivotal Cloud Foundry (PCF) Autoscaler
  • Gorouter

Purpose

Identify the cause why the Autoscaler is not scaling when expected.

The first thing we need to ascertain is whether the relevant metrics (HttpStartStop, originating from gorouter) are reaching the firehose, where the Autoscaler collects them. In order to do that, we'd like to nozzle the firehose of the test environment for `HttpStartStop` and then siege the tested application.

Cause 

Metrics may not be passed to Autoscaler if loggregator system is undersized or metrics are being lost.

Procedure

Follow these steps to validate Autoscaler functionality:

  1. ssh to the affected system
  2. Install the cf-nozzle plugin located at: `https://github.com/cloudfoundry-attic/firehose-plugin`
  3. Run cf app-nozzle <app_name> -f HttpStartStop
  4. In a separate window, run `watch cf apps`:

    Screen_Shot_2017-09-20_at_10.55.20_AM.png
  5. Run the test: ab -n 3000 -c 10 <app URL>/entriesScreen_Shot_2017-09-20_at_10.57.26_AM.png
  6. Many messages scroll through your cf app-nozzle window (if metrics are being received)
  7. You should observe application scaling.

Additional Information

If metrics are not reaching Autoscaler, then it is recommended to review the loggregator system which passes metrics as log messages. Oftentimes missing metrics are indicative of the problem in the loggregator system. See the following articles:

Why Loggregator may lose logs: https://discuss.pivotal.io/hc/en-us/articles/223207207-Why-Loggregator-may-lose-logs

Scaling Loggregator: http://docs.pivotal.io/pivotalcf/1-12/loggregator/log-ops-guide.html#scaling

Firehose-to-syslog: https://github.com/cloudfoundry-community/firehose-to-syslog

Loggregator Message Throughput and Reliability: https://docs.pivotal.io/pivotalcf/1-12/loggregator/log-ops-guide.html#throughput-reliability

Scaling Nozzles: https://docs-pcf-staging.cfapps.io/pivotalcf/1-12/loggregator/log-ops-guide.html#scaling-nozzles

Calculate Loggregator Throughput: https://discuss.pivotal.io/hc/en-us/articles/225564028-How-to-Calculate-the-Loggregators-Message-Throughput

Installing cf cli: https://docs.run.pivotal.io/cf-cli/install-go-cli.html.

Comments

Powered by Zendesk