Pivotal Knowledge Base

Follow

Autoscaler shows error "Unable to scale due to Cloud Controller error"

Environment

Autoscaler all versions

Symptom

In Apps Manager, in the Manage window for your Autoscaler instance, you see the following error in the Autoscaler events:

Unable to scale due to Cloud Controller error. 

Cause

This error can be caused for a number of reasons. Ultimately, this issue is caused when the autoscaler can not complete the autoscale request.

Resolution

There are multiple potential problems that could be causing this error, the first step is to determine the actual problem. To do that, grab the app’s GUID by running cf app <app-name> --guid then search for the  guid  in the autoscaler's logs by running  cf apps autoscale --recent | grep <app guid>. Once you have done that, look for any errors, failures, or incomplete requests containing that  guid. You can reference the examples below for what kind of errors you might see.

Examples where Autoscaler has received this error:

Example 1:

  • Autoscaler's Manage console in Apps Manager shows Unable to scale due to Cloud Controller error.
  • We targeted the Autoscale app with cf target -o system -s autoscaler .
  • We ran  cf apps autoscale --recent | grep <app guid>  and found the following error:
   2017-10-18T08:57:32.86-0700 [APP/PROC/WEB/0] OUT time="2017-10-18T15:57:32Z" 
level=error msg="Failed to scale application" app_guid=<guid>
body="{\n \"code\": 310003,\n \"description\": \"You have exceeded your space's memory limit.
\",\n \"error_code\": \"CF-SpaceQuotaMemoryLimitExceeded\"\n}\n" code=400 requested_instances=4
  • In this case, the bound app was in a space that had a Memory Quota of 10GB. When the autoscaler attempted to scale up the bound app, we hit the quota limit. There was already 10 app instances running in the space with 1GB of Memory allocated for each.
  • To get past a situation such as this, you can increase the space's Memory Quota, or you can run cf stop <app-name> on apps running in the space that don't need to be running in order to free some memory for another app's use.

Example 2:

  • Autoscaler's Manage console in Apps Manager shows Unable to scale due to Cloud Controller error.
  • We targeted the Autoscale app with cf target -o system -s autoscaler.
  • We ran cf apps autoscale --recent | grep <app guid> and found the following error:
2017-09-21T12:14:08.99-0400 [APP/PROC/WEB/0] OUT time="2017-09-21T16:14:08Z" 
level=warning msg="Failed to get application stats"
app_guid=8473ec5f-2c3b-43de-bfd7-b78ca26eb983 body="{\n \"description\":
\"Could not fetch stats for stopped app: GetLinkedAccounts\",\n \"error_code\":
\"CF-AppStoppedStatsError\",\n \"code\": 200003\n}\n" code=400
  • In this case, the bound app was in a non-running state. If the bound app is not running, crashed, or stopped, the Autoscaler can not complete the /stats call for it to determine if the app needs scaled up or down.
  • To get past a situation such as this, verify your app is not crashing occasionally via  cf events <app-name>. If your app has crashed you will see entries similar to the following in your  cf events output:

2017-09-14T01:56:56.00-0400 app.crash <app-name> index: 3, reason: CRASHED, exit_description: 2 error(s) occurred:
 

Additional Information

There are more reasons than this that you might receive the  Unable to scale due to Cloud Controller error  from Autoscaler. If you find another reason you have received this error, please contact Pivotal Support.

Comments

Powered by Zendesk