Pivotal Knowledge Base

Follow

Starting or staging an application results in an Insufficient Resources error

Environment

Product Version
Pivotal Cloud Foundry® (PCF) 1.7.x

Symptom

When starting or staging an application, the cf cli returns an error Insufficient Resources. The error message looks like the following:

2016-03-16T15:19:58.16+0100 [API/0] ERR Failed to stage application: insufficient resources

Cause

When getting this error, it means that you do not have enough free, unreserved space in the environment to either start or stage the application.  To start an application, you need to have at least the requested memory limit in free memory and the requested disk quota in free disk space.  To stage an application, you need to have at least 1G of memory or the requested memory limit free, whichever one is larger, and you need to have at least 6G of free disk space.  If you do not meet these requirements, starting or staging the app will fail with an insufficient resources error.

Resolution

The simplest resolution is to add more Diego Cells.  This will increase the available capacity on the platform and should clear up an insufficient resource errors.

If you are skeptical that the platform is actually low on resources, you can perform the following steps to investigate more.

  1. Check the total memory and disk metrics.  Make sure this is showing expected values based on the memory and ephemeral disk size of your Diego Cells.  Note some of the ephemeral disk space will be lost to swap.  See this KB for more details on Bosh deployed VMs and swap space.
  2. Check the available memory and disk metrics.  Make sure that at least one Diego Cell has enough memory and disk space to run or stage your application.  See the Cause section above for details on how much your application will need to run or stage.  Also confirm that the available container count has not hit zero.
  3. Check that your Cells are all reporting as healthy.  You can see this by looking at the metric opentsdb.nozzle.rep.UnhealthyCell.  When this returns 0, it means the cell is healthy.  When it returns 1, it means the cells is unhealthy.  Unhealthy cells will not take on new work and that will lower your capacity.

If you need additional assistance investigating why you're receiving insufficient resource errors, please open a support ticket.

Additional Information

You may notice Insufficient Resource errors when it appears like there is still unused memory and / or disk space on your Diego Cells (as reported by Bosh, Ops Manager or OS tools like free, top & ps).  This happens because Diego looks at unreserved memory and disk space, not what is actually being used.

For example, the following app consumes only 53.9MB memory and 100.5MB disk size, however it reserves 2GB memory and 1GB disk in Cell.

     state     since                    cpu    memory        disk           details   
#0   running   2016-06-24 02:03:31 PM   0.0%   53.9M of 2G   100.5M of 1G   

Diego would account for this consuming 2G of memory and 1G of disk, not just 53.9M of memory and 100.5M of disk space.

By default, Diego does not overcommit resources.  Starting in PCF 1.7, you can configure Diego to overcommit resources.  Instructions for doing that can be found here.

Comments

Powered by Zendesk