Pivotal Cloud Foundry® (PCF) Version 1.7.x
When starting or staging an application, the cf cli returns an error Insufficient Resources. The error message looks like the following:
2016-03-16T15:19:58.16+0100 [API/0] ERR Failed to stage application: insufficient resources
When getting this error, it means that you do not have enough free, unreserved space in the environment to either start or stage the application. To start an application, you need to have at least the requested memory limit in free memory and the requested disk quota in free disk space. To stage an application, you need to have at least 1G of memory or the requested memory limit free, whichever one is larger, and you need to have at least 6G of free disk space. If you do not meet these requirements, starting or staging the app will fail with an insufficient resources error.
The simplest resolution is to add more Diego Cells. This will increase the available capacity on the platform and should clear up the insufficient resource errors.
If you are skeptical that the platform is actually low on resources, you can perform the following steps to investigate more.
- Check the total memory and disk metrics. Make sure this is showing expected values based on the memory and ephemeral disk size of your Diego Cells. Note some of the ephemeral disk space will be lost to swap. See this article for more details on Bosh deployed VMs and swap space.
- Check the available memory and disk metrics. Make sure that at least one Diego Cell has enough memory and disk space to run or stage your application. See the Cause section above for details on how much your application will need to run or stage. Also, confirm that the available container count has not hit zero.
- Check that your Cells are all reporting as healthy. You can see this by looking at the metric opentsdb.nozzle.rep.UnhealthyCell. When this returns 0, it means the cell is healthy. When it returns 1, it means the cells are unhealthy. Unhealthy cells will not take on new work and that will lower your capacity.
If you need additional assistance investigating why you're receiving insufficient resource errors, please open a support ticket.
You may notice Insufficient Resource errors when it appears like there is still unused memory and/or disk space on your Diego Cells (as reported by Bosh, Ops Manager or OS tools like free, top & ps). This happens because Diego looks at unreserved memory and disk space, not what is actually being used.
For example, the following app consumes only 53.9MB memory and 100.5MB disk size, however, it reserves 2GB memory and 1GB disk in Cell.
state since cpu memory disk details #0 running 2016-06-24 02:03:31 PM 0.0% 53.9M of 2G 100.5M of 1G
Diego would account for this consuming 2G of memory and 1G of the disk, not just 53.9M of memory and 100.5M of disk space.
By default, Diego does not overcommit resources. Starting in PCF 1.7, you can configure Diego to overcommit resources. Instructions for doing that can be found here.