Pivotal Knowledge Base


Applications crash when GrootFS is enabled for PCF 2.0


Pivotal Cloud Foundry (PCF) 2.0.0 to 2.0.8


When upgrading to PCF 2.0.x, applications crash when GrootFS is enabled. (this option turns on by default)

Docker applications are more susceptible to the GrootFS problems.

Error Message:

Below are log examples of GrootFS related issues:


2018-02-28T16:19:33.609-05:00 [APP/PROC/WEB/0] [ERR] /bin/sh: /app/bin/run.sh: not found
2018-02-28T16:19:33.617-05:00 [APP/PROC/WEB/0] [OUT] Exit status 127


"error":"running image plugin create: making image: creating image: applying disk limits: disk limit is smaller than volume size\n: exit status 1",

Diego garden.log:

vcap.garden - - [instance@47450 director="" deployment="cf-c39abd91ffd4f2620223" group="diego_cell" az="HDCPOD1`" id="c083e8b2-957f-4821-85b3-8b0d794305ff"] {"timestamp":"1519847736.845199823","source":"guardian","message":"guardian.api.garden-server.create.failed","log_level":2,"data":{"error":"creating create command: no privileged_image_plugin provided","


PCF 2.0 introduces GrootFS on Diego Cells. GrootFS is the Cloud Foundry component that provides filesystem isolation for containers and deals with container (Docker and OCI) images.

There are multiple GrootFS issues in early builds of 2.0:

  • no privileged_image_plugin provided: this is caused by a missing BOSH property and has since been fixed. For more information on this issue, see: https://discuss.pivotal.io/hc/en-us/articles/360001421094
  • applying disk limits: disk limit is smaller than volume size: this technically not a GrootFS bug, rather that GrootFS fixes a bug in which disk quotas were not being calculated properly. However, the bug fix means that Docker images that used to fit in to quota would no longer fit in to the same quota size (as the previous bug meant that we were being too lenient with quotas). 
  •  /bin/sh: <some file path>: not found: this is the result of a bug in GrootFS in which some files insider docker-based apps may go missing once the container's been created. We have fixed the bug and expect the fix to make it into the next version of garden-runc-release (v1.12.0). See tracker story: https://www.pivotaltracker.com/n/projects/1158420/stories/155391026


Under Operations Manager, go to PAS and to Application Containers, uncheck option: Enable the GrootFS container image plugin for Garden RunC.

Click Apply Changes. This will disable GrootFS across your foundation.

A fix to this GrootFS issue is available in 2.0.9 and higher version of Pivotal Application Service (included in garden-runc-release v1.12.0).

Additional Information 

Reference: https://github.com/cloudfoundry/grootfs


Powered by Zendesk