Pivotal Knowledge Base

Follow

Applications crash when GrootFS is enabled for PCF 2.0

Environment

Pivotal Cloud Foundry (PCF) 2.0.0 to 2.0.8

Symptom

When upgrading to PCF 2.0.x, applications crash when GrootFS is enabled. (this option turns on by default)

Docker applications are more susceptible to the GrootFS problems.

Error Message:

Below are log examples of GrootFS related issues:

application.log

2018-02-28T16:19:33.609-05:00 [APP/PROC/WEB/0] [ERR] /bin/sh: /app/bin/run.sh: not found
2018-02-28T16:19:33.617-05:00 [APP/PROC/WEB/0] [OUT] Exit status 127

application.log:

"error":"running image plugin create: making image: creating image: applying disk limits: disk limit is smaller than volume size\n: exit status 1",

Diego garden.log:

vcap.garden - - [instance@47450 director="" deployment="cf-c39abd91ffd4f2620223" group="diego_cell" az="HDCPOD1`" id="c083e8b2-957f-4821-85b3-8b0d794305ff"] {"timestamp":"1519847736.845199823","source":"guardian","message":"guardian.api.garden-server.create.failed","log_level":2,"data":{"error":"creating create command: no privileged_image_plugin provided","

Cause 

PCF 2.0 introduces GrootFS on Diego Cells. GrootFS is the Cloud Foundry component that provides filesystem isolation for containers and deals with container (Docker and OCI) images.

There are multiple GrootFS issues in early builds of 2.0:

  • no privileged_image_plugin provided: this is caused by a missing BOSH property and has since been fixed. For more information on this issue, see: https://discuss.pivotal.io/hc/en-us/articles/360001421094
  • applying disk limits: disk limit is smaller than volume size: this technically not a GrootFS bug, rather that GrootFS fixes a bug in which disk quotas were not being calculated properly. However, the bug fix means that Docker images that used to fit in to quota would no longer fit in to the same quota size (as the previous bug meant that we were being too lenient with quotas). 
  •  /bin/sh: <some file path>: not found: this is the result of a bug in GrootFS in which some files insider docker-based apps may go missing once the container's been created. We have fixed the bug and expect the fix to make it into the next version of garden-runc-release (v1.12.0). See tracker story: https://www.pivotaltracker.com/n/projects/1158420/stories/155391026

Resolution

Under Operations Manager, go to PAS and to Application Containers, uncheck option: Enable the GrootFS container image plugin for Garden RunC.

Click Apply Changes. This will disable GrootFS across your foundation.

A fix to this GrootFS issue is available in 2.0.9 and higher version of Pivotal Application Service (included in garden-runc-release v1.12.0).

Additional Information 

Reference: https://github.com/cloudfoundry/grootfs

Comments

Powered by Zendesk