Pivotal Knowledge Base

Follow

Application crashes with 'cgroup out of memory'

Environment

Pivotal Cloud Foundry: All versions

Symptom

Application is crashing.

cf app <app-name> reports high memory utilization for application.

Diego cell reports kernel message: "cgroup out of memory."

Error Message:

/var/log/messages on container Diego cell:
<3>2017-04-26T19:12:36.397506+00:00 10.28.34.27 kernel [job=diego_cell index=2] [2437091.184092] Memory cgroup out of memory:
Kill process 20316 (java) score 1016 or sacrifice child

Cause 

The application has exhausted its allocated memory.

“Memory cgroup out of memory” is a message coming from the kernel when it kills something because of a memory cgroup restriction, such as those we place on containerized CF workload. If applications are running with high memory utilization then a memory spike without enough headroom could result in the kernel killing containerized process.

Resolution

The memory for the application needs to be increased or the application needs to be re-written to utilize less memory.

1. Check the current application memory usage

cf app <app-name>
state since CPU memory disk details
#0 running 2017-04-26 01:32:46 PM 0.0% 958.8M of 1G 252.3M of 2G

2. Increase the memory allocated to the application

cf scale <app-name> -m <higher memory level>

This will require an application restart

3. Afterwards, you can monitor memory utilization with the following command

cf app <app-name>
state since CPU memory disk details
#0 running 2017-04-26 01:32:46 PM 0.0% 1087.8M of 2G 252.3M of 2G

Comments

Powered by Zendesk