Pivotal Knowledge Base

Follow

Insufficient memory when using Java Buildpack 4.0+

Environment

  • Pivotal Cloud Foundry® (PCF) 1.9.x, 1.10.x, 1.11.x
  • Java Buildpack 4.0+

Symptom

After upgrading to java buildpack 4.0+, applications that previously ran with smaller memory limit are now getting the error "insufficient memory".

Error Message: 

2017-08-11T18:26:19.30+0000 [APP/PROC/WEB/0]ERR Cannot calculate JVM memory configuration: 
There is insufficient memory remaining for heap.
Memory limit 512M is less than allocated memory 673748K
(-XX:ReservedCodeCacheSize=240M, -XX:MaxDirectMemorySize=10M, -XX:MaxMetaspaceSize=93144K,
-XX:CompressedClassSpaceSize=17404K, -Xss1M * 300 threads)

Cause 

Java buildpack 4.0 has an enhancement to the JVM memory calculation and the new memory calculator now accounts for several different memory regions.

One major result of accounting for all of these memory regions is the likelihood that applications which previously ran in small containers (512M or less) will be unable to run. 

Resolution

The simplest and recommended solution to this problem is to increase the available memory to your application.

This can be done via "cf scale" command:

cf scale APP_NAME [-i INSTANCES] [-k DISK] [-m MEMORY] [-f]

Example:

cf scale my-application -m 1G

Additional Information

Another option that can help is to adjust the JVM memory settings. The Java Buildpack currently uses a 1MB thread stack size to provide an account for thread stack usage. In most cases, you can decrease that to 160k or 256k for apps that have more deep stacks.

The net result will be the reduction of thread stack size by about one quarter the default. Since the Java buildpack defaults to 300 threads, this can make a noticeable difference. As an example, the difference in the default thread count is 300M of thread stack versus 75M if you use a thread stack of 256k.

cf set-env my-application JAVA_OPTS: ‘-Xss349k’

cf restage my-application

It is only recommended to decrease this setting if you have a very clear understanding about the thread usage in your app under load. If you incorrectly reduce this value, your application may crash with StackOverflowExceptions or because it exceeds the memory limit assigned to your application.

If you have continued StackOverflowExceptions and app crashes after lowering thread stack, then increase the value for this setting by 256k until the application is stable.

This issue is referenced in the blog: https://www.cloudfoundry.org/just-released-java-buildpack-4-0/

This is GitHub issue for this problem: https://github.com/cloudfoundry/java-buildpack/issues/392

Comments

Powered by Zendesk