Pivotal Knowledge Base

Follow

Pivotal Cloud Foundry® Elastic Runtime upgrade fails due to UAA issue

Environment

Product Version
Pivotal Cloud Foundry® Elastic Runtime

1.7.x

Symptom

During the upgrade of Elastic Runtime, it may happen that UAA will fail to upgrade due to a transient issue.  Looking at the Catalina.log you will see an error similar to the following:

SEVERE: Exception unloading sessions to persistent storage
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1823)
at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:719)
at org.apache.catalina.session.StandardManager.doUnload(StandardManager.java:445)

Another error you may find in the logs is:

log4j:ERROR Failed to flush writer,
[2016-06-30 11:32:29+0000] java.io.IOException: No space left on device

BOSH Agent logs will show something similar to the following:

2016-05-30_10:22:30.53877 ./jdk/lib/amd64/jli/
2016-05-30_10:22:30.53877 ./jdk/lib/amd64/jli/libjli.so
2016-05-30_10:22:30.53878 ./jdk/lib/amd64/libjawt.so
2016-05-30_10:22:30.53878 ./jdk/ASSEMBLY_EXCEPTION
2016-05-30_10:22:30.53878 [Cmd Runner] 2016/05/30 10:22:30 DEBUG - Stderr: tar: ./jdk/jre/lib/amd64/libawt.so: Wrote only 9216 of 10240 bytes
2016-05-30_10:22:30.53878 tar: ./jdk/jre/lib/amd64/libsunec.so: Cannot write: No space left on device
2016-05-30_10:22:30.53879 tar: ./jdk/jre/lib/amd64/libsctp.so: Cannot write: No space left on device
2016-05-30_10:22:30.53879 tar: ./jdk/jre/lib/amd64/libsaproc.so: Cannot write: No space left on device 

Cause

The reason for this issue is not enough disk space on the VM for BOSH to install the required packages.  

Resolution

  • A retry of the upgrade should allow the install to complete successfully
  • Another option is to increase the disk space allocated to the failed VM

Comments

Powered by Zendesk