Pivotal Knowledge Base

Follow

Ops Manager Director ephemeral disk full: No space left on device

Environment

Pivotal Cloud Foundry all versions

Symptom

Ops Manager Director VM ephemeral disk (/var/vcap/data) becomes full and an error message such as the following can be seen in Ops Manager log files. 

[2017-05-26 06:52:22 #20887] [task:5425] ERROR -- DirectorJobRunner: 
Failed to extract release archive '/var/vcap/data/director/tmp/0000000221' into dir '/var/vcap/data/tmp/director/d20170526-20887-ge9jfr',
tar returned 2, output: tar: ./compiled_packages/golang1.7.tgz: Wrote only 8192 of 10240 bytes tar: ./compiled_packages/cc_uploader.tgz: Cannot write: No space left on device
Confirm the disk is full by ssh vcap@<director vm> (Use the VM credentials from the Ops Manager Director GUI credentials tab). 
Then run df -h /var/vcap/data and confirm its full.
 
Example:
/:~# df -h /var/vcap/data
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb2        46G  46G   0G   100% /var/vcap/data

Cause

The cause of this issue is that the ephemeral disk, mounted at /var/vcap/data/ on the Ops Manager Director VM has become full. We know this based on the error message "no space left on device" which we see when something attempts to write to a path beneath /var/vcap/data. While the reason for the disk being full can vary, it is commonly caused when there is higher than expected logging volume which fills up the disk. If you'd like to find the culprit, you can run

$ du -h -d 1 /var/vcap/data

to see which subdirectory is consuming the disk space.

Resolution

There are two general ways to resolve this issue: delete files to free up disk space or increase the disk space. See the sections below for each solution.

Delete Files

In general, we do not recommend deleting files from the Ops Manager Director VM, however, if you are running out of disk space because of log files, you can safely delete log files to free up space. Here are the steps to do that.

  1. SSH to the Ops Manager Director VM. You can get the IP from Ops Manager -> Director -> Status and you can get the VM credentials from Ops Manager -> Director -> Credentials, both in the Ops Manager Web UI.
  2. On the Director VM, run
    $ sudo su
    
    to get a root shell. Then run
    # cd /var/vcap/sys/log
  3. Before you delete log files, it is recommended that you scp the log files off of the Director VM to another system. This will allow further analysis of the logs to see why they are filling up the disk. In general, logs should not fill up the ephemeral disk so it's important to understand why so that we can prevent this from happening again in the future. If you need assistance with this task, please open a ticket with Pivotal Support.
  4. To continue deleting the logs, run
    # monit stop all
    This will instruct all of the processes on the Director VM to begin shutting down. Now run
    # watch monit summary
    and wait for all of the processes to stop.
  5. Run
    # rm -rf /var/vcap/sys/log/*
    This will delete all of the log files.
  6. Run
    # monit start all
    followed by
    # watch monit summary
    and wait for all of the processes to start.

At this point, all of the Director processes should be running again and you should have free disk space. You can verify this by running

# df -h /var/vcap/data

Increasing Disk Size

To increase the ephemeral disk for the Ops Manager Director, go to the Ops Manager Web UI and navigate to Ops Manager Director -> Settings -> Resource Config and change the VM type in Ops Manager Director row so that it has a larger ephemeral disk. Click the Save button. Navigate back to the Ops Manager Dashboard. Then click Apply Changes. This will redeploy the Ops Manager Director VM with a larger disk.

In general, we recommend that you pick the "Automatic" option for the VM Type, which is the default and allows Ops Manager to update the VM type to the current recommendation from Pivotal as you upgrade to newer versions of Ops Manager. 

Additional Information

In some cases, you might see Cannot write: No space left on device but the error message might be referring to /var/vcap/store.  This problem is slightly different as the disk that is becoming full is the persistent disk and not the ephemeral disk.  It is not safe to delete files from the persistent disk so to correct this problem you need to increase the size of the persistent disk.

To increase the size of the persistent disk for the Ops Manager Director, go to the Ops Manager Web UI and navigate to Ops Manager Director -> Settings -> Resource Config and change the persistent disk size.  This is independent of the VM type.  Click the Save button. Navigate back to the Ops Manager Dashboard. Then click Apply Changes. This will redeploy the Ops Manager Director VM with a larger disk.

Comments

Powered by Zendesk