Pivotal Knowledge Base

Follow

VMs running out of disk space when Riak-CS for PCF® is installed

Environment

Product Version
Pivotal Cloud Foundry® (PCF) Riak-CS for PCF 1.5.13 or older

Symptom

If you have installed the Riak-CS for PCF Tile 1.5.13 or older, you may find an issue whereby you are running out of disk space on your VMs. This document explains what is using this space and how to fix/workaround the issue.

Cause

The access logs on Riak-CS nodes are not correctly being rotated.

Resolution

Please follow the steps below to check if the issue is being caused by your Riak-CS access logs folder increasing in size, and using all available disk space.

  1. Connect via SSH onto the affected VM
  2. sudo -i
  3. df -h
  4. du -h /var/vcap/data/sys/log/riak-cs/
  5. Check if the /var/vcap/data/sys/log/riak-cs/ is using up the majority of your disk space.
  6. ls -alh /var/vcap/data/sys/log/riak-cs/: if you are seeing access that the access logs are not getting purged then you are affected by this bug
  7. Remove access logs older than two days old by running the following:
    find /var/vcap/data/sys/log/riak-cs/ -name "access.log.*" -type f -mtime +2 -exec rm -fr {} \;

Impact/Risks

This is a low-risk exercise as these logs are just historical access logs and not required for the running of Riak-CS.

Additional Information

Command explained

Find all files in the /var/vcap/data/sys/log/riak-cs/ folder with a name beginning with access.log that are older than 2 days old(-mtime +2) and delete them (rm -fr).

Crontab:

You can add a command similar to the above to crontab and schedule it to run once a day to delete older log files as needed.

Please note the following:

  • This issue is resolved in Riak CS 1.5.14 and newer.
  • Pivotal does not support Riak CS in production. Please work with Basho or another S3 vendor of your choice. 

 

 

Comments

Powered by Zendesk