Pivotal Knowledge Base

Follow

A Bug Feeds the Executor_Cache Until the Disk is Full

Environment

 Product  Version
 Pivotal Cloud Foundry  1.7

Purpose

Executor_cache is a place where the cell rep caches build packs, droplets, and the lifecycle binaries. If any of these is missing, it should just download them again. There is a bug that under certain circumstances duplicates the directories, and the zip files are placed in the executor_cache. The size of the executor_cache can then possibly exceed the specified cache size till disk becomes full. As a consequence, you could experience staging failures when pushing new applications. Besides staging failure, you could also notice this problem by checking the Status page in Elastic Runtime.

Resolution

This bug is fixed in 1.7.38 which was released on Nov 24th, 2016. You can download and upgrade to Elastic Runtime 1.7.38 or higher to fix this issue.

Besides upgrading the Elastic Runtime, you can also restart the Diego cell to clean up the executor_cache as a workaround. For example,

bosh restart diego_cell-partition-75437f0a091c320d8ca4 0

Impacts / Risk

When restarting Diego cell, you will experience the app crash temporarily. By the following command, you can check which apps are running and are affected in a Diego cell. For more details you could refer to How to find what apps are running in a Diego cell

Please replace the IP_address_of_diego_cell with the real one on your side:

curl http://ip_address_of_diego_cell:1800/state|python -m json.tool |grep process_guid|cut -d ':' -f2|cut -c 3-38|xargs -I '<guid>' cf curl '/v2/apps/<guid>'|grep '"name"'|cut -d ':' -f2|sort -u

 

Comments

Powered by Zendesk