Pivotal Cloud Foundry (PCF) versions 1.6 and above
This article explains the best practice for gathering Sys logs from Bosh virtual machine (VM) while troubleshooting.
A "bosh recreate" or "bosh cck" will destroy the affected VMs and then create a new VM. This process makes existing logs unrecoverable. It is important to extract the logs from a VM before recreating the VM. Especially important when conducting a root cause analysis.
A "bosh recreate" is executed automatically if the VM resurrection state is ON. This is the default configuration and is set during Elastic Runtime install. It can be changed on demand using Bosh VM resurrection [on/off].
To prevent losing the VM sys logs, consider using a Sys log dump to an external location.
Please see this article for more information on 3rd party sys log draining.
Review the following steps/guidelines before executing "bosh recreate" or "bosh cck" on any of your VMs:
- Are your logs being sent to an external Syslog service? If so, your logs should be safe and you can proceed to recreate.
- If logs are NOT being sent to an external Syslog service, please gather logs by following the instructions here.
- If logs are not able to be captured from the affected VMs, or if you are unable to do that work (perhaps due to time constraints), please note that an RCA will not be possible without logs and that by continuing with the recreated logs you are effectively declining an RCA.
- If you acknowledge this, proceed with the "bosh recreate" and mark this on the ticket. If not, circle back and work on capturing the logs.
bosh cck --report will provide a list of VMs that will be recreated. Review the list and capture the logs for those VMs. Execute "bosh cck" to proceed with the recreation.
How to obtain logs from PCF outlines the steps in capturing logs from the VMs.