Pivotal Knowledge Base

Follow

Best Practices for Collecting Logs in Pivotal Cloud Foundry when the VMs are Down

Environment

 Product  Version
 Pivotal Cloud Foundry (PCF)  1.6+ and above

Purpose

This article explains the best practice for gathering Sys logs from Bosh VM while troubleshooting.

Cause

bosh recreate or bosh cck will destroy the affected VMs and then create a new VM. This process makes existing logs unrecoverable. It is important to extract the logs from a VM before recreating the VM. Especially important when conducting a root cause analysis.

A bosh recreate is executed automatically if the VM resurrection state is ON. This is the default configuration and is set during Elastic Runtime install. It can be changed on demand using Bosh VM resurrection [on/off].

To prevent losing the VM sys logs, consider using a Sys log dump to an external location.

Please see this article for more information on 3rd party sys log draining. 

Procedure

Review the following steps/guidelines before executing bosh recreate or bosh cck on any of your VMs:

  1. Are your logs being sent to an external Syslog service? If so, your logs should be safe and you can proceed to recreate.
  2. If logs are NOT being sent to an external Syslog service, please gather logs by following the instructions here.
  3. If logs are not able to be captured from the affected VMs, or if you are unable to do that work (perhaps due to time constraints), please note that an RCA will not be possible without logs and that by continuing with the recreated logs you are effectively declining an RCA.
  4. If you acknowledge this, proceed with the bosh recreate and mark this on the ticket. If not, circle back and work on capturing the logs. 

bosh cck --report will provide a list of VMs that will be recreated. Review the list and capture the logs for those VMs. Execute bosh cck to proceed with the recreation. 

Additional Information

How to obtain logs from PCF outlines the steps in capturing logs from the VMs.

 

 

 

Comments

Powered by Zendesk