Pivotal Knowledge Base

Follow

BOSH VM resurrector remains disabled after PCF 1.7 to PCF 1.8 foundation upgrade

Environment

Pivotal Cloud Foundry®: 1.8

Purpose

This article discusses how to resolve an issue where all instance VMs resurrection values are set to paused after a 1.8 upgrade (Before the 1.8 upgrade, bosh vm resurrection off (or disable) was executed resulting in all instance VM's set to paused). For resurrector to work successfully, the resurrection value needs to set to active. 

PCF 1.6 and 1.7

With PCF 1.6 and 1.7 one could run a command from BOSH CLI (Opsman VM) as follows:
$ bosh vm resurrection off (or disable) which would result in the following:

  1. BOSH Resurrector would be disabled for the entire foundation (All deployments for the given Director)
  2. Output to 'bosh vms --details' would show the Resurrector as "paused"
+--------------------------------------+--------------+--------+
| Agent ID                             | Resurrection | Ignore |
+--------------------------------------+--------------+--------+
| 04f8657c-f3a2-4fae-9b03-ab4dda5fa1a0 | paused       | false  |

Conversely, 

$ bosh vm resurrection on (or enable) which would result in the following:

  1. BOSH Resurrector would be enabled for the entire foundation (All deployments for the given Director)
  2. Output to 'bosh vms --details' would show the Resurrector as "active"
+--------------------------------------+--------------+--------+
| Agent ID                             | Resurrection | Ignore |
+--------------------------------------+--------------+--------+
| 04f8657c-f3a2-4fae-9b03-ab4dda5fa1a0 | active       | false  |
Furthermore, after enabling or disabling the resurrector from the Ops Manager WebUI, after apply changes, we could see 'bosh vms --details' output would reflect whether the VMs/jobs were "active" or "paused"

PCF 1.8

However, since PCF 1.8 the behaviour has changed, all VMs/jobs will show Resurrection on an "instance level" only, no matter if a global change was applied or not. In 1.8, The director sets the resurrector option globally for a deployment and also on a per job instance basis. If the resurrector is disabled globally, it can still show up as being enabled for each job instance.

By default global resurrection status set to enabled. We currently don't show the current global status; this functionality is expected to exist in PCF 1.12. To disable resurrector run bosh vm resurrection off and to enable resurrector run bosh vm resurrection on.

Instance level can overwrite global value. 

Procedure

Write a script to iterate through each instance vm and run the following command on every VM instance bosh vm resurrection [<job>] [<index>] <new_state>

Example:

bosh vm resurrection uaa 2 true 

Additional Information

As mentioned above, by default global resurrection status set to enabled. We currently don't show the current global status; this functionality is expected to exist in PCF 1.12. 

 

Comments

Powered by Zendesk