Pivotal Knowledge Base

Follow

Persistent Disk is not attached to Director VM

Environment

Product Version
Pivotal Cloud Foundry® (PCF) 1.6

Symptom

The Director VM in your deployment no longer has its persistent disk (mounted to /var/vcap/store) attached.

An error message similar to the following appears:

/home/tempest-web/tempest/web/vendor/bundle/ruby/2.2.0/gems/bosh_vsphere_cpi-2.2.0/lib/
cloud/vsphere/cloud.rb:348:in `block in detach_disk': Disk
'disk-a2971b65-8c2b-41f4-86e3-c0cbe6a98948' is not attached to VM
'vm-351eb059-ccaa-4ad0-88de-8dac566bc6d5' (Bosh::Clouds::DiskNotAttached) 

Cause

This error can happen for a few different reasons. The most common causes would be if a user, script or automation at the IaaS level has specifically detached the disk from the virtual machine (VM) or if BOSH stopped or failed during unexpectedly on a previous deployment.  Depending on the timing of that failure, this can leave bosh in a state where it thinks the disk is still attached and will thus fail on subsequent attempts to detach the disk. 

This is not IaaS specific and it can happen to any BOSH-deployed VM with a persistent disk, not just the Director.

Resolution

Check if you can find the missing disk in your IaaS environment. If you browse each of your datastores and search for that name, you should be able to locate the disk. It's a detached disk, so it's separate from the Director VM.

Note: It is very important that this disk still exist.

If it does exist, please perform the following steps:

  1. In your IaaS, power off old Director VM.
  2. SSH to your Ops Manager VM. Run cd /var/tempest/workspaces/default/deployments.
  3. Make a backup copy of bosh_deployments.yml and then edit it (the actual file, not the backup).
  4. Look for the line vm_cid. This should have a Globally Unique Identifier (GUID), which is the name of the current Director VM in your IaaS. Delete that whole line. Save and close the file.
  5. Go to the Ops Manager Web UI. Click "Apply Changes". This should create a new VM for the Director.
  6. Update /var/tempest/workspaces/default/deployments/bosh-deployments.yml file with new vm_cid reference.
  7. Once the install completes, firstly Remove old Director VM from disk and then remove the backup copy of bosh_deployments.yml.

If the steps do not resolve the problem, submit a ticket to PCF support at support.pivotal.io.

Comments

Powered by Zendesk