Pivotal Knowledge Base


BOSH Recreate Tasks Queueing and Failing with Snapshot Missing Errors


Pivotal Cloud Foundry all versions


BOSH recreate or resurrection tasks kept queuing up. BOSH tries to recreate the missing VM and fails with stemcell snapshot missing errors.

This error can be seen in the bosh task <#> --debug output:

E, [2018-02-02 11:00:09 #10383] [task:32342] ERROR -- DirectorJobRunner: Error resolving problem '3876':
Unknown CPI error 'Unknown' with message 'The object[s] '<[Vim.VirtualMachine] vm-7068>'
should have the following properties: ["snapshot"] , but they were missing these: #<Set: {"snapshot"}> .' in 'create_vm' CPI method


BOSH deployments have multiple stemcell VMs. In a vSphere environment, BOSH will create a vSphere snapshot of the stemcell VM with the name "initial". Then BOSH will use this snapshot to clone new VMs for use in a deployment.

If the snapshot on the stemcell is deleted or missing, BOSH won't be able to create any new VMs using the stemcell. This can cause a backlog in the BOSH tasks if it is trying to resurrect a VM because the task will take few minutes to fail or timeout while other tasks are already queuing up.


  • Check the number of BOSH tasks: bosh tasks --no-filter
  • Check for the processing task and debug it to confirm the error: bosh task <#> --debug
  • Check the IaaS console for the stemcell VM used here.
  • For VMWare, vCenter web client can be used. Open the PCF Templates Folder to find the stemcell VM.
  • We should see a snapshot named "initial" for the stemcell VM as seen below:Screen_Shot_2018-02-05_at_12.07.10_PM.png
  • If the above snapshot is not present, follow this article to download and fix the stemcell. However, as there are too many tasks queued, it may not be possible to run any more tasks.
  • Workaround

    In vSphere vCenter console, create the snapshot manually, naming the snapshot "initial".

  • Log in to vCenter and find the stemcell VM under PCF Templates Folder.
  • Right-click the stemcell VM and create a new snapshot.
  • Type the name as initial. See the screenshots below for details.
  • Screen_Shot_2018-02-15_at_10.52.37_AM.png


    Additional Information

    The above workaround is in the case where we have too many tasks in queued state. These tasks are expected to complete successfully and disappear quickly.

    To cancel all the queued tasks, use director_ctl command:

    If multiple datastores are being used then BOSH will copy the stemcell VM to each datastore. You made need to verify that the "initial" snapshot exists on stemcells in each datastore.


Powered by Zendesk