Pivotal Knowledge Base


"Failed to Perform Blobstore Operation After Three Retries" Cloud Controller Clock Configured with an Incorrect Cert Path


Pivotal Cloud Foundry versions 1.8 and above


While running "cf copy-source APP1 APP2", users see the following error:

Failed to perform blobstore operation after three retries.

From cloud controller server /var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log

-cc22-4acd-572b-01b6467b1b22::1232a487-9a34-4565-a65d-ccb87da14477","error":"SSL verification failed","remaining_retries":0,"args":{"source_key":"65d57057-d
ng/lib/cloud_controller/blobstore/retryable_client.rb","lineno":131,"method":"rescue in with_retries"}

This is a known issue that is causing the failure of the background jobs that interact with WebDAV. The issue also results in blobs not being deleted. The fix is being worked on now and will be backported to 1.8, 1.9. Pivotal does not currently have a procedure available for reclaiming the lost space. If this error has caused you to lose a large amount of space, please contact Pivotal Support.


cloud_controller_worker config:

The file /var/vcap/jobs/cloud_controller_worker/config/cloud_controller_ng.yml referenced the following for certificates for the blobstores:

ca_cert_path: /var/vcap/jobs/cloud_controller_ng/config/certs/buildpacks_ca_cert.pem

But this is the path for cloud controller, not the cloud controller worker. The correct path for the worker should be:

ca_cert_path: /var/vcap/jobs/cloud_controller_worker/config/certs/buildpacks_ca_cert.pem

This is causing some worker tasks to fail with cert errors when accessing blobstore endpoints.

The fix is available in PCF version 1.8.33 and version 1.9.6 onwards.


Powered by Zendesk