Pivotal Knowledge Base


"Failed to Perform Blobstore Operation after Three Retries" Cloud Controller Clock Configured with an Incorrect Cert Path


 Product  Version
 Pivotal Cloud Foundry  1.8 and above


While running cf copy-source APP1 APP2 users see the following error:

Failed to perform blobstore operation after three retries.

From cloud controller server /var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log

-cc22-4acd-572b-01b6467b1b22::1232a487-9a34-4565-a65d-ccb87da14477","error":"SSL verification failed","remaining_retries":0,"args":{"source_key":"65d57057-d
ng/lib/cloud_controller/blobstore/retryable_client.rb","lineno":131,"method":"rescue in with_retries"}

This is a known bug (see the Tracker Story) that is causing the failure of the background jobs that interact with WebDAV. The issue also results in blobs not being deleted. The fix is being worked on now and will be backported to 1.8, 1.9. Pivotal does not currently have a procedure available for reclaiming the lost space. If this error has caused you to lose a large amount of space, please contact Pivotal Support.


cloud_controller_worker config:

The file /var/vcap/jobs/cloud_controller_worker/config/cloud_controller_ng.yml referenced the following for certificates for blobstores:

ca_cert_path: /var/vcap/jobs/cloud_controller_ng/config/certs/buildpacks_ca_cert.pem

But this is the path for cloud controller, not the cloud controller worker. The correct path for the worker should be:

ca_cert_path: /var/vcap/jobs/cloud_controller_worker/config/certs/buildpacks_ca_cert.pem

This is causing some worker tasks to fail with cert errors when accessing blobstore endpoints.

The fix is available in PCF 1.8.33 and 1.9.6 onwards.



Powered by Zendesk