Pivotal Knowledge Base

Follow

"Failed to Perform Blobstore Operation after Three Retries" Cloud Controller Clock Configured with an Incorrect Cert Path

Environment

 Product  Version
 Pivotal Cloud Foundry  1.8 and above

Overview

While running cf copy-source APP1 APP2 users see the following error:

FAILED
Failed to perform blobstore operation after three retries.

From cloud controller server /var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log

{"timestamp":1488480555.86369,"message":"cp_file_between_keys-retry","log_level":"debug","source":"cc.blobstore.dav_client","data":{"request_guid":"c1674444
-cc22-4acd-572b-01b6467b1b22::1232a487-9a34-4565-a65d-ccb87da14477","error":"SSL verification failed","remaining_retries":0,"args":{"source_key":"65d57057-d
1e6-4766-a97c-89e976180770","destination_key":"b966ad4b-18a1-4072-a3ba-fe99ddd7b09f"}},"thread_id":46914699153700,"fiber_id":46914742844680,"process_id":118
45,"file":"/var/vcap/data/packages/cloud_controller_ng/5f59122e1ae729a0ec63a4d511cf61f284df9e97.1-6e01d314dfb3752e6b32c8a467467889309c34b7/cloud_controller_
ng/lib/cloud_controller/blobstore/retryable_client.rb","lineno":131,"method":"rescue in with_retries"}

This is a known bug (see the Tracker Story) that is causing the failure of the background jobs that interact with WebDAV. The issue also results in blobs not being deleted. The fix is being worked on now and will be backported to 1.8, 1.9. Pivotal does not currently have a procedure available for reclaiming the lost space. If this error has caused you to lose a large amount of space, please contact Pivotal Support.

Description

cloud_controller_worker config:

The file /var/vcap/jobs/cloud_controller_worker/config/cloud_controller_ng.yml referenced the following for certificates for blobstores:

ca_cert_path: /var/vcap/jobs/cloud_controller_ng/config/certs/buildpacks_ca_cert.pem

But this is the path for cloud controller, not the cloud controller worker. The correct path for the worker should be:

ca_cert_path: /var/vcap/jobs/cloud_controller_worker/config/certs/buildpacks_ca_cert.pem

This is causing some worker tasks to fail with cert errors when accessing blobstore endpoints.

The fix is available in PCF 1.8.33 and 1.9.6 onwards.

 

Comments

Powered by Zendesk