Pivotal Knowledge Base

Follow

Cloud Controller cannot access the Elastic Runtime Blobstore: Could not create object 403

Environment 

Pivotal Cloud Foundry® (PCF)  1.9.x, 1.10.x, 1.11.x
Iaas  vSphere
Component  ElasticRuntime

Symptom

This article explains how to troubleshoot 'blobstore could not create object' error seen in the Cloud Controller vm logs.

When deploying an Elastic Runtime job, the deploy fails and returns the following error: 

E, [2017-09-27 13:47:48 #756] [canary_update(cloud_controller/2ccd7586-4718-46e1-9efa-2885624ff620 (0))] 
ERROR -- DirectorJobRunner: Error updating canary instance
: # /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.0.0/lib/bosh/director/agent_client.rb:247:in `handle_method' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.0.0/lib/bosh/director/agent_client.rb:302:in `handle_message_with_retry' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.0.0/lib/bosh/director/agent_client.rb:55:in `method_missing' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.0.0/lib/bosh/director/agent_client.rb:357:in `get_task_status' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.0.0/lib/bosh/director/agent_client.rb:166:in `wait_for_task' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.0.0/lib/bosh/director/agent_client.rb:321:in `send_message' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.0.0/lib/bosh/director/agent_client.rb:132:in `run_script'/pre>

In the /var/vcap/sys/log/cloud_controller_ng logs of the cloud_controller vm it reports the following error:

[2017-09-27 13:47:45+0000] + install_buildpacks 
[2017-09-27 13:47:45+0000] + pushd /var/vcap/packages/cloud_controller_ng/cloud_controller_ng 
[2017-09-27 13:47:45+0000] + chpst -u vcap:vcap bundle exec rake buildpacks:install 
[2017-09-27 13:47:47+0000] rake aborted! 
[2017-09-27 13:47:47+0000] CloudController::Blobstore::BlobstoreError: Could not create object, 403/ 
[2017-09-27 13:47:47+0000] 403 Forbidden
[2017-09-27 13:47:47+0000] nginx

Cause

Some private networks require extra configuration so that internal file storage (WebDAV) can communicate with other PCF processes.

Resolution

  1. Go to the Step 17: (Optional) Enable Advanced Features - https://docs.pivotal.io/pivotalcf/1-9/customizing/config-er-vmware.html#er-advanced
  2. Go to Section 'Whitelist for Non-RFC-1918 Private Networks'
  3. Add a new allow rule to the existing contents of the Whitelist for non-RFC-1918 Private Networks field for your private network.
  4. Save 
  5. Apply Changes

Comments

Powered by Zendesk