Pivotal Knowledge Base

Follow

Ops Manager S3 blobstore configuration error: 'Unable to verify certificate'

Environment

Pivotal Cloud Foundry: 1.8, 1.9, 1.10

Symptom

When attempting to add S3 Compatible Blobstore to Ops Manager -> Director Tile -> Director Config, apply changes fail with "Unable to verify certificate."

Error Message:

Errors::CertificateError SSL_connect returned=1 errno=0 state=SSLv3 read
server certificate B: certificate verify failed (OpenSSL::SSL::SSLError) Unable to verify certificate 

Cause 

There are two issues that you will encounter when trying to use an S3 compatible blobstore that has a custom cert.  The first is that Ops Manager will not be able to successfully validate the connection to your blobstore.  This presents as the error message listed in the symptoms section above.  This can be worked around by manually trusting the custom certificate or CA on the Ops Manager VM.  However, you cannot workaround the second issue which is that the custom certificate or CA will not be trusted on the Bosh Director.  In this case, when you click Apply Changes the installation will start and fail halfway through complaining that it cannot upload items to the blobstore because the certificate is not trusted. 

Custom certs for an S3 compatible blobstore are not currently supported on the Bosh Director.

Resolution

Firstly, confirm if you are using a self-signed or a public CA certificate when configuring S3.

If using Self-Signed CA

There is no way to install a custom CA cert on the Bosh director up until the 1.12 version. For now, the options are switch to use the internal blobstore with the Bosh Director or install a trusted certificate on your S3 compatible blobstore.

If using a Public CA Signed certificate then it should just work.  If you are seeing an issue, you can use the following instructions to confirm that your certificate is trusted.

1. Only server certificates signed by a publicly signed Certificate Authority whose public cert exist in the /etc/ssl/certs/ on the Ubuntu stemcell will work.

2. Check this directory for the listings of the server certificates of known Certificate Authorities.  For the certificate to be trusted, your CA will need to be listed here.

ubuntu@<opsmanager>:# ls /etc/ssl/certs 
...
Verisign_Class_3_Public_Primary_Certification_Authority_-_G3.pem
VeriSign_Class_3_Public_Primary_Certification_Authority_-_G4.pem
VeriSign_Class_3_Public_Primary_Certification_Authority_-_G5.pem
Verisign_Class_3_Public_Primary_Certification_Authority.pem
VeriSign_Universal_Root_Certification_Authority.pem
Visa_eCommerce_Root.pem
WellsSecure_Public_Root_Certificate_Authority.pem
WoSign_China.pem
WoSign.pem
XRamp_Global_CA_Root.pem

3. Verify that the certificate presented by the S3 compatible blobstore configured under the Ops Manager Director Tile is signed by one of the well known Certificate Authorities listedin the /etc/ssl/certs directory.

Additional Information 

Full error stacktrace:

Excon::Errors::CertificateError SSL_connect returned=1 errno=0 state=SSLv3 read server
certificate B: certificate verify failed (OpenSSL::SSL::SSLError) Unable to verify certificate.
This may be an issue with the remote host or with Excon.Excon has certificates bundled, but these
can be customized.`Excon.defaults[:ssl_ca_path] = path_to_certs`, `ENV['SSL_CERT_DIR'] =
path_to_certs`, `Excon.defaults[:ssl_ca_file] = path_to_file`, `ENV['SSL_CERT_FILE'] =
path_to_file`, `Excon.defaults[:ssl_verify_callback] = callback` (see
OpenSSL::SSL::SSLContext#verify_callback), or `Excon.defaults[:ssl_verify_peer] = false`
(less secure). /home/tempest-web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/ssl_socket.rb:120:in `connect_nonblock' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/ssl_socket.rb:120:in `initialize' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/connection.rb:404:in `new' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/connection.rb:404:in `socket' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/connection.rb:106:in `request_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/middlewares/mock.rb:47:in `request_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/middlewares/instrumentor.rb:25:in `request_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/middlewares/base.rb:15:in `request_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/middlewares/base.rb:15:in `request_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/middlewares/base.rb:15:in `request_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/connection.rb:250:in `request' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/middlewares/idempotent.rb:26:in `error_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/middlewares/base.rb:10:in `error_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon
0.49.0/lib/excon/middlewares/base.rb:10:in `error_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/connection.rb:273:in `rescue in request' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/connection.rb:221:in `request' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/middlewares/idempotent.rb:26:in `error_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/middlewares/base.rb:10:in `error_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/middlewares/base.rb:10:in `error_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/connection.rb:273:in `rescue in request' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/connection.rb:221:in `request' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/middlewares/idempotent.rb:26:in `error_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/middlewares/base.rb:10:in `error_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/middlewares/base.rb:10:in `error_call' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/connection.rb:273:in `rescue in request' /home/tempest-
web/tempest/web/vendor/bundle/ruby/2.3.0/gems/excon-
0.49.0/lib/excon/connection.rb:221:in `request' . 

Comments

Powered by Zendesk