Pivotal Knowledge Base

Follow

Spring Application restTemplate returns bad_certificate error

Environment

  • Java Offline Buildpack 4.5
  • Spring Web 4.3.9

Symptom

Spring app deployed using the java buildpack fails to establish SSL handshake with server endpoint that exists outside of the PAS Platform. The error observed when making the request is "Received fatal alert: bad_certificate"

Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_152]
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) ~[na:1.8.0_152]
        at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2033) ~[na:1.8.0_152]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135) ~[na:1.8.0_152]
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[na:1.8.0_152]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[na:1.8.0_152]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[na:1.8.0_152]
        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) ~[na:1.8.0_152]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[na:1.8.0_152]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162) ~[na:1.8.0_152]
        at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
        at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
        at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
        at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:99) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
        at org.springframework.cloud.sleuth.instrument.web.client.TraceRestTemplateInterceptor.response(TraceRestTemplateInterceptor.java:69) ~[spring-cloud-sleuth-core-1.2.4.RELEASE.jar!/:1.2.4.RELEASE]
        at org.springframework.cloud.sleuth.instrument.web.client.TraceRestTemplateInterceptor.intercept(TraceRestTemplateInterceptor.java:63) ~[spring-cloud-sleuth-core-1.2.4.RELEASE.jar!/:1.2.4.RELEASE]
        at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:86) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
        at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:70) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
        at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
        at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:652) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
        ... 82 common frames omitted

Additionally, one might verify the server cert is trusted by a known root certificate authority and commands like curl or openssl will succeed in verifying the certificate chain given the root ca certificate is passed in as an argument.

ubuntu@bosh-stemcell:~$ curl https://10.1.1.31:25555 --cacert /path/root_ca_certificate -vv
* Rebuilt URL to: https://10.1.1.31:25555/
* Hostname was NOT found in DNS cache
*   Trying 10.1.1.31...
* Connected to10.1.1.31 (10.1.1.31) port 25555 (#0)
* successfully set certificate verify locations:
*   CAfile: /var/tempest/workspaces/default/root_ca_certificate
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
* 	 subject: C=US; O=Pivotal; CN=10.1.1.31
* 	 start date: 2017-10-01 22:30:28 GMT
* 	 expire date: 2019-10-01 22:30:28 GMT
* 	 subjectAltName: 10.1.1.31 matched
* 	 issuer: C=US; O=Pivotal
* 	 SSL certificate verify ok.

Cause 

After enabling SSL debug logging, the problem surfaces because the server has mutual TLS enabled and the application is using the default Diego instance certificate. The SSL handshake behavior is as follows. For reference the client in this case is the deployed spring application

ssl-error-diagram.jpeg

The server is returning the SSL handshake error because it does not trust the Diego root instance authority as a valid Certificate Authority.

In this case the Application is using "restTemplate.exchange" to send a request to the HTTPS endpoint. By default the java buildpack will add the required root ca certs and source the client certs from Diego. The restTemplate will source these certs and use them in the default HTTP client.

Resolution

Use a Custom HTTP Client

Using a custom HTTP client is an option that can be used to override default settings provided by the system. One way to do that is to build your own HTTP client and define the SSL context that suits your needs. Here is an example of that.

HttpClientBuilder httpClientBuilder = HttpClients.custom()
                .setSSLContext(SSLContext.getDefault())
                    .useSystemProperties();

restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory(httpClientBuilder.build())

Change the Default Client Certs

As per the java buildpack docs, users can override the cert location using environmental variables.

Additional Information 

Troubleshooting tips

  • Enabling SSL debug logging to view the full SSL handshake trace
    • Add the javax.net.debug JAVA_OPTS to your application manifest
      ---
      applications:
        - name: spring-app
          memory: 1024M
          instances: 1
          env:
            JAVA_OPTS: "-Djavax.net.debug=ssl:all:verbose"
    • More options are available in the Oracle Documentation
              all        turn on all debugging
              ssl        turn on ssl debugging
      
              The following can be used with SSL:
                  record          enable per-record tracing
                  handshake       print each handshake message
                  keygen          print key generation data
                  session         print session activity
                  defaultctx      print default SSL initialization
                  sslctx          print SSLContext tracing
                  sessioncache    print session cache tracing
                  keymanager      print key manager tracing
                  trustmanager    print trust manager tracing
      
              handshake debugging can be widened with:
                  data            hex dump of each handshake message
                  verbose         verbose handshake message printing
      
              record debugging can be widened with:
                  plaintext       hex dump of record plaintext
                  packet          print raw SSL/TLS packets

 

Comments

Powered by Zendesk