Pivotal Knowledge Base

Follow

Rotating CA Certificates for Pivotal Cloud Foundry Services

Environment

Any Pivotal Cloud Foundry (PCF) service that supports TLS.

Purpose

Warning: Do not attempt this procedure on your own. Contact Pivotal Support and perform this procedure with their assistance.

This Knowledge Base article describes how to rotate CA certificates in BOSH CredHub when using a Pivotal Cloud Foundry (PCF) service tile that supports TLS.

Procedure

Configuring service tiles for TLS requires either providing a CA certificate to CredHub or generating it with CredHub. Operators must rotate the CA certificate before it expires or if it is compromised.

Add New CA Certificate

Perform the following steps to add a new CA certificate to CredHub:

1. Retrieve the IP address of the BOSH Director VM and the Director credentials by performing the steps in Gather Credential and IP Address Information. Both the UAA and CredHub servers are colocated on the BOSH Director VM.

2. SSH into the Ops Manager VM by performing the steps in SSH into Ops Manager of Advanced Troubleshooting with the BOSH CLI.

3. If you are using PCF v1.12, you must download the CredHub CLI onto the Ops Manager VM. Run the following command:

  • wget https://github.com/cloudfoundry-incubator/credhub-cli/releases/download/1.6.0/credhub-linux-1.6.0.tgz

4. Use the CredHub CLI to generate a CA certificate or provide an existing one.               

Note: Your PCF deployment may have multiple CA certificates. Pivotal recommends a dedicated CA certificate for services.

  • If you do not have a CA certificate, use the CredHub CLI to generate one. Enter the following command:
    • credhub generate --name="/services/tls_ca" --type="certificate" --is-ca --common-name="rootCA"
  • If you have an existing CA certificate that you want to use, create a new file called "root.pem" with the contents of the certificate. Then enter the following command, specifying the path to `root.pem` and the private key for the certificate:
    • credhub set --name="/services/tls_ca" --type="certificate" --certificate=./root.pem --private=ERKSOSMFF...

5. Use the BOSH CLI v2 to extract the "certificate" portion from the CA certificate and print it. Run the following command:

  • bosh2 int <(credhub get --name=/services/tls_ca) --path /value/certificate

6. Copy the output.

7. Navigate to the Ops Manager Installation Dashboard and click the Ops Manager Director tile. Click Security.

8. Append the contents of the new CA certificate to the old CA certificate under Trusted Certificates. Do not remove the old CA certificate. Click Save.

9. Return to the Ops Manager Installation Dashboard and click Apply Changes.
This restarts all the VMs in your PCF deployment and applies your CA certificate. 

Warning: Restarting all of the VMs in your PCF deployment in order to apply a CA certificate takes a long time to complete.

Regenerate Certificates and Redeploy Service Instances

Perform the following steps to regenerate your certificates and redeploy your service instances:

  1. From the Ops Manager VM, log in to CredHub with the CredHub CLI, using the client secret you set when you first created a UAA client for CredHub. For example:
    • credhub login --client-name=credhub --client-secret=abcdefghijklm123456789
  2. Use curl to make a call to the CredHub server on the BOSH Director VM to bulk regenerate all certificates signed by the old CA certificate. Run the following command, replacing BOSH-DIRECTOR with the IP address of the BOSH Director VM that you retrieved above:
    • curl "https://BOSH-DIRECTOR/api/v1/bulk-regenerate" -X POST -d '{ "signed_by": "/services/tls_ca" }' -H "authorization: bearer $(credhub --token)" -H 'content-type: application/json'
  3. Perform the steps in Log in to the BOSH Director to log in to the BOSH Director from the Ops Manager VM.
  4. Run the upgrade-all-service-instances errand on the BOSH deployment for your service to redeploy all service instances with the new certificates. Enter the following command, specifying the name of the BOSH deployment for your service. For example:
    • bosh2 -d p-mysql run-errand upgrade-all-service-instances                                      Note: If you do not know the name of the BOSH deployment for your service, run bosh2 -e my-env deployments to list deployments.
  5. All bound apps that do not validate server certificates using the trusted store must be re-bound to the service in order to receive the updated CA public key. This includes all apps that are not written in Java or Spring. Until these apps have been re-bound, they cannot reconnect to the service.

Remove Old CA Certificate

After all apps are able to reconnect to service instances with server certificates that have been generated by the new CA, follow these steps to remove the old CA certificate:

Perform the following steps:

  1. From the Ops Manager VM, use the CredHub CLI to remove the old CA certificate at `/services/tls_ca`. Enter the following command:
    • credhub delete --name '/services/tls_ca'
  2. Navigate to the Ops Manager Installation Dashboard and click the Ops Manager Director tile. Click Security.
  3. Delete the old CA certificate in Trusted Certificates and click Save.
  4. Return to the Ops Manager Installation Dashboard and click Apply Changes. This restarts all the VMs in your deployment.                                                               

WARNING: Restarting all of the VMs in your PCF deployment in order to remove a CA certificate takes a long time to complete.

Comments

Powered by Zendesk