Pivotal Knowledge Base

Follow

Enabling EBS Volume Encryption for PCF on AWS

Environment

  • Pivotal Cloud Foundry® (PCF) 1.12.x, 2.0.x, 2.1.x
  • Operations Manager
  • Pivotal Application Service (PAS) AKA Elastic Runtime (ERT)
  • AWS

Purpose

Refer to this article when an Operator needs to change an existing AWS deployment from without EBS Volume encryption to with EBS Volume Encryption, or when an Operator needs to rotate the AWS default key or the KMS key ARN (2.0.4 and above).

In PAS 2.0.4 and above, you can specify a KMS key ARN otherwise PCF will use the default AWS key. PCF can only use the default AWS key for Elastic Runtime 1.12.x and below. 

Procedure

Method 1 - Upgrade

Upgrade to the latest Operations Manager, PAS, and Service Tile maintenance releases with a new Stemcell. Review the Operations Manager, PAS, and Service Tiles release notes. 

  1. Open AWS Config from the Operations Manager, then select the AWS Management Console Config page to check the Encrypt EBS volumes box. The default AWS key will be used unless a KMS key ARN is entered (2.0.4 and above).
  2. Increase the persistent disk size of the BOSH Director and all of its managed VMs to have the new key applied to newly created persistent disks.
  3. Click Apply Changes to encrypt all of the BOSH Director VM, PAS, and the Service Tile disks.

Method 2 - Non-Upgrade

An operator does not upgrade but still needs to Encrypt EBS Volumes for an existing AWS PCF deployment, he or she needs to use the following steps to first encrypt the BOSH Director and all its managed VMs.

1. Open the Operations Manager AWS Config then select the AWS Management Console Config page to check the Encrypt EBS volumes. The default AWS key will be used unless a KMS key ARN is entered (2.0.4 and above).

2. Encrypt the BOSH Director VM.

1. SSH into Operations Manager VM, go to the folder /var/tempest/workspaces/default/deployments

2. Backup the bosh-state.json file

3. Edit the bosh-state.json file to remove the current_stemcell_id and stemcells values. 

The following is the example of an updated bosh-state.json file

{
...
    "current_stemcell_id": "",
...
    "stemcells": [],
    "releases": [
...

4. Increase the persistent disk size of the Director VM.

5. Operators should watch the AWS Management Console to ensure the private snapshot and AMI are created, and the new BOSH Director EBS volume is encrypted with a private AMI.

3. Enable BOSH managed VMs encryption for existing PAS and Service tiles: 

  1. SSH into Operations Manager VM, use the BOSH CLI to login.
  2. Use "bosh stemcells" and "bosh deployments" to determine the correct Stemcells used by the BOSH deployed VMs.
  3. Go to the folder /var/tempest/stemcells and use "bosh upload stemcell <stemcell name> --fix" to enforce the BOSH Director to encrypt the Stemcells and re-upload them.
  4. Increase the persistent disk sizes of all deployed VMs to have the new key applied to existing persistent disks. Alternatively, you can refer to the Create custom persistent disk sizes link below.
  5. Check the Recreate all VMs option in the Director Config page to apply the new key to all disks.
  6. Click Apply Changes

4. Encrypt the Operations Manager VM root disk (Optional).

Encrypt Operations Manager AMI through AWS CLI. Alternatively, check the documentation link for the manual steps (PCF 2.0/2.1 Ops Manager AMI)

$ aws ec2 copy-image --source-region <source region> --source-image-id <OM AMI at source region>  --region <dest region> --name <new encrypted AMI name> --encrypted --kms-key-id <KMS key ARN>

1. Launch a new Operations Manager instance from the encrypted AMI.

2. Take a backup from the original Operations Manager and import it to new Operations Manager instance. Click Apply Changes.

Impact / Risk

An HA deployment without any singleton jobs is required for zero downtime consideration.

Additional Information 

PCF 2.0 and 2.1 Operations Manager Release Notes

https://docs.pivotal.io/pivotalcf/2-0/pcf-release-notes/opsmanager-rn.html

https://docs.pivotal.io/pivotalcf/2-1/pcf-release-notes/opsmanager-rn.html

PCF 2.0 and 2.1 Operations Manager AMI 

https://docs.pivotal.io/pivotalcf/2-0/customizing/pcf-aws-manual-config.html#pcfaws-om-ami

https://docs.pivotal.io/pivotalcf/2-1/customizing/pcf-aws-manual-config.html#pcfaws-om-ami

Reference to v2.0 and 2.1 Encrypt EBS volumes with KMS key

https://docs.pivotal.io/pivotalcf/2-0/customizing/cloudform-om-ebs-config.html

https://docs.pivotal.io/pivotalcf/2-1/customizing/cloudform-om-ebs-config.html

Reference to v1.12 Encrypt EBS volumes

https://docs.pivotal.io/pivotalcf/1-12/customizing/cloudform-om-ebs-config.html

Create custom persistent disk sizes

http://opsman-dev-api-docs.cfapps.io/#overriding-defaults-with-custom-disk-types

Note: If you have any question or need assistance with the steps mentioned above, please open a case with Pivotal Support before proceeding.

Comments

Powered by Zendesk