Pivotal Knowledge Base

Follow

How to re-apply IPsec settings if IPsec is lost

Environment

 Product  Version
 Pivotal Cloud Foundry  1.8, 1.9, 1.10

Purpose

IPsec settings are generally recommended to be applied during the time of initial install. These settings, however, may be lost in the rare event that BOSH run-time configuration is overwritten (see Cause section for details). If IPsec settings are lost from the VM's, then the VM's will not be able to communicate and start failing. The purpose of this article is to describe how to remedy this scenario of IPsec settings getting lost.

Symptom

Settings are not present when you run the command:

bosh runtime-config

If IPsec settings are lost, then installation will run until the point where the router is updated, at which point it will get a timeout pinging the router VM:

Started updating job router > router/1 (1c0a4237-467b-44fb-81a8-e9c6401c4034). 
Failed: Timed out pinging to 1245ec6f-6837-4d1f-8ec1-5537b8a4a117 after 600 seconds (00:16:17) Error 450002: Timed out pinging to 1245ec6f-6837-4d1f-8ec1-5537b8a4a117 after 600 seconds

Cause 

Certain OSS procedures involve manually updating bosh runtime-config:

# bosh update runtime-config runtime-config.yml

This command will overwrite existing runtime config and cause IPsec settings to be lost. The configuration should *append* existing config with updates when running this command, otherwise, the existing settings such as IPsec will be removed from the VM's.

Procedure

If IPsec is version 1.6 or later:

For IPsec version 1.6 or later, follow step 12 of the Pivotal Cloud Foundry IPsec Add-On document to resolve this issue.

Prerequisites: 
  • Acquire the IPsec manifest: ipsec-addon.yml 
  • An administrator should have created ipsec-addon.yml when originally deploying the IPSec tile, as documented in the link above. See the Notes section below if you do not have this configuration file.

Steps:

  1. Set the optional flag to true
  2. Navigate to your Installation Dashboard in Ops Manager
  3. Click Apply Changes
  4. Wait for the installation to complete
  5. Set the optional flag to false
  6. Update the runtime config
    $ bosh update runtime-config PATH/ipsec-addon.yml
  7. After updating runtime config, verify that the settings are correct
    $ bosh runtime-config
  8. Navigate to your Installation Dashboard
  9. Click Apply Changes

If IPsec is version 1.5 or lower:

There is no optional flag prior to 1.5, so we simply need to perform these steps:

  1. In OpsManager, revert changes to existing installation which failed
  2. Update the runtime config
    $ bosh update runtime-config PATH/ipsec-addon.yml
  3. After updating runtime config, verify that the settings are correct
    $ bosh runtime-config
  4. Navigate to your Installation Dashboard
  5. Click Apply Changes

Once applying changes, the IPsec settings will be re-applied to any VM's missing this setting. Once VM's are updated, they will be able to communicate again and this will resolve the failing state issue of the VM's.

Notes

If IPsec settings are lost then you will see following in the installation.log

    addons:
- - name: ipsec-addon  
-   jobs:
-   - name: ipsec
-     release: ipsec
-   properties:
-     ipsec:
-       ipsec_subnets:
-       - ""
-       - ""
-       - ""
-       - ""
-       - ""
-       - ""
-       - ""
-       no_ipsec_subnets:
-       - ""
-       - ""
-       - ""
-       instance_certificate: ""
-       instance_private_key: ""
-       ca_certificates:
-       - ""
-       prestart_timeout: ""

All the information to recreate the runtime config can be found in the directory `/var/vcap/jobs/ipsec/etc` on any VM that did not get updated. This could be used if you do not have the original IPsec manifest to update runtime-config.

Comments

Powered by Zendesk