Pivotal Knowledge Base

Follow

Scaling gorouter errors with IP already taken when assigning a static IP

Environment

Pivotal Operations Manager Version 1.x

Symptom

When attempting to scale up gorouter and assigning a static IP, Operations Manager Returns the following error for one of the assigned IPs

IP "10.10.10.10" is already taken, can not be taken for 'cf-3923uh29f2992f3wf_router-9w8h9ehfwihfe9wifhw'
The following ips you requested for 'Pivotal Elastic Runtime - Router' can not be found in any assigned neteorks: ["10.10.10.10"]

The output of bosh vms command does not include an existing vm with the ip of 10.10.10.10

# bosh vms
+------------------------------------------------------------------------+---------+-----+-------------+---------------+
| VM                                                                     | State   | AZ  | VM Type     | IPs           |
+------------------------------------------------------------------------+---------+-----+-------------+---------------+
| router/0 (45caf45b-9ad4-4244-b96c-3bd4a7e137cc)                        | running | az1 | micro       |10.10.10.7  |
| router/1 (61625641-b326-434f-837e-4e3129eb4127)                        | running | az1 | micro       |10.10.10.8  |
| router/2 (b58cb6fd-ff8b-4143-9e2c-a8bee49ff3e4)                        | running | az1 | micro       |10.10.10.9  |
+------------------------------------------------------------------------+---------+-----+-------------+---------------+

Cause 

In this case the operator is attempting to assign ip address 10.10.10.10 to the new gorouter, but sadly operations manager has already allocated that ip address to one of the errands. In this case the notifications-ui errand holds the ip address.

We can check for this by downloading the installation.yml using this kb How to modify Operations Manager installation.yml and determine which job has the ip assigned.

    "ip_assignments": {
        "assignments": {
            "cf-b726f387316441065827": {
                "notifications-ui-3f639252e9c255dade79": {
                    "a25903f1b4277d63b736": [
                        "10.10.10.10"
                    ]
                }

Resolution

If we can not choose a difffernt ip address for the new gorouter then we have to manually modify the operations manager configuration files to remove the ip from the errand VM. This will force operations manager to select a different ip address for the errand VM and assign the desired IP to the new gorouter.

  1. Use the mentioned in How to modify Operations Manager installation.yml to download the installation.yml
  2. Modify the ip assignments for the errand that is using the desired ip address to include no entries
        "ip_assignments": {
            "assignments": {
                "cf-b726f387316441065827": {
                    "notifications-ui-3f639252e9c255dade79": {
                        "a25903f1b4277d63b736": []
                    }
  3. ssh into the operations manager VM and backup the actual-installation.yml
    cp /var/tempest/workspaces/default/actual-installation.yml ~/actual-installation.yml
  4. Decrypt the actual-installation.yml. Make sure to use the operations manager decryption passphrase when prompted
    ubuntu@pivotal-ops-manager:/tmp$ sudo -u tempest-web RAILS_ENV=production /home/tempest-web/tempest/web/scripts/decrypt /var/tempest/workspaces/default/actual-installation.yml /tmp/actual-installation.yml
    Passphrase:
  5. Locate the entry for the notifications-ui errand ip assignment
                notifications-ui-3f639252e9c255dade79:
            a25903f1b4277d63b736:
            - 10.10.10.10
  6. Edit this entry and remove the ip assignment from the notifications-ui errand
          notifications-ui-3f639252e9c255dade79:
            a25903f1b4277d63b736: []
  7. Encrypt the modified actual-installation.yml and replace the existing one. Make sure to use the operations manager decryption passphrase when prompted
    ubuntu@pivotal-ops-manager:/tmp$ sudo -u tempest-web RAILS_ENV=production /home/tempest-web/tempest/web/scripts/encrypt /tmp/actual-installation.yml /var/tempest/workspaces/default/actual-installation.yml
    Passphrase:
  8. Proceed to apply changes in Operations Manager and confirm the gorouter is able to aqcuire the assigned ip address

 

Comments

Powered by Zendesk