Pivotal Knowledge Base

Follow

Unable to scale gorouters using fixed IP addresses across different Availability Zones

Environment

Pivotal Cloud Foundry® 1.9 to 1.12

Symptom

Unable to scale out gorouters using fixed IP addresses across different Availability Zones.

Multiple gorouters are present across different Availability Zones. Scaling out of the gorouters fails if fixed reserved IP Addresses are used for additional gorouter instances to be added. An example of the error is seen as below:

IP '10.0.0.22' is already taken, cannot be taken for 'cf-abc123_router-xyz456-partition-def789'

Analysis

Let's take an example where a network env01 with 3 Availability Zones az01, az02 and az03 containing 4 gorouter IP Addresses as follows:

az01: router-partition-abc123
10.0.0.20
10.0.0.23
az02: router-partition-def456
10.0.0.21
az03: router-partition-ghi789
10.0.0.22

4 goRouter instances:
Assigned IPs: 10.0.0.20,10.0.0.21,10.0.0.22,10.0.0.243

Increased gorouter to 8 instances (under Resource Config).
Changed Assigned IPs to: 10.0.0.20,10.0.0.21,10.0.0.22,10.0.0.23,10.0.0.24-10.0.0.27

Clicked Apply changes.
Error:

Please review the errors below
IP '10.0.0.22' is already taken, cannot be taken for
'cf-xyz123_router-0d0ddaaba72ea77ab934-partition-abc123'

Cause

This is a known issue and currently, there is no fix available for this issue.

Resolution

There are currently two workarounds available:

1. Scale down the instances to just one instance and remove all IPs from fixed IPs list except the first instance IP. Follow these detailed steps very carefully to workaround this issue:

  • Leave the first IP in Assigned IPs in ERT for router and scale down to 1 router instance in Resource config tab. In this example, leave the IP 10.0.0.20 in Assigned IPs.
  • Click Apply changes. This should complete successfully.
  • Add the failing IP to Reserved IPs in Ops Manager tile Networks tab to avoid further conflict. In this example, add the IP 10.0.0.22 in Ops Manager tile - Networks tab - Reserved IPs for the used network.
  • Go back to ERT tile and add rest of the IPs in router's Assigned IPs list. Note that, we'll have one less IP than we desire as the failing IP is now Reserved in Ops manager Networks tab. In this example, the assigned IP list would be: 10.0.0.20,10.0.0.21,10.0.0.23,10.0.0.24-10.0.0.27.
  • Increase the number of instances to one less than desired. In this example, seven.
  • Click Apply changes. This should complete successfully.
  • Lastly, remove the failing IP (in this example, 10.0.0.22) from Reserved IPs in Ops Manager tile Networks tab and add that IP to ERT router Assigned IPs list. In this example, the final list of Assigned IPs should be: 10.0.0.20,10.0.0.21,10.0.0.22,10.0.0.23,10.0.0.24-10.0.0.27.
  • Increase the number of instances to the desired number now. In this example, eight.
  • Click Apply changes. This should complete successfully.

2. Re-order the IP Addresses of the 4 existing and 4 new IP Addresses in following order:

    • Let's consider the example above in Analysis section.
    • {existing1},{existing2},{new1},{existing3},{new2},{new3},{existing4},{new4}
    • To understand the above technique to re-order, consider 4 IP Addresses distributed with one each in the 3 AZs and the last IP Address in 1st AZ. To insert the 5th (new) IP Address in 2nd AZ, we add it to the 3rd location.
    • The final result of the above re-ordering would look like:
az01: router-partition-abc123
10.0.0.20 {existing1}
10.0.0.22 {existing3}
10.0.0.23 {existing4}
az02: router-partition-def456
10.0.0.21 {existing2}
10.0.0.25 {new2}
10.0.0.27 {new4}
az03: router-partition-ghi789
10.0.0.24 {new1}
10.0.0.26 {new3}
  • The trick here is to keep the existing IP Addresses limited to the first and maybe second AZ. However, all new IP Addresses must be outside the first AZ.

For any other use case, please contact Pivotal Support for further assistance.

Comments

Powered by Zendesk