Pivotal Knowledge Base

Follow

Pivotal Cloud Foundry 1.11 or 1.12 Upgrade Fails with Diego Cell Update Failures due to "acquire subnet lease: http status 409: No lease available " error

Environment

Pivotal Cloud Foundry (PCF) 1.11 and above

Symptom

PCF 1.11 or PCF 1.12 upgrade fails with Diego cell update failures. From the Diego cell logs, the following errors are logged in the silk-daemon.stdout.log and silk-daemon.stderr.log:

silk-daemon.stdout.log :

{"timestamp":"1520997730.240546465","source":"cfnetworking.silk-daemon","message":"cfnetworking.silk-daemon.http-client",
"log_level":2,"data":{"body":"{\"error\": \"No lease available\"}","code":409,"error":"No lease available"}}

silk-daemon.stderr.log :

2018/03/14 03:22:10 cfnetworking.silk-daemon error: acquire subnet lease: http status 409: No lease available 

Cause

Starting PCF 1.11, if  container networking is enabled, it will use the vxlan overlay network for an app to app communication.

The subnet IP addressing scheme for container networks is also changed when container networking is enabled. In the new addressing scheme, the 4th octet of the subnet is reserved for containers per Diego cell. This equals 254 containers per Diego cell. The first 3 octets are for the number of Diego cells.

Thus, if 10.254.0.0/22 is configured, and since the last octet is used for containers per cell, the two bits from the 3rd octet that is available as part of the /22 CIDR range will only accommodate 3 Diego cells (22 - 1) = 3. When more than 3 Diego cells are configured in Elastic Runtime, the “Apply Changes” will fail with the above errors. This is because the number of Diego cells are limited by the subnet selection for the "Overlay Subnet". See table below on how the subnet CIDR range for the Overlay Subnet will affect cell capacity in ERT/PAS:

Overlay Subnet CIDR

Number of cells

Containers per cell

/22

3

254

/20

63

254

/16

255

254

/12

4095

254

By default, 10.255.0.0/16 Subnet CIDR range is used for the Overlay Subnet for container networking.

Resolution

Under Elastic Runtime or PAS Tile, Go to Settings and Networking, check the Overlay Subnet box. If this Subnet is configured for 10.254.0.0/22, remove this setting or add the default Subnet of 10.255.0.0/16 and click Apply Changes.

Additional Information

  1. Starting PCF 1.12, the container to container networking is the default configuration and the legacy container networking option is not available.
  2. There are no changes to the networking requirements for CF infrastructure components. Traffic to application containers from the GoRouter and from the application containers to external services continue to use the cell IP and NAT.
  3. CF container networking is currently only supported on Linux.

Comments

Powered by Zendesk