Pivotal Knowledge Base

Follow

Ops Manager Using More IP Addresses than Expected or giving a Not Enough IPs Available Message

Environment

 Product  Version
 Pivotal Cloud Foundry  1.3.x, 1.4.x, 1.5.x

Symptom

When trying to deploy Pivotal Cloud Foundry® or a service tile through Ops Manager, you get the error:

"Not enough available IPs, IP cannot be allocated for 'xxxx.'"

OR

"It appears like Pivotal Cloud Foundry is using more IP addresses than you would expect it to be using."

Cause

There is generally not one cause to this problem; rather, there are a few different things that all contribute to running out of IP addresses or Pivotal Cloud Foundry seeming like it is using more IP addresses than you'd expect.

As you might expect, each VM listed on the resource page of a product in Ops Manager is allocated one IP address. This is what you'd generally expect, one IP address per VM.

Things start to differ when the Ops Manager will allocate IP addresses for every VM that's necessary for the installation, even if it's just a temporary VM. This means that VMs like compilation and errand VMs will be allocated an IP address even though they only run for short periods of time. Another way to think of this is that Ops Manager is reserving IP addresses for those temporary VMs.

The last complication is regarding static IP addresses. Ops Manager will allocate two IP addresses for each VM instance that requires a static IP address. The first IP address is used to bootstrap the VM and the second IP address is the static IP that will be used once the VM is up and running.  After the VM is up and running, the first bootstrap IP will no longer be used, but it is still allocated to that VM, and no other VMs can use it.  

A common example of this is the HAProxy VM because it's very often given a static IP address by the user. However, there are also VMs that get assigned static IP addresses by Ops Manager itself and not based on user input, which means there are likely more static IP addresses being used in an installation than you would expect.

Resolution

At the moment, there's no easy way to see what IP addresses are being used by Pivotal Cloud Foundry. It is possible to inspect the usage by looking at the installation configuration file, though. The instructions below walk through the process.

  1. Navigate to the Ops Manager GUI, using this URL: https://<your-om-ip-or-dns>/debug/files
  2. Search the page for `actual_installation @`. Below this line, you'll see the configuration file for your Pivotal Cloud Foundry installation.  Copy the configuration file and past it into a text editor.
  3. Search for "ips:." This will show you the blocks where the IP addresses are listed. It will look like this.
      ips:
        nats-part-a4600306c7ba2773f6f5:
        - 10.64.37.12
        etcd_server-part-5974eedd6e5f36ef11f1:
        - 10.64.37.13
        nfs_server-part-5907c1447f30c9366c7b:
        - 10.64.37.14
        ccdb-part-26376c994b4837f08622:
        - 10.64.37.15
        ...
    The IPs block takes the following form:
      ips:
        <vm-name>:
        - ip#1
        - ip#2
        - etc..
    <next-vm-name>:
    - ip#1
    - ip#2
    - etc..

From this form, you can see which IP or IP addresses are assigned to specific VMs or count them up to see how many IP addresses have been allocated in total.

Within the IP's block, there is one special section that merits some explanation, and that's dynamic_for_static. As mentioned above, VMs that require static IP addresses are allocated two IP addresses. This section lists the bootstrap IPs that are used by VMs who are assigned static IPs.

Please note that each tile deployed will have it's own IPs block. This means you'll minimally have two, one for the Director tile and one for the Pivotal Cloud Foundry tile. Each additionally deployed service tile will have it's own IPs block as well.

Impact/Risks

Be very careful with the installation configuration file that is copied out of Ops Manager. This file contains highly sensitive information about your environment such as IP addresses, private keys, and usernames (passwords are masked out). Be sure to delete the file when you're done inspecting it.

Comments

Powered by Zendesk