Pivotal Knowledge Base

Follow

Application Container Unable to Resolve DNS in Google Cloud Platform

Environment

Pivotal Cloud Foundry (PCF) 1.8, 1.9, 1.10

Elastic Runtime 1.8, 1.9, 1.10

Google Cloud Platform (GCP)

Symptom

After fresh installed PCF, apps manager is able to login but no information retrieved. Further investigation found application container fail to resolve DNS from 169.254.169.254 .

Root Cause

In Elastic Runtime, the application container inherits its DNS configuration from BOSH network configuration. The "Installing Ops Manager" documentation suggests that you configure the Bosh Director's DNS as `169.254.169.254, 8.8.8.8`. However, if you omit the second DNS entry, this will configure the Director and all Bosh Deployed VMs to only use GCP's internal DNS.

This effectively gives Bosh deployed VM's a `/etc/resolv.conf` with the following:

nameserver 127.0.0.1
nameserver 169.254.169.254

127.0.0.1 is added by the consul job running on the Diego Cell VM as part of the service discovery function via DNS. 169.254.169.254 is GCP's metadata endpoint and also functions as an internal DNS resolver.

As mentioned above, Elastic Runtime and your application containers inherit their DNS configuration from the Bosh Director's DNS configuration. If the Bosh Director's DNS is configured as suggested by the documentation, you'll end up with `169.254.169.254, 8.8.8.8`. If you omit `8.8.8.8`, you'll end up with just `169.254.169.254`. The second situation, omitting `8.8.8.8` (or some public DNS server) is what causes the issue described in this KB.

The reason for this is because of the Consul endpoint, `127.0.0.1` is not added to the container's DNS list, and the metadata endpoint of cloud providers like AWS, Azure and Google Cloud Platform is not accessible from inside the container. If it were accessible, the endpoint would expose sensitive environment information about your GCP environment to potentially untrusted apps running in the container. Because this endpoint is blocked and because no public DNS server was specified, the application container does not have access to a DNS resolver.

Resolution

A new field in Elastic Runtime -> Application Containers -> DNS Servers has been introduced in version 1.10 and above. This value overrides DNS servers to be used in containers. The user can specify 169.254.169.254 in the BOSH DNS configuration, and then particularly specify 8.8.8.8 in the application container DNS configuration to override the default value 169.254.169.254 . 

Comments

Powered by Zendesk