Pivotal Knowledge Base

Follow

Errands intermittently fail with EOF error when executing 'cf auth' on NetScaler

Environment

  • Pivotal Application Service (PAS) 1.11, 1.12, 2.0, and 2.1
  • NetScaler Load Balancer

Symptom

CF CLI fails with the following End of File (EOF) error when executing "cf auth"

cf auth system_verification [REDACTED]
API endpoint: https://api.system.domain.com
Authenticating...
Post https://login.system.domain.com/oauth/token: EOF
FAILED

Cause

When the CF CLI communicates with the Elastic Runtime UAA resource it will need to make two HTTP requests to UAA.

  • GET /login
    • The expected response includes information about the login endpoints.
  • POST /oauth/token
    • CF CLI expects the UAA to return an access token.

CF CLI reports an EOF error when sending the second request, "POST /oauth/token". This happens when NetScaler is the load balancer and configured to only allow a single request per connection.

When CF CLI sends the first GET request it is setting the HTTP header "Connection: close" which means the TCP connection will close after the response is received. NetScaler does not send back the same header in the response resulting in CF CLI attempting to reuse the same TCP connection for the post request. While sending the post request the NetScaler closes the TCP session and as a result, an EOF error is returned.

See the diagram to understand the scenario.

sqd.jpeg

Resolution

The CF CLI fix is in release V6.36.1. The fix for Pivotal Application Service tile and other impacted services are listed below.

 Tile and Product

 Fix

 Concourse

3.11.0/3.13.0

 PAS

 2.1.4

 MySQL

 TBA

 Pivotal On Demand Service Broker SDK

0.21.1 

Workaround for PAS 2.0

The new collocated errand feature in PAS 2.0 allows us to circumvent the name resolution of the login.system.domain.com URL.

  1. When executing the errand review the BOSH task logs to identify which instance is executing the failing errand. Optionally you can run the errand manually and specify which instance you want the errand to run on with "--instance=" flag
  2. Once the instance is identified, "BOSH SSH" to it and modify the "/etc/hosts" file to include an entry that points the login.system.domain.com to the IP address of one of the Gorouters.
    [IP OF GO ROUTER] login.SYSTEM.DOMAIN.COM
  3. Run the errand again and confirm it completes successfully.

Workaround PAS 1.12 or earlier

In PAS 2.0 the new collocated errand feature allows us to circumvent the name resolution of the login.system.domain.com URL.

  1. Execute the errand manually using the "--keep-alive" flag to prevent BOSH from deleting the instance once the errand fails.
  2. Use BOSH to SSH into the errand VM and modify the "/etc/hosts" file to include an entry that points the login.system.domain.com to the IP address of one of the Gorouters.
    [IP OF GO ROUTER] login.SYSTEM.DOMAIN.COM
  3. After SSH into the errand, VM determines how to execute the errand directly. Do not run "BOSH run-errand" as it will recreate the VM instance.

Comments

Powered by Zendesk