Pivotal Knowledge Base

Follow

Infinite Redirect Loop in Browser while Accessing Cloud Foundry Metrics Dashboard

Environment

 Product  Version
 Pivotal Cloud Foundry (PCF) Metrics  1.2.x

Symptom

When attempting to login to PCF Metrics dashboard via the browser, the login request is processed but results in a continuous redirect loop with a blank screen. 

Error Message:

In the Chrome developer tools, under Network tab, the following URLs are seen cycled in a continuous loop: 

https://login.run.<domain>/oauth/authorize?authorities=uaa.resource&client_id=apps_metrics&redirect_uri=https%3A%2F%2Fmetrics.run.<domain>%2Fauthorization%2Fcode&state=https%3A%2F%2Fmetrics.run.<domain>%2F&response_type=code&grant_type=authorization_code&scope=cloud_controller.admin%2Ccloud_controller.read
https://metrics.run.<domain>/authorization/code?code=4W65bI&state=https://metrics.run.<domain>/ 

From the logs for the metrics app, the following relevant error is seen: 

2017-01-19T17:28:17.73-0600 [APP/0] OUT {"name":"apm","hostname":"0143779f-4ae7-4def-7b93-5786c46ea13c",
"pid":57,"level":50,"err":{"name":"FetchError","message":"request to https://login.run.<domain>/oauth/token failed,
reason: unable to verify the first certificate","type":"system","errno":"UNABLE_TO_VERIFY_LEAF_SIGNATURE",
"code":"UNABLE_TO_VERIFY_LEAF_SIGNATURE"},"msg":"access token fetch failed","time":"2017-01-19T23:28:17.736Z","v":0}

To access the logs for the metrics app, run the following commands:

1. Login as admin via CF CLI:

$ cf login -a api.system.<domain> -u admin -o system -s metrics-v1-2

2. In a browser window, access the metrics dashboard, and from a terminal window, capture the logs from the "metrics" app:

$ cf logs metrics 

Cause 

As noted in the above error logs, the request to the login server failed with "unable to verify the first certificate" error.

This indicates that the issue is related to the failure with validating SSL certificates during the TLS negotiation between the client metrics node.js app and UAA node.js (and thus the node.js build pack) hard code roots CA certs in its source code and ignores Bosh configured trusted root certificates. If the custom root CA certificate installed in Cloud Foundry is not in node.js trusted root CA cert list, SSL check will fail. 

Resolution

As a workaround, the metrics apps and its associated worker apps can be configured to disable SSL checking via an environment variable. To disable SSL checking, follow the steps shown below:

  1. Run "cf apps" from within the "system" org and "metrics-v1-2" space
    $ cf apps 
    Getting apps in org system / space metrics-v1-2 as admin...
    OK

    name requested state instances memory disk urls
    elasticsearch-logqueue started 2/2 512M 1G elasticsearch-logqueue.system.10.193.69.250.piv.io
    elasticsearch-logqueue-blue stopped 0/2 512M 1G elasticsearch-logqueue.system.10.193.69.250.piv.io
    metrics started 1/1 512M 2G metrics.system.10.193.69.250.piv.io
    metrics-aggregator started 1/1 256M 1G
    metrics-aggregator-blue stopped 0/1 256M 1G
    metrics-ingestor started 1/1 512M 1G
    metrics-ingestor-blue stopped 0/1 512M 1G
    mysql-logqueue started 2/2 1G 1G mysql-logqueue.system.10.193.69.250.piv.io
    mysql-logqueue-blue stopped 0/2 1G 1G mysql-logqueue.system.10.193.69.250.piv.io
    worker-app-dev started 1/1 1G 1G
    worker-app-logs started 1/1 1G 1G
    worker-health-check started 1/1 1G 1G
    worker-reaper started 1/1 1G 1G
  2. From the output of the above command, the env settings needs to be changed for apps metrics, worker-app-dev, worker-app-logs, worker-health-check and worker-reaper. Run the following command to change the env variable for SSL checking for the metrics app 
    $ cf set-env metrics SKIP_CERT_VERIFY true
    Repeat the above command for the remaining of the 4 worker-apps worker-app-dev, worker-app-logs, worker-health-check and worker-reaper.
     
  3. Restage apps metrics, worker-app-dev, worker-app-logs, worker-health-check and worker-reaper. For restaging metrics app, run the following command
    $ cf restage metrics
    Repeat the above command for the remaining of the worker-apps worker-app-dev, worker-app-logs, worker-health-check and worker-reaper

  4. Once all of the identified apps are restaged, users will now be able to login via the metrics dashboard.

Note: The above resolution is only a workaround. The final fix will be available in a future PCF Metrics release.

Additional Information  

For additional troubleshooting, please refer to the PCF Metrics Troubleshooting documentation. 

 

Comments

Powered by Zendesk