Pivotal Elastic Runtime all versions
Sometimes in order to diagnose where an issue is occurring it may be beneficial to bypass any proxies or load balancers and make HTTP requests directly to the Cloud Foundry GoRouters.
This can help to determine if the proxy/load balancer is the cause of an issue or if the issue exists inside Cloud Foundry.
- Get the FQDN of the app.
$ cf app test-app Showing health and status for app test-app in org test / space test as admin... OK requested state: started instances: 1/1 usage: 1G x 1 instances urls: test-app.apps.example.com last uploaded: Mon Nov 27 13:30:00 UTC 2017 stack: cflinuxfs2 buildpack: php 4.3.40 state since cpu memory disk details #0 running 2017-11-27 01:31:34 PM 0.7% 24M of 1G 170.6M of 1G
- Get the IP address of a GoRouter, router/0 will do:
$ bosh vms | router/0 (394b1c87-7591-4def-90e5-c69fa744278e) | running | AZ01 | micro | 192.0.2.35 | | router/1 (2857d5bc-ed37-435d-942d-8d2fb5fed1b2) | running | AZ01 | micro | 192.0.2.36 | | router/2 (2931d750-da0d-480f-9c4c-43010daef38f) | running | AZ01 | micro | 192.0.2.37 |
- First use
curlto make a request using the APP-FQDN which will follow the standard network flow through the proxy/load balancer:
Example: curl -vvv test-app.apps.example.com Example output:
* Trying 192.0.2.250... * TCP_NODELAY set * Connected to test-app.apps.example.com (192.0.2.250) port 80 (#0) > GET / HTTP/1.1 > Host: test-app.apps.example.com > User-Agent: curl/7.54.0 > Accept: */* > < HTTP/1.1 403 Forbidden < Content-Type: text/html; charset=UTF-8 < Date: Wed, 29 Nov 2017 16:49:28 GMT < Server: lighttpd < Vary: Accept-Encoding < X-Vcap-Request-Id: d9cba8be-840e-478a-73ab-9ad258fe6cef < Content-Length: 36 < * Connection #0 to host test-app.apps.example.com left intact <p>Oops... something went wrong!</p>
- Then use
curlto construct a HTTP request targeting the application but send it directly to the GoRouter IP address:
Syntax: curl -vvv -H 'Host: APP-FQDN' GOROUTER-IP
Example: curl -vvv -H 'Host: test-app.apps.example.com' 192.0.2.35
* Trying 192.0.2.35... * TCP_NODELAY set * Connected to 192.0.2.35 (192.0.2.35) port 80 (#0) > GET / HTTP/1.1 > Host: test-app.apps.example.com > User-Agent: curl/7.54.0 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: text/html; charset=UTF-8 < Date: Wed, 29 Nov 2017 16:45:02 GMT < Server: Apache < Vary: Accept-Encoding < X-Vcap-Request-Id: 94733c88-565e-4ff1-7e08-f4a9a9d9b812 < Content-Length: 17 < * Connection #0 to host 192.0.2.35 left intact <p>Hello World</p>
- Now if we compare the differences in the HTTP responses:
- Request 1 returned HTTP 403 and Request 2 returned HTTP 200.
- Request 1 returned "<p>Oops... something went wrong!</p>" where as the Request 2 "<p>Hello World</p>"
- Request 1 reported the server as lighttpd and Request 2 reported the server as Apache
Lighttpd is not a standard part of Cloud Foundry so this would raise suspicions. The example here could be interpreted as a misconfigured proxy/load balancer which is forwarding the requests to a lighttpd server and the lighttpd server is expecting the requests to be authenticated.
This is just a very basic example of comparing a good and bad request. In a real situation the differences may be less obvious but the key is to look for anything which does not look normal in a Cloud Foundry environment.