Pivotal Knowledge Base

Follow

Pivotal Cloud Cache smoke-test fails when load balancer disallows HTTP

Environment

Pivotal Cloud Cache service: All versions

Pivotal Cloud Foundry® (PCF)

Symptom

Pivotal cloud cache smoke-test fails with a connection error.

PCF environment uses a load balancer which does not allow HTTP traffic.

Error Message:

(1) Executing - connect --use-http --url=http://cloudcache-4a952638-ab7f-4422-85a5-cfdccad5ef78.<system domain>/gemfire/v1 --user=cluster_operator --password=*****
Connection Error occurred.
------------------------------
ï Failure [1224.985 seconds]
Gemfire Broker smoke tests [It] checks that the service is working 
/var/vcap/packages/smoke-tests/src/cloudcache-broker-tests/smoke-tests/smoke_test.go:84
Expected error:
<*errors.errorString | 0xc420194cb0>: {
s: "Connection Error occurred.\n\n: GFSH exited non-zero: 1",
}
Connection Error occurred.

Cause 

Pivotal cloud-cache smoke-test uses HTTP for smoke-test and will encounter problems if load balancer does not allow HTTP traffic.

Resolution

Currently, Pivotal cloud-cache tile does not support configuring HTTPs for a smoke-test errand. This will be fixed in a future release of the cloud-cache tile. This should only impact the cloud-cache smoke-test and not other functionality of cloud-cache tile.

For apps bound to cloud-cache, the apps will talk to the locators directly using their IP's and therefore the traffic will not go through the load balancers. So applications will not be affected by failure that smoke-test encounters.

If using Gemfire shell (gfsh) connection can be established to cloud-cache by executing the following steps:

  1. On a machine with gfsh installed, create a truststore that contains the certificate to trust
    keytool -import -alias <env>-ssl -file <path-to-ERT-cert> -keystore <env>.truststore
  2. You will be prompted to enter a keystore password. Create a password and remember it!

  3. You will then be prompted with the certificate details. Type yes to trust the certificate. See sample output below
    Owner: CN=*.moonstone.pcf-gemfire.com, OU=Cloud Foundry, O=Pivotal, L=New York, ST=New York, C=US
    Issuer: CN=*.moonstone.pcf-gemfire.com, OU=Cloud Foundry, O=Pivotal, L=New York, ST=New York, C=US Serial number: bd84912717b5b665 Valid from: Sat Jul 29 09:18:43 EDT 2017 until: Mon Apr 07 09:18:43 EDT 2031 Certificate fingerprints: MD5: B9:17:B1:C0:6C:0A:F7:A3:56:51:6D:67:F8:3E:94:35 SHA1: BF:DA:23:03:17:C0:DF:37:D9:6F:47:05:05:00:44:6B:24:A1:3D:77 SHA256: F6:F4:4E:B8:FF:8F:72:92:0A:6D:55:6E:59:54:83:30:76:49:80:92:52:3D:91:4D:61:1C:A1:29:D3:BD:56:57 Signature algorithm name: SHA256withRSA Version: 3 Extensions: #1: ObjectId: 2.5.29.19 Criticality=true BasicConstraints:[ CA:true PathLen:0 ] #2: ObjectId: 2.5.29.17 Criticality=false SubjectAlternativeName [ DNSName: *.sys.moonstone.pcf-gemfire.com DNSName: *.apps.moonstone.pcf-gemfire.com DNSName: *.uaa.sys.moonstone.pcf-gemfire.com DNSName: *.login.sys.moonstone.pcf-gemfire.com DNSName: *.moonstone.pcf-gemfire.com DNSName: *.ws.moonstone.pcf-gemfire.com ] Trust this certificate? [no]:
  4. Before starting gfsh, export the following JAVA_ARGS
    export JAVA_ARGS="-Djavax.net.ssl.trustStore=<path to generated truststore>"
  5. Open the gfsh console and connect via HTTPs
    gfsh>connect --use-http --url=https://cloudcache-be4ab03f-54cc-4644-bdbf-b4722c23eb63.moonstone.pcf-gemfire.com/gemfire/v1 --user=cluster_operator --password=<password>

    Successfully connected to: GemFire Manager HTTP service @ https://cloudcache-be4ab03f-54cc-4644-bdbf-b4722c23eb63.moonstone.pcf-gemfire.com/gemfire/v1 

Additional Information

Full error message:

(1) Executing - connect --use-http --url=http://cloudcache-4a952638-ab7f-4422-85a5-cfdccad5ef78.<system domain>/gemfire/v1 --user=cluster_operator --password=*****

Connection Error occurred.

------------------------------
ï Failure [1224.985 seconds]
Gemfire Broker smoke tests [It] checks that the service is working 
/var/vcap/packages/smoke-tests/src/cloudcache-broker-tests/smoke-tests/smoke_test.go:84 [91mExpected error:
<*errors.errorString | 0xc420194cb0>: {
s: "Connection Error occurred.\n\n: GFSH exited non-zero: 1",
}
Connection Error occurred.

: GFSH exited non-zero: 1
not to have occurred /var/vcap/packages/smoke-tests/src/cloudcache-broker-tests/smoke-tests/smoke_test.go:68
------------------------------
Summarizing 1 Failure: [91m[Fail] Gemfire Broker smoke tests [It] checks that the service is working 
/var/vcap/packages/smoke-tests/src/cloudcache-broker-tests/smoke-tests/smoke_test.go:68 [1mRan 1 of 1 Specs in 1375.102 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped --- FAIL: TestCloudCacheSmokeTests (1375.10s)
FAIL Ginkgo ran 1 suite in 23m6.684276238s
Test Suite Failed /var/vcap/packages/smoke-tests/src/cloudcache-broker-tests/smoke-tests/smoke_test.go:68
- 

This issue will be fixed in the future releases of Pivotal Cloud Cache.

Comments

Powered by Zendesk