Pivotal Knowledge Base

Follow

Error connecting to the WebSocket (WSS) via F5 Load Balancer

Environment

Product Version
Pivotal Cloud Foundry® (PCF) All version
F5/BIGIP Firewall 11.4.0 and older

Symptom

This issue manifests itself when trying to connect to a Loggregator WSS endpoint and the connection fails. You can try accessing the WSS endpoint in many different ways including via JMX Bridge, cf logs, or when running an errand. You may also experience the OpenTSDB Firehose nozzle VM appearing to go up and down in JMX Bridge Tiles Status tab.   

Errors: JMX Bridge/OpenTSDB Firehose nozzle

You may receive error messages such as the following:

2016/08/01 13:02:05 Error while reading from the firehose: websocket: close 1008 Client did not respond to ping before keep-alive timeout expired.
2016/08/01 16:02:05 Disconnected because nozzle couldn't keep up. Please try scaling up the nozzle.
2016/08/01 16:02:05 Closing connection with traffic controller due to websocket: close 1008 Client did not respond to ping before keep-alive timeout expired.
2016/08/01 16:02:05 OpenTSDB Firehose Nozzle shutting down...
2016/08/01 16:02:05 Starting OpenTSDB Firehose Nozzle...
2016/08/01 16:02:05 Error while reading from the firehose: websocket: close 1008 Client did not respond to ping before keep-alive timeout expired.
2016/08/01 16:02:05 Disconnected because nozzle couldn't keep up. Please try scaling up the nozzle.
2016/08/01 16:02:05 Closing connection with traffic controller due to websocket: close 1008 Client did not respond to ping before keep-alive timeout expired.
2016/08/01 16:02:05 OpenTSDB Firehose Nozzle shutting down...
2016/08/01 16:02:06 Starting OpenTSDB Firehose Nozzle...

Error: cf logs

The following error is received when trying to connect via the command line using cf logs.

cf logs <App_Name>

     websocket: close 1008 (policy violation): Client did not respond to ping before keep-alive timeout expired.                         

Error: Errand

The following error is received when running errands.

websocket: close 1008 (policy violation): Client did not respond to ping before keep-alive timeout expired.
panic: sync: negative WaitGroup counter
goroutine 412 [running]:

Cause 

The Loggregator API endpoint must be accessible via a WebSocket (WSS). This problem is caused by an issue with the firmware on your F5/BIGIP dropping WSS packets. This issue applies to versions of F5 Firmware prior to version 11.4.0. Please check if this issue is applicable to the version of Firmware you are currently running on your F5.

Resolution

If the issues and versions of the software outlined above are applicable to you, please apply the following:

Workaround:

1. Use an iRule to selectively disable the HTTP profile for WebSocket traffic, as outlined here [2]

2. https://support.f5.com/kb/en-us/solutions/public/14000/800/sol14814.html

Note: In the link above several solutions are given, the iRule change is confirmed to work with PCF. 

Additional Information

3. F5 /BIG-IP support for the WebSocket protocol [3] is available here

 

 

 

 

 

 

 

Comments

Powered by Zendesk