Pivotal Knowledge Base

Follow

GemFire: Leaking P2P message reader threads

Environment

Product Version
Pivotal GemFire All
conserve-sockets false

Symptom

The number of P2P message reader threads can grow indefinitely over time for certain configurations. This can certainly be problematic for a variety of reasons related to resource consumption. This issue is not a concern for conserve-sockets=true environments.

Cause

For performance reasons, customers sometimes use various configuration settings to avoid hitting various limitations with respect to the number of threads or timing out of threads. While this can be beneficial, it can lead to a situation where we leak GemFire P2P message reader threads. The combination of settings to be avoided are discussed in this article. 

The properties to be discussed include settings for each of the following:

  • socket-lease-time=0
  • max-threads=0
  • conserve-socket=false
  • gemfire.IDLE_THREAD_TIMEOUT

RCA

This is the result of a product issue that continues to exist with GemFire.   There is no imminent resolution pending given the various workarounds that resolve this issue. Basically, when trying to avoid the costs of starting threads and tearing down connections or throttling the system, customers may use settings where we never timeout a thread and never set a limit on the maximum number of threads. This can result in a leak of P2P message reader threads, because the mechanisms in place to limit the number and clean them up have been overridden

Resolution

Any of the following options can be used to eliminate the leaking P2P threads: 

  1. Do not use socket-lease-time = 0.  The default is 60 seconds. If you feel that keeping sockets around longer than this is warranted, set this property to some higher value that will clean up the idle socket in time.  The maximum value is 10 minutes (600000 milliseconds)  
  2. Set max-threads to a non-zero value. Setting max-threads to 0 causes the server to use a new thread for every client connection. When used in conjunction with socket-lease-time=0, the result can be many threads and sockets that does not get cleaned up properly when no longer in use. Setting max-threads greater than 0 will limit the number of threads that can service client connections, thus reducing the leaking threads.
  3. The system property gemfire.IDLE_THREAD_TIMEOUT controls how long an idle thread will wait, in milliseconds, before it is removed from its thread pool. The default time is 30 minutes.

Additional Information

Please refer to our Product documentation related to the above settings for additional information on socket and thread usage. 

Comments

Powered by Zendesk