Pivotal Knowledge Base

Follow

Understand and get better control of client subscriptions monitoring

Environment

Product Version
Pivotal GemFire All versions

Purpose

The purpose of this article is to describe what messages are logged for client subscriptions to monitor these and to handle issues. It will also describe what issues can arise with subscriptions and how to configure GemFire to avoid these. Lastly it describes how you can gain more control of the monitoring by implementing a BridgeObserver.

Description

Description of log messages

When a client with subscription-enabled="true" is started, messages like below will be logged in the GemFire client log. If subscription-redundancy is not set, there will be one of these; if it is set to 1, there will be two, etc. The Cache Client Updater Thread is the thread waiting for events from the server.

[info 2016/09/13 10:40:06.915 PDT client :64154 port 50166> tid=0x1a] Cache Client Updater Thread  on boglesbymac-2(server-2:831):64154 port 50166 (boglesbymac-2:50166) : ready to process messages.

If the server to which the Cache Client Updater Thread is connected crashes, then messages like this will be logged:

[info 2016/09/13 10:44:06.315 PDT client :56099 port 50148> tid=0x1b] Primary subscription endpoint boglesbymac-2:50148 crashed. Scheduling recovery.

[info 2016/09/13 10:44:06.316 PDT client  tid=0x34] SubscriptionManager redundancy satisfier - primary endpoint has been lost. Attempting to recover.

[info 2016/09/13 10:44:06.316 PDT client :56099 port 50148> tid=0x1b] Cache client updater for Queue on endpoint boglesbymac-2:50148 exiting. Scheduling recovery.

If another server is available, then the client will connect to it, and a message like this will be logged:

[info 2016/09/13 10:44:06.420 PDT client :15304 port 50173> tid=0x35] Cache Client Updater Thread  on boglesbymac-2(server-3:838):15304 port 50173 (boglesbymac-2:50173) : ready to process messages.

If no other server is available, then an error message like this will be logged:

[error 2016/09/13 10:45:29.351 PDT client  tid=0x34] Could not find any server to create primary client queue on. Number of excluded servers is 0 and exception is null.

Factors for inconsistent cache scenarios

Redundancy can take different roads depending on configuration:

  • If subscription-redundancy > 0, and the primary server crashes, then a redundant queue on the server will take over as primary, and the client will continue as before.
  • If subscription-redundancy==0, and the primary server crashes, then the client's queue of events is lost.

When the client reconnects to a new server, its interest registrations will be recovered, and its local cache will be cleared:

  • If the interest result policy is InterestResultPolicy.NONE, then client's cache will remain empty
  • if it is InterestResultPolicy.KEYS_VALUES, then its cache will be rebuilt from the server. So, the client's cache won't be inconsistent, but it may have missed some events (any events in its queue and any that occurred while it was disconnected).

Using a BridgeObserver

A ClientMembershipListener will tell the client when it connects to and disconnects from servers, but it doesn't have any notion of subscriptions. To see activity related to subscription servers, a BridgeObserver is needed. This is an internal class in the com.gemstone.gemfire.internal.cache package.

If a BridgeObserver is installed in the client, then there will be callbacks for: - before and after interest registration - after subscription server connection is lost - before and after redundant subscription server promotion to primary - after primary subscription server is recovered - before interest registration recovery

Attached is a BridgeObserver example that provides callbacks in the following cases:

If the redundant subscription server is lost, messages like this will be logged:

TestBridgeServerObserver: Lost redundant subscription server: boglesbymac-2:55481
TestBridgeServerObserver: About to recover interest

If the primary subscription server is lost, and there is a redundant subscription server, messages like this will be logged:

TestBridgeServerObserver: Lost primary subscription server: boglesbymac-2:55497
TestBridgeServerObserver: About to promote backup subscription server to primary
TestBridgeServerObserver: Promoted backup subscription server to primary: boglesbymac-2:55467
TestBridgeServerObserver: Recovered primary subscription server: boglesbymac-2:55467

If the primary subscription server is lost with subscription-redundancy="0", and there are other servers available, messages like this will be logged:

TestBridgeServerObserver: Lost primary subscription server: boglesbymac-2:55710
TestBridgeServerObserver: About to recover interest
TestBridgeServerObserver: Recovered primary subscription server: boglesbymac-2:55722

If the primary subscription server is lost, and there are no other servers available, a message like this will be logged:

TestBridgeServerObserver: Lost primary subscription server: boglesbymac-2:55467

Once a server is available, and the client connects to it, messages like this will be logged:

TestBridgeServerObserver: About to recover interest
TestBridgeServerObserver: Recovered primary subscription server: boglesbymac-2:55672

Register it before the client cache is created like:

Properties properties = new Properties();
properties.put("pool-name", "pool");
new TestBridgeServerObserver().init(properties);

It can also be registered as an initializer in the GemFire cache xml file like below, but there is potential to miss callbacks by doing it this way so that is not recommended:

<initializer>
    <class-name>TestBridgeServerObserver</class-name> 
    <parameter name="pool-name">
        <string>pool</string>
    </parameter>
</initializer>

Comments

Powered by Zendesk