Pivotal Knowledge Base

Follow

GemFire Error: "New Cache Time Offset Calculated is off more than "XXX" ms from Earlier Offset

Environment

 Product  Version
 Pivotal GemFire  7.x - 8.x
 OS  All Supported OS

Symptom

The following warning entry is observed in the server logs:

[warning 2016/10/31 11:57:17.496 JST  tid=0x1e] New cache time offset calculated is off more than 212 ms from earlier offset.

Cause

GemFire internally has a distributed clock synchronization implementation. When membership views change, this distributed clock synchronization sends a request for the current MilliSecond clock to all members at a configurable interval. It computes the average round-trip time and the average clock value, throwing out samples outside the standard deviation. The result is used to compute and send clock offsets to each member.

Typically, this warning message is harmless and it can be ignored. But it might indicate a CPU overload or high network latency between the members if the offset is higher than 1000 MilliSeconds.

The current GemFire distributed clock synchronization, in Gemfire6.x to 8.x, is implemented based on a Berkeley Algorithm for clock synchronization. GemFire 9.x will replace this implementation with a new clock synchronization service and we can expect to see this message less often. 

Resolution

1. You can dismiss the error message if it is only in MilliSecond unit. If the offset is in seconds, then check the GemFire statistics for network latency or high CPU utilization.

2. GemFire 9.x will implement a new clock synchronization service replacing the current GemFire distributed clock synchronization, which is expected to decrease the occurrence of the above message.

 

Comments

Powered by Zendesk