Pivotal Knowledge Base

Follow

Initializing a Persistent Region Takes Longer than Usual

Environment 

  • Pivotal GemFire 7.0 through maintenance/minor releases
  • GemFire 8.2.9 and prior to 8.x
  • GemFire 9.4 and above 9.x

Symptom

When creating a persistent region with indexes via cache.xml, it can take longer than usual to initialize the region after recovering the data entries from the disk stores (more than an hour has been observed). You can see that it is taking a long time to initialize the region by examining the timestamps in the relevant log messages. For example, it takes more than 5 hours to initialize the ExampleRegion in the below sample:

[info 2018/03/26 15:36:11.123 UTC MyCacheServer1 <main> tid=0x1] Initializing region ExampleRegion
  :
[info 2018/03/26 20:42:32.548 UTC MyCacheServer1 <main> tid=0x1] Initialization of region ExampleRegion completed

Cause

Examining thread dumps, you will see a thread stack for the main thread like:

"main" #1 prio=5 os_prio=0 tid=0x000000001a452800 nid=0x17e2 runnable [0x00002b52eb240000]
   java.lang.Thread.State: RUNNABLE
    at com.gemstone.gemfire.cache.query.internal.index.MemoryIndexStore.getOldKey(MemoryIndexStore.java:246)
    at com.gemstone.gemfire.cache.query.internal.index.MemoryIndexStore.basicRemoveMapping(MemoryIndexStore.java:370)
    at com.gemstone.gemfire.cache.query.internal.index.MemoryIndexStore.removeMapping(MemoryIndexStore.java:270)
    at com.gemstone.gemfire.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.applyProjection(CompactRangeIndex.java:1682)
    at com.gemstone.gemfire.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1614)
    at com.gemstone.gemfire.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1624)
    at com.gemstone.gemfire.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1465)
    at com.gemstone.gemfire.cache.query.internal.index.CompactRangeIndex.removeMapping(CompactRangeIndex.java:159)
    at com.gemstone.gemfire.cache.query.internal.index.AbstractIndex.removeIndexMapping(AbstractIndex.java:500)
    at com.gemstone.gemfire.cache.query.internal.index.IndexManager.processAction(IndexManager.java:1133)
    at com.gemstone.gemfire.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:989)
    at com.gemstone.gemfire.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:963)
    at com.gemstone.gemfire.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:734)
    at com.gemstone.gemfire.internal.cache.AbstractRegionMap.destroyEntry(AbstractRegionMap.java:3271)
    at com.gemstone.gemfire.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1540)
    - locked <0x0000000c6449bf00> (a com.gemstone.gemfire.internal.cache.VersionedThinDiskRegionEntryHeapStringKey2)
    at com.gemstone.gemfire.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6900)
    at com.gemstone.gemfire.internal.cache.LocalRegion.destroyRecoveredEntry(LocalRegion.java:11210)
    at com.gemstone.gemfire.internal.cache.DiskRegion$1.handleRegionEntry(DiskRegion.java:304)
    - locked <0x0000000c6449bf00> (a com.gemstone.gemfire.internal.cache.VersionedThinDiskRegionEntryHeapStringKey2)
    at com.gemstone.gemfire.internal.cache.LocalRegion.foreachRegionEntry(LocalRegion.java:7844)
    at com.gemstone.gemfire.internal.cache.DiskRegion.destroyOldTomstones(DiskRegion.java:296)
    at com.gemstone.gemfire.internal.cache.DiskRegion.finishInitializeOwner(DiskRegion.java:270)
    at com.gemstone.gemfire.internal.cache.DistributedRegion.cleanUpDestroyedTokensAndMarkGIIComplete(DistributedRegion.java:1697)
    at com.gemstone.gemfire.internal.cache.DistributedRegion.getInitialImageAndRecovery(DistributedRegion.java:1476)
    at com.gemstone.gemfire.internal.cache.DistributedRegion.initialize(DistributedRegion.java:1209)
       :

This indicates that the main thread was repeatedly removing the indexes that were created for each old TombStone recovered from the disk store. Also, the TombStones were being removed one-by-one. This can be a time-consuming task.

Resolution

To resolve this, create the indexes for the problem region via a gfsh command or API call after starting the target members instead of creating it in the cache.xml. This will eliminate the time-consuming indexes removals from the region initialization phase (i.e., do not create indexes until the older TombStones have been removed, which occurs during the recovery process).

Additional Information

This issue has been addressed by GEM-1936/GEODE-4823 and will be fixed in the newer versions beginning with the GemFire 8.2.10 and GemFire 9.5 maintenance/minor releases.

 

Comments

Powered by Zendesk