Pivotal Knowledge Base

Follow

Slow Startup When Using Hash Indexes

Environment

Pivotal GemFire: 7.0.X, 8.2.x, 9.0.x, 9.1.X

Symptom

Startup time for members hosting a persistent region having a hash index defined is extremely slow.

The statistic RegionStats-indexName#IndexInitializationInProgress shows several threads running for a long time and all of those threads have a stack trace similar to the ones below.

"Recovery thread for bucket _B__RegionName_bucketNumber" #446 prio=5 os_prio=0 tid=0x00007f25890a5800 nid=0x1d76 runnable [0x00007f15e8224000]
	java.lang.Thread.State: RUNNABLE
	at com.gemstone.gemfire.cache.query.internal.index.HashIndexSet.insertionIndex(HashIndexSet.java:507)
	at com.gemstone.gemfire.cache.query.internal.index.HashIndexSet.rehash(HashIndexSet.java:602)
	at com.gemstone.gemfire.cache.query.internal.index.HashIndexSet.preInsertHook(HashIndexSet.java:1050)
	at com.gemstone.gemfire.cache.query.internal.index.HashIndexSet.add(HashIndexSet.java:397)
	- locked <0x00007f18403b5d50> (a com.gemstone.gemfire.cache.query.internal.index.HashIndexSet)
	at com.gemstone.gemfire.cache.query.internal.index.HashIndex.basicAddMapping(HashIndex.java:228)

"Recovery thread for bucket _B__RegionName_bucketNumber" #451 prio=5 os_prio=0 tid=0x00007f25890af800 nid=0x1d7b runnable [0x00007f15e8107000]
	java.lang.Thread.State: RUNNABLE
	at com.gemstone.gemfire.cache.query.internal.index.HashIndexSet.insertionIndex(HashIndexSet.java:507)
	at com.gemstone.gemfire.cache.query.internal.index.HashIndexSet.add(HashIndexSet.java:398)
	- locked <0x00007f184805e110> (a com.gemstone.gemfire.cache.query.internal.index.HashIndexSet)
	at com.gemstone.gemfire.cache.query.internal.index.HashIndex.basicAddMapping(HashIndex.java:228)
	at com.gemstone.gemfire.cache.query.internal.index.HashIndex.access$200(HashIndex.java:86)
	at com.gemstone.gemfire.cache.query.internal.index.HashIndex$IMQEvaluator.applyProjectionForIndexInit(HashIndex.java:1280)

Cause

GemFire uses an open-addressed hash table to store the index contents, which is backed up by an implementation of the ConcurrentHashMap Java classWhenever more capacity is needed, the internal structure must be re-sized and every single entry should be processed to compute its hashCode again, all of these operations are synchronized (contention) and, thus, this is the reason for the slow performance during startup.

Resolution

Avoid the usage of indexes of type hash and use functional/range indexes instead.

We have seen several problems with this type of index, including extremely slow performance during startup, and a decision has been made to entirely remove it from the product in coming releases.

The hash index was primarily created to save memory and it does not get hash table level performance; the functional/range index itself, on the other hand, will always perform better than the hash index in updates and search operations.

Comments

Powered by Zendesk