Pivotal Knowledge Base

Follow

User Threads remain as Zombies even after Server was Shut Down

Environment

 Product  Version
 Pivotal GemFire  All Supported Versions
 OS  All Supported OS

Symptom

Some user threads/process which are triggered from a cacheserver, such as a cachelistener or a SpringContextBootstrappingInitializer by the ThreadPoolTaskExecutor/ThreadPoolTaskScheduler, may remain in the JVM as a zombie process even after the cacheserver process was shut down or was forced out of the Gemfire cluster by some member-timeout mechanism.

Here is a thread dump example of a zombie process:

Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode):

"scheduler-97" prio=10 tid=0x0000000001ceb000 nid=0x2bf5 waiting on condition [0x00007f36efd2f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for   (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- None
......
"ThreadPoolTaskExecutor-3" prio=10 tid=0x00007f36f9093800 nid=0x75d8 waiting on condition [0x00007f36f1c70000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for   (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- None
......

Cause

Non-daemon threads running the user process, which were triggered by the GemFire cacheserver, may not exit from the JVM since they are non-daemon threads.

Resolution

Use a daemon thread instead of a non-daemon thread when creating a user thread/process by the ThreadPoolTaskExecutor/ThreadPoolTaskScheduler.

For example:

<bean id="scheduler" class="org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler"> 
<property name="poolSize" value="100"/>
......
<property name="daemon" value="true"/>
</bean>

<bean id="ThreadPoolTaskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="maxPoolSize" value="20" />
<property name="queueCapacity" value="10000" />
......
<property name="daemon" value="true"/>
</bean>

 

Comments

Powered by Zendesk