Pivotal Knowledge Base

Follow

NullPointerException is thrown if executing multiple "gfsh query" commands at the same time

Environment 

Pivotal GemFire: 8.x, 9.x

Symptom

When executing multiple "gfsh query" commands at the same time, you may observe that the "gfsh query" commands are failing with the message, "Reason: null," and find a corresponding NullPointerException with the following stack trace in the locator logs

Error Message: 

java.lang.NullPointerException
        at com.gemstone.gemfire.management.internal.cli.functions.DataCommandFunction$SelectQuitStep.exec(DataCommandFunction.java:962)
        at com.gemstone.gemfire.management.internal.cli.multistep.CLIMultiStepHelper.chooseStep(CLIMultiStepHelper.java:232)
        at com.gemstone.gemfire.management.internal.cli.commands.DataCommands.query(DataCommands.java:1176)
        at sun.reflect.GeneratedMethodAccessor157.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.gemstone.gemfire.management.internal.cli.util.spring.ReflectionUtils.invokeMethod(ReflectionUtils.java:51)
        at com.gemstone.gemfire.management.internal.cli.remote.RemoteExecutionStrategy.execute(RemoteExecutionStrategy.java:105)
        at com.gemstone.gemfire.management.internal.cli.remote.CommandProcessor.executeCommand(CommandProcessor.java:97)
        at com.gemstone.gemfire.management.internal.cli.remote.CommandStatementImpl.process(CommandStatementImpl.java:59)
        at com.gemstone.gemfire.management.internal.cli.remote.MemberCommandService.processCommand(MemberCommandService.java:47)
        at com.gemstone.gemfire.management.internal.beans.MemberMBeanBridge.processCommand(MemberMBeanBridge.java:1737)
        at com.gemstone.gemfire.management.internal.beans.MemberMBean.processCommand(MemberMBean.java:400)
        at com.gemstone.gemfire.management.internal.beans.MemberMBean.processCommand(MemberMBean.java:393)
        at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
        at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)
        at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)
        at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)
        at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
        at sun.reflect.GeneratedMethodAccessor133.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
        at sun.rmi.transport.Transport$1.run(Transport.java:200)
        at sun.rmi.transport.Transport$1.run(Transport.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Cause

Currently (As of Jun 2017), the temporary result sets for "gfsh query" commands are stored as a static variable that can be shared between each worker thread on a member instance. Suppose that thread A tries to store a temporary result set into this static variable from a "gfsh query" command while thread B tries to clean up the same static variable after executing a "gfsh query" command on the same member. In such a situation, thread A could store a non-null result set into the static variable which thread B would immediately set to null to clean up its own temporary result set. At this point, if thread A touches the static variable for additional processing, it will throw a NullPointerException.

Resolution

As a workaround for the affected versions, do not execute "gfsh query" from multiple gfsh clients at the same time in order to prevent the above race condition.

Additional Information

This issue has been addressed via bug ID GEM-1435 and will be fixed with the future release of Pivotal GemFire.

 

Comments

Powered by Zendesk