Post

2 followers Follow
0
Avatar

ResultCollector.execute thows exception: Pool unexpected IOException connection=Pooled Connection to 10.27.73.202:10000: Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts

Hi,

I was getting "Pool unexpected IOException connection=Pooled Connection to 10.27.73.202:10000: Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts" exception when my gemfire client was executing one of gemfire server functions, however the other functions were executing successfully. Looking at the error message itself it is hard to understand what kind of IOException is that.

Only when enabled fine level logging I saw a message indicating jar mismatch between server and client:
[debug 2015/11/02 12:03:32.227 AMT <main> tid=0x1] OpExecutor.handleException on Connection to 10.27.73.202:10000
java.io.InvalidClassException: com........; incompatible types for field processed
at java.io.ObjectStreamClass.matchFields(ObjectStreamClass.java:2255)
at java.io.ObjectStreamClass.getReflector(ObjectStreamClass.java:2149)
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:657)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2966)
at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3210)
at com.gemstone.gemfire.DataSerializer.readArrayList(DataSerializer.java:2230)
at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2860)
at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3210)
at com.gemstone.gemfire.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:110)
at com.gemstone.gemfire.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:77)
at com.gemstone.gemfire.internal.cache.tier.sockets.CacheServerHelper.deserialize(CacheServerHelper.java:55)
at com.gemstone.gemfire.internal.cache.tier.sockets.Part.getObject(Part.java:233)
at com.gemstone.gemfire.internal.cache.tier.sockets.Part.getObject(Part.java:238)
at com.gemstone.gemfire.cache.client.internal.ExecuteFunctionOp$ExecuteFunctionOpImpl.processResponse(ExecuteFunctionOp.java:529)
at com.gemstone.gemfire.cache.client.internal.AbstractOp.processResponse(AbstractOp.java:215)
at com.gemstone.gemfire.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:153)
at com.gemstone.gemfire.cache.client.internal.AbstractOp.attempt(AbstractOp.java:369)
at com.gemstone.gemfire.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:267)
at com.gemstone.gemfire.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:319)
at com.gemstone.gemfire.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:930)
at com.gemstone.gemfire.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:158)
at com.gemstone.gemfire.cache.client.internal.PoolImpl.execute(PoolImpl.java:716)
at com.gemstone.gemfire.cache.client.internal.ExecuteFunctionOp.execute(ExecuteFunctionOp.java:180)
at com.gemstone.gemfire.internal.cache.execute.ServerFunctionExecutor.executeOnServer(ServerFunctionExecutor.java:183)
at com.gemstone.gemfire.internal.cache.execute.ServerFunctionExecutor.executeFunction(ServerFunctionExecutor.java:99)
at com.gemstone.gemfire.internal.cache.execute.ServerFunctionExecutor.execute(ServerFunctionExecutor.java:326)

The message is printed with debug level, I cannot even see it when the log level is config, neither the exception object that is thrown has an indication of invalid class exception.

The question I have is how to retrieve that message, so that I present a more reasonable message or at least understand that this is due to jar mismatch issue but not due to some other IOException?

Regards,
Hovhannes

Hovhannes Antonyan

Please sign in to leave a comment.

4 comments

0
Avatar

I can reproduce this issue. The function doesn't log the actual exception and keeps retrying forever. This looks like a bug. I'll verify and let you know.

Barry Oglesby 0 votes
0
Avatar

In my case it doesn't retry forever, it fails once it exceeds connection retry attempts and then it throws an exception. However in the exception message there is no word about "java.io.InvalidClassException: com........; incompatible types for field processed". It just says "Pool unexpected IOException connection=Pooled Connection to 10.27.73.202:10000: Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts"

Hovhannes Antonyan 0 votes
0
Avatar

I filed GEM-118 for this issue. This will probably be fixed for 9.0 unless you need something before then. What version are you running?

Barry Oglesby 0 votes