Pivotal Knowledge Base

Follow

Troubleshooting heap memory issues

Applies to

GemFire 6 and later

Purpose

The purpose of this article is to show some common ways to trouble shoot heap memory issues.

Description

One way to know that there is a heap memory issue is when an OutOfMemoryError with the message 'Java heap space' is thrown by GemFire. Many other error messages may accompany an OutOfMemoryError, but any error that occurs after the OutOfMemoryError can usually be ignored because it is likely a symptom of the memory problem. The error must contain the 'Java heap space' message and not the 'unable to create new native thread' message. See the article Trouble shooting Native Memory Issue for details on that issue. An example is shown below.

[severe 2009/02/23 19:38:48.293 UTC <Gateway Event Processor> tid=0xbd] Uncaught exception in thread <Gateway Processor>
java.lang.OutOfMemoryError: Java heap space

Another way to determine if there is a heap memory issue is to use either the gemfire stats command or VSD to display free, used and maximum heap memory values contained in a given GemFire statistics archive. There are several categories that contain the used and maximum heap memory values. The most useful categories are: VMMemoryUsageStats-vmHeapMemoryStats VMMemoryPoolStats-CMSOldGen-Heapmemory

Trouble shooting tips

Use either VSD or the gemfire stats command to investigate the memory usage recorded in the statistics files. In the following examples gemfire stats is used. Use the gemfire stats command to display the VMMemoryUsageStats-vmHeapMemoryStats maxMemory and usedMemory values, the VMMemoryPoolStats-XXOldGen-Heapmemory maxMemory and usedMemory values and the VMStats freeMemory value contained in a given GemFire statistics archive as shown below.

The gemfire stats command below shows the VMMemoryUsageStats-vmHeapMemoryStats maxMemory and usedMemory values in the stats.gfs archive.

$ gemfire stats vmHeapMemoryStats.maxMemory -archive=stats.gfs
[info] Found 1 match for "vmHeapMemoryStats:VMMemoryUsageStats.maxMemory"
vmHeapMemoryStats, 7980, VMMemoryUsageStats: "2009/02/23 16:49:50.157 UTC" samples=5511
maxMemory bytes: samples=5511 min=28633137152 max=30870077440 average=30743435202.68 stddev=517011962.91

$ gemfire stats vmHeapMemoryStats.usedMemory -archive=stats.gfs
[info] Found 1 match for "vmHeapMemoryStats:VMMemoryUsageStats.usedMemory"
vmHeapMemoryStats, 7980, VMMemoryUsageStats: "2009/02/23 16:49:50.157 UTC" samples=5511
usedMemory bytes: samples=5511 min=352716120 max=25037825520 average=6330675097.48 stddev=3523402783.98

The gemfire stats command below shows the VMMemoryPoolStats-XXOldGen-Heapmemory maxMemory and usedMemory values in the stats.gfs archive. Notice in this case that the max in both cases is the same. This demonstrates an out of memory condition.

$ gemfire stats "PS Old Gen-Heap memory.currentUsedMemory" -archive=stats.gfs
[info] Found 1 match for "PS Old Gen-Heap memory.currentUsedMemory"
PS Old Gen-Heap memory, 7980, VMMemoryPoolStats: "2009/02/23 16:49:50.157 UTC" samples=5511
currentUsedMemory bytes: samples=5511 min=1088016 max=21474836480 average=954817373.72 stddev=3840947896.44
$ gemfire stats "PS Old Gen-Heap memory.currentMaxMemory" -archive=stats.gfs
[info] Found 1 match for "PS Old Gen-Heap memory.currentMaxMemory"
PS Old Gen-Heap memory, 7980, VMMemoryPoolStats: "2009/02/23 16:49:50.157 UTC" samples=5511
currentMaxMemory bytes: samples=5511 min=21474836480 max=21474836480 average=21474836480 stddev=0

The gemfire stats command below shows the VMStats freeMemory value in the stats.gfs archive.

$ gemfire stats vmStats.freeMemory -archive=stats.gfs
[info] Found 1 match for ":VMStats.freeMemory"
vmStats, 7980, VMStats: "2009/02/23 16:49:50.157 UTC" samples=5511
freeMemory bytes: samples=5511 min=3595311632 max=30517361320 average=24403029136.91 stddev=4009011946.31

Solution

There are several ways to eliminate a heap memory issue:

  • Increase the maximum amount of heap available to the VM using the -Xmx VM argument
  • Use partitioning to distribute the data over additional machines
  • Overflow or expire the region data to reduce the heap memory footprint of the regions

Comments

Powered by Zendesk