Pivotal Knowledge Base

Follow

How To Collect Basic Information For GemFire Issues?

Environment

Product Version
Pivotal GemFire 6.6.x - 8.2.x
OS All Supported OS

Purpose

This document contains guidelines for collecting basic information, such as logs, statistics files, thread dumps, and heap dumps, for GemFire related issues. Besides providing the artifacts it is important to provide a timeline or overview of the issue with details on impact and actions taken.

Please note that in GemFire 8.2.x you can use the provided Log Collection Utility to make it easier to collect artifacts from a GemFire cluster as described here.

What to collect when

  • For all GemFire issues, support will need logs and statistics from all cluster members covering the time period when the issue occurred.
  • For issues where members are hung (any time an unscheduled "restart" becomes necessary), support will also need thread dumps from, at a minimum, the members that appear unresponsive. It is important than more than one thread dump is taken on each host.
  • For tuning issues, support will also require GC logs.
  • For out-of-memory issues and memory leaks, a heap dump will be required. (If this is not feasible, a heap histogram is better than nothing.)

Procedure

1. Logs:

  1. Locator, CacheServer, and GemFire Agent logs

    Copy logs from the location configured by the log-file property in the gemfire.properties file or given as a parameter in your startup script. For example (from the gemfire.properties file):

    log-level=fine
    log-file=log/cacheserver1.log

    Make sure to provide complete logs, including those that cover the header and startup information. This information is valuable when investigating an issue.

  2. Java Client logs

    Copy any client logs from the location defined in the Java client code or gemfire.properties file. For example (in code):

    ClientCache cache = new ClientCacheFactory()
      .set("name", "CqClient")
      .set("cache-xml-file", "xml/CqClient.xml")
      .set("log-level", "fine")
      .set("log-file", "cqclient.log")
      .create();
  3. Native Client logs

    Copy logs from the path defined in the log-file property of the gfcpp.properties file or defined in native code. For example (in gfcpp.properties):

    log-level=fine
    log-file=log/nativeclient1.log
  4. Databrowser logs

    Copy the logs from the path defined in the Databrowser "Options" setting menu:

    Options-->Preferences-->Miscellaneous-->Log file directory

    For example:

    C:\Users\userxxxx\GemStone\GemFire\DataBrowser\logs\data-browser_7916.log.0
  5. GFMon logs

    Copy the logs from the path defined in the GFMon "Preferences" setting dialog:

    Preferences-->General-->Log file directory

    For example:

    C:\Users\userxxxx\GemStone\GemFire\GFMon\logs\2013-12-30-22-26-31-849_8366723371373487377_gfmon.log
  6. Security logs

    Copy the logs from the path defined by the security-log-file property of the gemfire.properties or gfsecurity.properties file. For example (gemfire.properties):

    security-log-file=log/locatorsecurity.log
  7. GFSH logs

    By default, gfsh session logging is disabled. To enable gfsh logging, you must set the Java system property -Dgfsh.log-level=<desired_log_level> where desired_log level is one of the following values: severe, warning, info, config, fine, finer, finest. For example, in Linux:

    $ export JAVA_ARGS=-Dgfsh.log-level=info

    Then, start gfsh.

    Copy any logs from the directory in which the gfsh command was run. For example, if the gfsh command was run from /home/user1/GemWorkdir1, the gfsh log would be in a file like:

    /home/user1/GemWorkdir1/gfsh-2013-12-31_17-36-25.log
  8. GC Logs

    GC logging is enabled with startup parameters added to the JVM. The following parameters should be added when enabling GC logging for GemFire GC tuning issues:

    -XX:+PrintGC     (or the alias: -verbose:gc)
    -XX:+PrintGCDetails
    -XX:+PrintGCTimeStamps
    -Xloggc:<file>

    Use Xloggc to specify the path and file name of the GC log file. Default is standard out.

    The overhead of GC logging is usually rather small so it is generally recommended to have it enabled. However, it is good to know that it is not needed to decide on this at JVM startup. The JVM has a category of flags called "manageable". For manageable flags, it is possible to change their values at run time. All the GC logging flags that start with "PrintGC" belong to the "manageable" category. Thus, it is possible to activate or deactivate GC logging for a running JVM.

    Manageable flags can be set with the use of a JMX client calling the setVMOption operation of the HotSpotDiagnostic MXBean or using the jinfo tool shipped with the JDK.

2. Statistics files:

  1. Locator Statistics files, CacheServer Statistics files

    Copy any stats files from the location defined by the statistic-archive-file property of gemfire.properties. For example (gemfire.properties):

    statistic-sampling-enabled=true
    statistic-archive-file=myStatisticsArchiveFile.gfs
    enable-time-statistics=false

    The statistic-sample-rate can be changed from the default sample rate of 1000 milliseconds, but this shouldn't be needed as impact is very small.

    Note that time statistics should only be enabled in dev and QA environments and not in production as this setting has a relatively large impact on GemFire performance.

    To setup rolling of statistics files use the following parameters:

    archive-disk-space-limit=1000
    archive-file-size-limit=100

    This will makes gfs files roll when they reach 100MB and keep the last 10 files.

  2. Java Client Statistics files

    Copy any client-side stats files from the path defined in code or in the gemfire.properties file. For example (in code):

    ClientCache cache = new ClientCacheFactory()
      .set("name", "CqClient")
      .set("cache-xml-file", "xml/CqClient.xml")
      .set("log-level", "fine")
      .set("log-file", "cqclient.log")
      .set("statistic-archive-file", "myClientStats.gfs"
      .set("statistic-sampling-enabled", "true")
      .create();
  3. Native Client Statistics files

    Copy any stats files from the location defined in the statistic-archive-file property of the gfcpp.properties file or defined in native code. For example (gfcpp.properties):

    statistic-sampling-enabled=true
    statistic-archive-file=myClientStats.gfs

    For more about logging information, you can also refer the following document: http://pubs.vmware.com/vfabric53/topic/com.vmware.vfabric.gemfire.7.0/managing/statistics/setting_up...

3. Thread dumps

For some issues, such as hung systems or performance issues, thread dumps from the server or client are essential to analyzing the issue. It is very important that multiple thread dumps are taken periodically (i.e. every few seconds) over a period of time.

Thread dumps can be taken using the following procedure:

  • Step 1. Find out the relevant GemFire process id, i.e:
    $ jps -l
    7904 sun.tools.jps.Jps
    5388 sample.JClient
    
  • Step 2. Generate the thread dump(s).

    On Solaris, Linux, and other Unix platforms, sending a SIGQUIT signal to the GemFire Java process will generate a thread dump(s), i.e.:

     kill -QUIT <pid>

    In Windows, you can press the CTRL-Break keys in the command shell where the GemFire Java process was started.

    Alternatively, these tools can be used to generate the thread dump:

    1. jstack command: jstack <pid>
    2. Java VisualVM (jvisualvm)
    3. jconsole

4. Heap dump

For investigating issues, such as an Out-of-Memory issue or Memory Leak, a heap dump will help track down underlying issues.

Generate the heap dump of gemfire process using the following:

  • Step 1. First, identify the specific gemfire process id using a command like jps (as in the procedure for getting thread dumps).
  • Step 2. Generate the heap dump.
    1. Using jmap command
      [JDK_INSTALLATION]/bin/jmap -dump:live,format=b,file=heap.dump.out <pid>>
    2. Using Java VisualVM (jvisualvm):
      1. run [JDK_INSTALLATION]/bin/jvisualvm,
      2. select the target process and select, [Application] menu-->[Heap Dump], and select the generated heap dump, then choose [Save As] "to local disk."
    3. Getting a Heap Dump Automatically on an "Out Of Memory" error:

      Add the following jvm parameter to the java process before it starts.

      -XX:+HeapDumpOnOutOfMemoryError<
      -XX:HeapDumpPath

      For example (on Windows):

      JAVA_OPTS=%JAVA_OPTS% "-XX:+HeapDumpOnOutOfMemoryError" "-XX:HeapDumpPath=C:\TEMP"

 

Additional Information

Internal Comments 

Section

All Help & Support > Pivotal GemFire Knowledge Base > About Getting Support

Notes

None

 

Comments

Powered by Zendesk