Pivotal Knowledge Base

Follow

How to Collect Basic Information for GemFireXD Issues?

Environment

Product Version
GemFireXD 1.0.x to 1.4.x

Purpose

This document contains guidelines for collecting basic information such as logs, statistics files, thread dumps, and heap dumps for GemFireXD issues.

Resolution

  1. Logs of Locator and GemfireXD Server

    Copy logs from the location defined in the log-file property of gemfirexd.properties file or given as a parameter of your startup script at the time of the issue.

    For example (from the gemfirexd.properties file):

    log-level=fine
    log-file=log/node1.log
    

    For additional details about GemFireXD logging, refer to the User's Guide:
    http://gemfirexd.docs.pivotal.io/latest/userguide/index.html...

  2. Statistics files of Locator and GemFireXD Server

    Copy any stats files from the location defined by the statistic-archive-file property of the gemfirexd.properties file.

    For example (from the gemfirexd.properties file):

    statistic-sampling-enabled=true
    statistic-sample-rate=2000
    statistic-archive-file=myStatisticsArchiveFile.gfs
    enable-time-statistics=true
    

    For more about GemFireXD stats collection, refer the User's Guide:
    http://gemfirexd.docs.pivotal.io/latest/userguide/index.html...

  3. Thread dumps

    For some issues, such as hung systems or performance issues, thread dumps from the server or client are essential for root cause analysis. Best practice recommends that multiple thread dumps are taken periodically (i.e. every few seconds) over a period of time.

    Thread dumps can be taken using the following procedure:

    Step 1. Find out the relevant GemFireXD process id, i.e:

    $ jps -l
    7904 sun.tools.jps.Jps
    5388 sample.JClient
    

    Step 2. Generate the thread dump(s).

    On Solaris, Linux, and other Unix platforms, sending a SIGQUIT signal to the GemFireXD Java process will generate a thread dump(s), i.e.:

    kill -QUIT <pid>

    In Windows, you can press the CTRL-Break keys in the command shell where the GemFireXD Java process was started.

    Alternatively, these tools can be used to generate the thread dump:

    1. jstack command: jstack <pid>
    2. Java VisualVM (jvisualvm)
    3. jconsole

    GemFireXD also provides the following internal command to collect the thread dumps for the entire distributed system.

    Usage: gfxd print-stacks [-all-threads][<filename>] [-J-D<vmprop>=<prop-value>]
    [-mcast-port=<port> (default 10334)]
    [-mcast-address=<address> (default 239.192.81.1)]
    [-locators=<addresses>]
    [-bind-address=<addr> (default is hostname or localhost if hostname points to a local loopback address)]
    [-<prop-name>=<prop-value>]*
    

    For example:

    gfxd print-stacks -all-threads allstacks.txt -mcast-port=0 -locators=192.168.159.132[10101] -log-level=config -log-file=pringstackslogs.log
    gfxd print-stacks -all-threads allstacks.txt -mcast-port=0 -locators=192.168.159.132[10101] -log-level=config -log-file=printstackslogs.log -user=locatoradmin -password=java -auth-provider=BUILTIN

    Note: when are using gemfirexd.properties, please run this command in a path with gemfirexd.properties.

  4. Heap dump

    For investigating memory related issues, such as an Out Of Memory issue or a Memory Leak, a heap dump will help track down underlying issues.

    Generate the heap dump for the GemFireXD process using the following procedure:

    Step 1. Identify the specific GemFireXD process ID using a command like jps (as in the procedure for getting thread dumps).

    Step 2. Generate the heap dump.

    1. Using jmap command:

      [JDK_INSTALLATION]/bin/jmap -dump:live,format=b,file=heap.dump.out <pid>
    2. Using Java VisualVM (jvisualvm):

      1. run [JDK_INSTALLATION]/bin/jvisualvm,
      2. select the target thread and select, [Application] menu-->[Heap Dump], and select the generated heap dump, then choose [Save As] "to local disk."
    3. Getting a Heap Dump Automatically on an "Out Of Memory" error:

      Add the following jvm parameter to the java process before it starts.

      -XX:+HeapDumpOnOutOfMemoryError
      -XX:HeapDumpPath
      

      For example (on Windows):

      JAVA_OPTS=%JAVA_OPTS% "-XX:+HeapDumpOnOutOfMemoryError" "-XX:HeapDumpPath=C:\TEMP"

Comments

Powered by Zendesk