Pivotal Knowledge Base

Follow

How to collect core files for analysis

Environment

Pivotal Greenplum (GPDB): 4.3.x

OS: RHEL 6.x

Purpose

As we are not guaranteed to have the precise library versions present on a client's system, it is necessary to collect additional information to read core files.

To achieve that, there was a large amount of information exchange between Pivotal support and customers to get the core file readable for engineering.

To help simplify this problem, there is a utility script (packcore) that will collect all the necessary files and generate the script to run gdb such that the core file is readable.

This tool will be available under $GPHOME/sbin for GPDB >= 4.3.2.0.  For versions lesser than 4.3.2.0, the attached file can be used. It would be good to use gpmt where packcore utility is embedded.

Usage:

There are currently two ways to get the packcore from the system:

1. Through gpmt :

This needs to be run on the server where the core file is located and then the tar archive needs to be sent to Pivotal support along with the gp_log_collector archive. 

[gpadmin@mdw ~]$./gpmt packcore --help
USAGE:
    ./gpmt packcore -cmd COMMAND -core COREFILE [ -binary BINARY ]
        [ -keep_tmp_dir ] [ -ignore_missing ]

DESCRIPTION:
    Packcore takes a corefile, extracts the name of the binary which generated
    the core, executes ldd (List Dynamic Dependencies) to get the required
    shared libraries and packages everything into a single tarball archive.

    Note: Packcore should be executed on the host where the core file was generated.
    The same PATH and LD_LIBRARY_PATH should also be set so that ldd can find the
    correct binary and shared libraries.

COMMANDS:
    collect               Collect core file and associated libraries

OPTIONS:
    -core                  Corefile
    -binary                Binary
    -keep_tmp_dir          Do not delete temp directory
    -ignore_missing        Ignore missing libraries

EXAMPLES:
    Collect core info for 'core.1234' with verbose logging (binary will be automatically detected).")
        ./gpmt packcore -cmd collect -core core.1234 -verbose\n", os.Args[0])

    Collect core info for 'core.1234' with binary '/usr/local/greenplum-db/postgres'.")
        ./gpmt packcore -cmd collect -core core.1234 -binary /usr/local/greenplum-db/postgres\n

 

2. Through using only packcore from the current build OR attachment and then shipping the file:

Instructions to use the tool:

    1. Copy the attached script (packcore) to the client's system OR use $GPHOME/sbin/packcore if it exists.
    2. To collect a core file, run the command: "packcore <core_file>" where "<core_file>" is the path to the core file to be collected.
      $GPHOME/sbin/packcore corefile OR ./packcore corefile
      
    3. The script will run, copying the core file, libraries, and other information to the local directory. Once finished, it will create a tar file named as shown below:
      packcore-<core_file>.tgz
    4. Send the compressed tar file to Pivotal support.
    5. For Pivotal support only: To read the core file, untar the file on any linux system, cd into the packcore-<core_file> directory and run the "runGDB.sh" script . 

Example of packcore generation and the information that gets collected.

[gpadmin@mdw ~]$ $GPHOME/sbin/packcore core.3326
Unable to find full path to binary for core file   <<<<<< In case of this issue we have to give binary path as shown below.

[gpadmin@mdw ~]$ ps -ef | grep silent gpadmin 3326 1 0 May10 ? 00:00:02 /usr/local/greenplum-db-4.3.8.0/bin/postgres -D /data/master/gpseg-1 -p 5432 -b 1 -z 8 --silent-mode=true -i -M master -C -1 -x 0 -E
[gpadmin@mdw ~]$ $GPHOME/sbin/packcore core.3326 -b /usr/local/greenplum-db-4.3.8.0/bin/postgres
[gpadmin@mdw ~]$ ls -lrth | tail -2 -rw-rw-r--. 1 gpadmin gpadmin 335M May 18 10:23 core.3326 -rw-rw-r--. 1 gpadmin gpadmin 148M May 18 10:25 packcore-core.3326.tgz
[gpadmin@mdw ~]$ tar -xzf packcore-core.3326.tgz
[gpadmin@mdw ~]$ cd packcore-core.3326
[gpadmin@mdw packcore-core.3326]$ ls -lrth total 386M -rw-rw-r--. 1 gpadmin gpadmin 22 May 18 10:24 uname.out -rw-r--r--. 1 gpadmin gpadmin 27 May 18 10:24 system-release -rw-r--r--. 1 gpadmin gpadmin 27 May 18 10:24 redhat-release -rwxr-xr-x. 1 gpadmin gpadmin 52M May 18 10:24 postgres -rw-rw-r--. 1 gpadmin gpadmin 335M May 18 10:24 core.3326 -rw-r--r--. 1 gpadmin gpadmin 27 May 18 10:24 centos-release drwxrwxr-x. 3 gpadmin gpadmin 4.0K May 18 10:24 usr -rwxr-xr-x. 1 gpadmin gpadmin 150 May 18 10:24 runGDB.sh drwxrwxr-x. 2 gpadmin gpadmin 4.0K May 18 10:24 lib64

Customers only need to send files such as packcore-core.3326.tgz to support.

Updates

  • Current version is 0.4 beta
  • This will now clean up work directory
  • Provide better error messages
  • '-b' flag to specify binary if unable to determine from core file (SuSE issue)
  • '-l' flag to force using ldd to resolve library dependancies

 

Comments

Powered by Zendesk