Pivotal Knowledge Base

Follow

How to collect core files for analysis

Environment

Product Version
Pivotal Greenplum (GPDB) 4.3.x
OS RHEL 6.x

Purpose

As we are not guaranteed to have the precise library versions present on a client's system, it is necessary to collect additional information to read core files.

To achieve that, there was a large amount of information exchange between Pivotal support and customers to get the core file readable for engineering.

To help simplify this problem, there is a utility script (packcore) that will collect all the necessary files and generate the script to run gdb such that the core file is readable.

This tool will be available under $GPHOME/sbin for GPDB >= 4.3.2.0.  For versions lesser than 4.3.2.0, the attached file can be used. It would be good to use gpmt where packcore utility is embedded.

Usage:

There are currently two ways to get the packcore from the system:

1. Through gpmt :

This needs to be run on the server where the core file is located and then the tar archive needs to be sent to Pivotal support along with the gp_log_collector archive. 

[gpadmin@mdw ~]$./gpmt packcore --help
USAGE:
    ./gpmt packcore -cmd COMMAND -core COREFILE [ -binary BINARY ]
        [ -keep_tmp_dir ] [ -ignore_missing ]

DESCRIPTION:
    Packcore takes a corefile, extracts the name of the binary which generated
    the core, executes ldd (List Dynamic Dependencies) to get the required
    shared libraries and packages everything into a single tarball archive.

    Note: Packcore should be executed on the host where the core file was generated.
    The same PATH and LD_LIBRARY_PATH should also be set so that ldd can find the
    correct binary and shared libraries.

COMMANDS:
    collect               Collect core file and associated libraries

OPTIONS:
    -core                  Corefile
    -binary                Binary
    -keep_tmp_dir          Do not delete temp directory
    -ignore_missing        Ignore missing libraries

EXAMPLES:
    Collect core info for 'core.1234' with verbose logging (binary will be automatically detected).")
        ./gpmt packcore -cmd collect -core core.1234 -verbose\n", os.Args[0])

    Collect core info for 'core.1234' with binary '/usr/local/greenplum-db/postgres'.")
        ./gpmt packcore -cmd collect -core core.1234 -binary /usr/local/greenplum-db/postgres\n

 

2. Through using only packcore from the current build OR attachment and then shipping the file:

Instructions to use the tool:

    1. Copy the attached script (packcore) to the client's system OR use $GPHOME/sbin/packcore if it exists.
    2. To collect a core file, run the command: "packcore <core_file>" where "<core_file>" is the path to the core file to be collected.
      $GPHOME/sbin/packcore corefile OR ./packcore corefile
      
    3. The script will run, copying the core file, libraries, and other information to the local directory. Once finished, it will create a tar file named as shown below:
      packcore-<core_file>.tgz
    4. Send the compressed tar file to Pivotal support.
    5. For Pivotal support only: To read the core file, untar the file on any linux system, cd into the packcore-<core_file> directory and run the "runGDB.sh" script . 

Example of packcore generation and the information that gets collected.

[gpadmin@mdw ~]$ $GPHOME/sbin/packcore core.3326
Unable to find full path to binary for core file   <<<<<< In case of this issue we have to give binary path as shown below.

[gpadmin@mdw ~]$ ps -ef | grep silent gpadmin 3326 1 0 May10 ? 00:00:02 /usr/local/greenplum-db-4.3.8.0/bin/postgres -D /data/master/gpseg-1 -p 5432 -b 1 -z 8 --silent-mode=true -i -M master -C -1 -x 0 -E
[gpadmin@mdw ~]$ $GPHOME/sbin/packcore core.3326 -b /usr/local/greenplum-db-4.3.8.0/bin/postgres
[gpadmin@mdw ~]$ ls -lrth | tail -2 -rw-rw-r--. 1 gpadmin gpadmin 335M May 18 10:23 core.3326 -rw-rw-r--. 1 gpadmin gpadmin 148M May 18 10:25 packcore-core.3326.tgz
[gpadmin@mdw ~]$ tar -xzf packcore-core.3326.tgz
[gpadmin@mdw ~]$ cd packcore-core.3326
[gpadmin@mdw packcore-core.3326]$ ls -lrth total 386M -rw-rw-r--. 1 gpadmin gpadmin 22 May 18 10:24 uname.out -rw-r--r--. 1 gpadmin gpadmin 27 May 18 10:24 system-release -rw-r--r--. 1 gpadmin gpadmin 27 May 18 10:24 redhat-release -rwxr-xr-x. 1 gpadmin gpadmin 52M May 18 10:24 postgres -rw-rw-r--. 1 gpadmin gpadmin 335M May 18 10:24 core.3326 -rw-r--r--. 1 gpadmin gpadmin 27 May 18 10:24 centos-release drwxrwxr-x. 3 gpadmin gpadmin 4.0K May 18 10:24 usr -rwxr-xr-x. 1 gpadmin gpadmin 150 May 18 10:24 runGDB.sh drwxrwxr-x. 2 gpadmin gpadmin 4.0K May 18 10:24 lib64

Customers only need to send files such as packcore-core.3326.tgz to support.

Updates

  • Current version is 0.4 beta
  • This will now clean up work directory
  • Provide better error messages
  • '-b' flag to specify binary if unable to determine from core file (SuSE issue)
  • '-l' flag to force using ldd to resolve library dependancies

 

Comments

Powered by Zendesk