Pivotal Knowledge Base

Follow

Obtaining a web server stack trace in Linux/Unix (2000998)

Obtaining a web server stack trace in Linux/Unix (2000998)

Symptoms

Stack dumps from a debugger may be useful if you experience any of these symptoms:
  • Server crashes and creates a coredump file
  • Server hangs or is unresponsive
  • The error log contains the error message:

    possible core dump at <path>

Purpose

This article provides information on obtaining a stack trace against a core dump or running server process. When a server crashes or hangs, you can use the stack trace to troubleshoot the issue.
 
Note: The stack trace is also known as a traceback, backtrace, or unwind.

Resolution

Caution: Before proceeding, ensure that you are aware of the following points about web server stack traces:
  • When analyzing a coredump after a server failure, it is important that you do it on the server where it occurred, so that the debugger has access to the server environment in order to resolve code references in the coredump. Do not upload web server core files to VMware Support; the stack trace is much smaller and much more helpful.
  • To analyze a coredump, you need to know the actual server executable (not the startup script). For Apache HTTP Server, it is usually httpd . For vFabric ERS, it is httpsd.worker or httpsd.prefork depending on MPM (the default is worker). For vFabric Web Server, it is httpd.prefork, httpd.worker, or httpd.event (the default is worker).
  • If your server is hung, you can take a live stack trace on running server processes. Unless you know which process is problematic, obtain a stack trace for the control process (which is the oldest one and parent to the others) and at least one child process (which actually handles requests); ideally take a stack trace for all child processes.
  • To raise a support request, VMware recommends that you capture the full output of the debugger to a file. Do not cut and paste from your terminal; instead, use tee or shell redirection as in the examples below.
Note: In all commands that follow, substitute the following with your environment specifics:
  • /path/to/httpd The server executable
  • /path/to/core  A coredump file that has been left by a server crash
  • PID A server process ID that you find via the ps command.

Using gdb

gdb is available for free and is suitable for most Unix-like environments. It is best suited for Linux and is also available for Solaris and other platforms.
 
To obtain a stack trace from a core file, invoke gdb using this command:

# gdb /path/to/httpd /path/to/core | tee trace.txt

For a live stack trace, invoke gdb using this command:

# gdb -p PID | tee trace.txt

In either case, wait for initialization messages to complete and display the (gdb) prompt.
 
After the (gdb) prompt appears, run these commands:
 
(gdb) set pagination off
(gdb) thread apply all bt
(gdb) quit

A trace.txt file that contains your stack trace is created.

Using dbx

dbx is available on AIX or Solaris. You can invoke dbx in the same way as gdb, but the later commands are different.

To obtain a stack trace from a core file, invoke dbx using this command:

# dbx /path/to/httpd /path/to/core | tee trace.txt

For a live stack trace, invoke dbx using this command:

# dbx -a PID | tee trace.txt

In either case, wait for initialization messages to complete and display the (dbx) prompt.
 
After the (dbx) prompt appears:
  1. Run the info threads command to list all threads.
  2. For each thread, run the thread # command to change to the thread and run the where command to trace the thread.

A trace.txt file that contains your stack trace is created.

Using pstack and pflags

pstack and pflags are standard Solaris tools.

For a stack trace on a core dump, run these commands:

# pstack /path/to/core  > trace.txt
# pflags /path/to/core  >> trace.txt
 
Note: For a core, pflags indicates where the fault occurred. Other debuggers include this with the trace.

For a stack trace on a live process, run this command:

# pstack PID > trace.txt

A trace.txt file that contains your stack trace is created.

Additional Information

For related information, see the Apache Debugging Guide.

Tags

stack-trace

Confidential or Internal Solution information

We would refer to similar information for Windows, but it's almost impossible to do this on that platform.
©VMware 2013

Comments

Powered by Zendesk