Apache - Troubleshooting - Determine why an Apache process crashed

Apache - Troubleshooting - Determine why an Apache process crashed

Tools such as gcore and pstack are not of much use if the Apache process exited due to an alignment or segmentation violation.

Prior to the process exiting, a core file will typically be written to the directory that is defined in the ServerRoot directive.

In order for the core file to be written to this directory, the user id that the httpd process runs as will need to be have permission to write to this directory.
The core file size resource limit will also need to be set to a value large enough to allow a core file to be created.
If security policy prohibits allowing the user to write directly to the ServerRoot, the CoreDumpDirectory directive can be used to control where core files are written.

On some platforms this is still insufficient, since core files are not generated reliably. If you are experiencing server problems on once of these platforms,

The mod_backtrace module can be used to supplement the creation of core files.

mod_backtrace works by enabling a signal handler for each non-recoverable signal (such as SIGSEGV, SIGBUS, etc.).
When one of the non-recoverable signals is received, mod-backtrace will call a platform specific library call to generate a stack trace for the process, and then write this stack trace to the error_log.

NOTE: In order to use mod_backtrace, the Apache web server will need to be built with the exception hook.

This can be enabled at configure time by adding the –enable-exception-hook to the configure command line.

Once the module is compiled, installed and the exception hook enabled, the module can be loaded with the LoadModule directive, and the EnableExceptionHook directive can be set to On to enable mod_backtrace.

For reference, here are the Apache configuration file entries needed to enable mod_backtrace:

LoadModule backtrace_module modules/mod_backtrace.so 
EnableExceptionHook On

Install mod_backtrace

mod_backtrace is available as source code, and can retrieved from the apache contributors website:

wget http://people.apache.org/~trawick/mod_backtrace.c

Compile and install the module

apxs -ci mod_backtrace.c

NOTE: The options are:

c: Compile the module.
i: Install the module.

Checking the logs

After the mod_backtrace module is enabled, an entry similar to the following will be written to the error_log each time Apache receives a critical signal:

kill -SIGSEGV 12345

returns:

[Wed Dec 20 16:48:32 2006] pid 12345 mod_backtrace backtrace for sig 11 (thread "pid" 12345)
[Wed Dec 20 16:48:32 2006] pid 12345 mod_backtrace main() is at 26bc8
/var/tmp/apache/modules/mod-backtrace.so:bt_exception_hook+0x108
/var/tmp/apache/bin/httpd:ap_run_fatal_exception+0x34
/var/tmp/apache/bin/httpd:0x32788
/lib/libc.so.1:0xc01dc
/lib/libc.so.1:0xb52d4
/lib/libc.so.1:_so_accept+0x8 [ Signal 11 (SEGV)]
/var/tmp/apache/bin/httpd:unixd_accept+0x10
/var/tmp/apache/bin/httpd:0x3a3c0
/var/tmp/apache/bin/httpd:0x3a6c8
/var/tmp/apache/bin/httpd:0x3a798
/var/tmp/apache/bin/httpd:ap_mpm_run+0x9d4
/var/tmp/apache/bin/httpd:main+0x710
/var/tmp/apache/bin/httpd:_start+0x5c
[Wed Feb 26 12:13:14 2016] pid 12345 mod_backtrace end of backtrace

NOTE: mod_backtrace uses the backtrace() routine on GNU systems, and the printstack() method on Solaris systems to programmatically retrieve a backtrace.

In some cases symbol names may not be resolved correctly.
If this happens, the nm, readelf and elfdump utilities can be used to supplement the information provided by mod_backtrace.

Table of Contents

Apache - Troubleshooting - Determine why an Apache process crashed

Install mod_backtrace

Compile and install the module

Checking the logs