====== Apache - Troubleshooting - Determine why an Apache process crashed ======

Tools such as **gcore** and **pstack** are not of much use if the Apache process exited due to an alignment or segmentation violation.

Prior to the process exiting, a core file will typically be written to the directory that is defined in the **ServerRoot** directive.

  * In order for the core file to be written to this directory, the user id that the httpd process runs as will need to be have permission to write to this directory.
  * The **core file size** resource limit will also need to be set to a value large enough to allow a core file to be created.
  * If security policy prohibits allowing the user to write directly to the ServerRoot, the **CoreDumpDirectory** directive can be used to control where core files are written.

On some platforms this is still insufficient, since core files are not generated reliably. If you are experiencing server problems on once of these platforms, 

The **mod_backtrace** module can be used to supplement the creation of core files.

  * mod_backtrace works by enabling a signal handler for each non-recoverable signal (such as SIGSEGV, SIGBUS, etc.).
  * When one of the non-recoverable signals is received, mod-backtrace will call a platform specific library call to generate a stack trace for the process, and then write this stack trace to the **error_log**.

<WRAP info>
**NOTE:**  In order to use mod_backtrace, the Apache web server will need to be built with the exception hook.

This can be enabled at configure time by adding the **–enable-exception-hook** to the configure command line.

Once the module is compiled, installed and the exception hook enabled, the module can be loaded with the **LoadModule** directive, and the **EnableExceptionHook** directive can be set to **On** to enable mod_backtrace. 

For reference, here are the Apache configuration file entries needed to enable mod_backtrace:

<code>
LoadModule backtrace_module modules/mod_backtrace.so 
EnableExceptionHook On
</code>

</WRAP>

----

===== Install mod_backtrace =====

mod_backtrace is available as source code, and can retrieved from the apache contributors website:

<code bash>
wget http://people.apache.org/~trawick/mod_backtrace.c
</code>

----

===== Compile and install the module =====

<code bash>
apxs -ci mod_backtrace.c
</code>

<WRAP info>
**NOTE:**  The options are:

  * **c**:  Compile the module.
  * **i**:  Install the module.

</WRAP>

----

===== Checking the logs =====

After the mod_backtrace module is enabled, an entry similar to the following will be written to the error_log each time Apache receives a critical signal:

<code bash>
kill -SIGSEGV 12345
</code>

returns:

<code bash>
[Wed Dec 20 16:48:32 2006] pid 12345 mod_backtrace backtrace for sig 11 (thread "pid" 12345)
[Wed Dec 20 16:48:32 2006] pid 12345 mod_backtrace main() is at 26bc8
/var/tmp/apache/modules/mod-backtrace.so:bt_exception_hook+0x108
/var/tmp/apache/bin/httpd:ap_run_fatal_exception+0x34
/var/tmp/apache/bin/httpd:0x32788
/lib/libc.so.1:0xc01dc
/lib/libc.so.1:0xb52d4
/lib/libc.so.1:_so_accept+0x8 [ Signal 11 (SEGV)]
/var/tmp/apache/bin/httpd:unixd_accept+0x10
/var/tmp/apache/bin/httpd:0x3a3c0
/var/tmp/apache/bin/httpd:0x3a6c8
/var/tmp/apache/bin/httpd:0x3a798
/var/tmp/apache/bin/httpd:ap_mpm_run+0x9d4
/var/tmp/apache/bin/httpd:main+0x710
/var/tmp/apache/bin/httpd:_start+0x5c
[Wed Feb 26 12:13:14 2016] pid 12345 mod_backtrace end of backtrace
</code>

<WRAP info>
**NOTE:**  mod_backtrace uses the **backtrace()** routine on GNU systems, and the printstack() method on Solaris systems to programmatically retrieve a backtrace.

  * In some cases symbol names may not be resolved correctly.
  * If this happens, the **nm**, **readelf** and **elfdump** utilities can be used to supplement the information provided by mod_backtrace.
</WRAP>