Tracking Down Crashes on Linux

By Alex Peshkov, May 2008


Also works for other POSIX systems

One of the most efficient ways to track a crash of any program on Linux is to get a core dump of it at the moment of the crash. Using it you can easily create a stack backtrace, which in many cases makes it possible for developers to solve your problem. Or (if you using Firebird debug binaries) the whole core file can be sent to the developers for deeper analysis.

This document describes the process of how to get a core dump of the Firebird server on Linux and then analyzing it with gdb, using Firebird's debuginfo package. Depending on OS version (and other issues) the file with the core dump may be named simply “core” or “core.PID”, where PID is a number of the process which of which the core was dumped. Always use the actual name of the core file rather than “core” in examples below.

Getting a core dump

Firebird 2.0.4 or higher

You should set parameter BugcheckAbort=1 in the firebird.conf file. Do not forget to restart superserver after you modify the firebird.conf. Resulting core dump will be found in the /tmp directory.

Setting BugcheckAbort=1 will change Firebird's working directory from "/" to "/tmp". This is done because the core file is created in the current working directory (there does not seem to be a way to send it to another location). Linux docs do describe a way, but it's system-wide and can not be changed from a non-root process. Therefore when running as 'firebird' we are unable to get core files for Classic. (SuperServer has run in /tmp since InterBase V6). Therefore when BugcheckAbort is set we have to change the current working directory to writable /tmp.

Firebird 2.0.3 or earlier

The first thing you should consider if you are using an old version of Firebird is to upgrade it. The Firebird development team regularly fixes various bugs in the code base, therefore it's quite possible that your problem has already solved. But if you do need to get core dump of an old Firebird version. You will need to perform the following actions:

To get dump of process's core you should satisfy the following requirements:

1.Set ulimit's RLIMIT_CORE value big enough to to allow it to handle the dump of the Firebird server (its even simpler and better to make it unlimited). This is what the command “ulimit -c unlimited” normally does. 2.Make the working directory of Firebird server writable for the server process – the core file is always created in current directory.

Note: Just typing
# ulimit -c unlimited

at the shell prompt does not affect this setting for any daemon, running on your system – this includes Firebird. So here is how you get a core file for both Superserver and Classic.


To make RLIMIT_CORE to unlimited, you should edit /etc/init.d/firebird file. This is the startup script for Firebird, however it's exact location may differ for on various versions of Linux, but in most cases it usually is in /etc/init.d directory. See your OS documentation for exact location of the startup script. Script contents may also differ much depending on Linux version. In all files you can find a line starting with:

export FIREBIRD=
(followed by FbRoot directory, usually /opt/firebird). Add right after this command:

ulimit -c unlimited

and restart Firebird.

Superserver always sets it's working directory to /tmp, which is normally writable to everyone. You Should find the core dump in /tmp after next crash.

Classic server

The Classic server is normally started using xinetd, but there is no easy way to separately configure ulimit for services, started in this manner. The only way to is to set ulimit for xinetd itself. To do this, you should modify startup the script for xinetd. These scripts differ for the various Linux versions, but in most cases adding:

ulimit -c unlimited

near the beginning of the script should work. Do not forget to restart xinetd after modifying the startup script.

Classic server has / (root) as it's working directory, which is normally writable only to root user. Therefore if you are not running Firebird as root, you will not get core dump.

To solve this problem do one of the following:

use script to temporarily run firebird as root (to revert back to the Firebird runuser, user There is a slim possibility that some crashes may not be reproducible when running as root. make / writable for user firebird. This is probably the best solution for systems with ACLs support.

Analyzing a core dump with gdb

To analyze the core dump you should download the debuginfo package from the main Firebird download pages (for non normal Firebird project supplied packages or OS's where the debuginfo packages are not available you may need to contact the package maintainer for the debuginfo package). Debuginfo package is not an install – just untar it in the root of your filesystem. After if you may use:

# gdb /opt/firebird/bin/.debug/fb_inet_server.debug core

Then use the following gdb command:

(gdb) thread apply all bt

to analyze core dump of classic server or:

# gdb /opt/firebird/bin/.debug/fbserver.debug core

Then use the following gdb command:

(gdb) thread apply all bt

to analyze core dump of superserver.

Firebird's debuginfo before 2.5.0 has one problem – all binaries in it are linked as runtime, not debug. This is not critical for superserver, but can cause problems for classic. To analyze classics's core properly, do:

# mv /opt/firebird/lib/ /opt/firebird/lib/
# mv /opt/firebird/lib/.debug/ /opt/firebird/lib/

After analyzing core, rename the files back.

The sample is provided for Firebird 1.5.5, but if you substitute your version number, it's correct for any Firebird version.

For Firebird versions starting with 2.5.0 you do not need to rename the files - just use gdb as normal, abnd it will pick up the debuginfo automatically.