File better bugs with coredumpctl

An unfortunate fact of life is that all software has bugs, and some bugs can make the system crash. When it does, it often leaves a data file called a core dump on disk. This file contains data about your system when it crashed, and may help determine why the crash occurred. Often developers request data in the form of a backtrace, showing the flow of instructions that led to the crash. The developer can use it to fix the bug and improve the system. Here’s how to easily generate a backtrace if you have a system crash.

Getting started with coredumpctl

Most Fedora systems use the Automatic Bug Reporting Tool (ABRT) to automatically capture dumps and file bugs for crashes. However, if you have disabled this service or removed the package, this method may be helpful.

If you experience a system crash, first ensure that you’re running the latest updated software. Updates often contain fixes that have already been found to fix a bug that causes critical errors and crashes. Once you update, try to recreate the situation that led to the bug.

If the crash still happens, or if you’re already running the latest software, it’s time to use the helpful coredumpctl utility. This utility helps locate and process crashes. To see a list of all core dumps on your system, run this command:

coredumpctl list

Don’t be surprised if you see a longer list than expected. Sometimes system components crash silently behind the scenes, and recover on their own. An easy way to quickly find a dump from today is to use the –since option:

coredumpctl list --since=today

The PID column contains the process ID used to identify the dump. Note that number, since you’ll use it again along the way. Or, if you don’t want to remember it, assign it to a variable you can use in the rest of the commands below:

MYPID=<PID>

To see information about the core dump, use this command (either use the $MYPID variable, or substitute the PID number):

coredumpctl info $MYPID

Install debuginfo packages

Debugging symbols translate between data in the core dump and the instructions found in original source code. This symbol data can be quite large. Therefore, symbols are shipped in debuginfo packages separately from the packages most users run on Fedora systems. To determine which debuginfo packages you must install, start by running this command:

coredumpctl gdb $MYPID

This may result in a large amount of information to the screen. The last line may tell you to use dnf to install more debuginfo packages. Run that command with sudo to continue:

sudo dnf debuginfo-install <packages...>

Then try the coredumpctl gdb $MYPID command again. You may need to do this repeatedly, as other symbols are unwound in the trace.

Capturing the backtrace

Run the following commands to log information in the debugger:

set logging file mybacktrace.txt
set logging on

You may find it helpful to turn off the pagination. For long backtraces this saves time.

set pagination off

Now run the backtrace:

thread apply all bt full

Now you can type quit to quit the debugger. The mybacktrace.txt file includes backtrace information you can attach to a bug or issue. Or if you’re working with someone in real time, you can upload the text to a pastebin. Either way, you can now provide more assistance to the developer to fix the problem.

Fedora Project community For Developers Using Software

3 Comments

  1. ifoolb

    Sometimes the system is completely unresponsive, even sysrq doesn’t work, while the display is still on. Any way to generate crash dump in such situations(especially on a laptop, no firewire, serial ports )?

    • wk3

      There is. Use the Magic SysRq key!

      When the system has become useless, and you cannot create nor resume an alternate console session (Ctrl+Alt+F2, Ctrl+Alt+F3, etc.), press the keystroke sequence to cause a system crash:

      Alt + SysRq c

      See https://en.wikipedia.org/wiki/Magic_SysRq_key for all the commands available for the Magic SysRq key.

      By the way, if you can switch to an alternate console session, list the highest CPU usage processes with “top”. In Fedora 24 and 25, it has almost always been (for me) caused by gnome-shell. A system hang can perhaps be cleared up by killing it (and for gnome-shell, letting it self restart). Then switch back to GUI mode (with Ctrl+Alt+F1) and see if the problem clears up.

      If you run “top” as root (use “su” to get the # prompt), “top” has a command to kill processes. When the offending process is at the top of the list, press “k” and it will prompt with
      PID to signal/kill [default pid = 23123]
      If it is correct, press Enter; otherwise type in the pid of the suspected errant process then Enter. “Top” prompts with
      Send pid 23123 signal [15/sigterm]
      Many processes will exit cleanly with SIGTERM, so just press Enter. If that doesn’t kill it, I find that signal 9 (SIGKILL) will do the job (programs cannot ignore or catch SIGKILL). See “man 7 signal” for details.

  2. Leslie Satenstein

    Very informative article. Because Fedora is so stable, I do not see many crashes or aborts. Hence, this software is something I would not come across in daily use. It’s good to know for any future situation.

    Thank you.

Comments are Closed

The opinions expressed on this website are those of each author, not of the author's employer or of Red Hat. Fedora Magazine aspires to publish all content under a Creative Commons license but may not be able to do so in all cases. You are responsible for ensuring that you have the necessary permission to reuse any work on this site. The Fedora logo is a trademark of Red Hat, Inc. Terms and Conditions