An unfortunate fact of life is that all software has bugs, and some bugs can make the system crash. When it does, it often leaves a data file called a core dump on disk. This file contains data about your system when it crashed, and may help determine why the crash occurred. Often developers request data in the form of a backtrace, showing the flow of instructions that led to the crash. The developer can use it to fix the bug and improve the system. Here’s how to easily generate a backtrace if you have a system crash.
Getting started with coredumpctl
Most Fedora systems use the Automatic Bug Reporting Tool (ABRT) to automatically capture dumps and file bugs for crashes. However, if you have disabled this service or removed the package, this method may be helpful.
If you experience a system crash, first ensure that you’re running the latest updated software. Updates often contain fixes that have already been found to fix a bug that causes critical errors and crashes. Once you update, try to recreate the situation that led to the bug.
If the crash still happens, or if you’re already running the latest software, it’s time to use the helpful coredumpctl utility. This utility helps locate and process crashes. To see a list of all core dumps on your system, run this command:
coredumpctl list
Don’t be surprised if you see a longer list than expected. Sometimes system components crash silently behind the scenes, and recover on their own. An easy way to quickly find a dump from today is to use the –since option:
coredumpctl list --since=today
The PID column contains the process ID used to identify the dump. Note that number, since you’ll use it again along the way. Or, if you don’t want to remember it, assign it to a variable you can use in the rest of the commands below:
MYPID=<PID>
To see information about the core dump, use this command (either use the $MYPID variable, or substitute the PID number):
coredumpctl info $MYPID
Install debuginfo packages
Debugging symbols translate between data in the core dump and the instructions found in original source code. This symbol data can be quite large. Therefore, symbols are shipped in debuginfo packages separately from the packages most users run on Fedora systems. To determine which debuginfo packages you must install, start by running this command:
coredumpctl gdb $MYPID
This may result in a large amount of information to the screen. The last line may tell you to use dnf to install more debuginfo packages. Run that command with sudo to continue:
sudo dnf debuginfo-install <packages...>
Then try the coredumpctl gdb $MYPID command again. You may need to do this repeatedly, as other symbols are unwound in the trace.
Capturing the backtrace
Run the following commands to log information in the debugger:
set logging file mybacktrace.txt set logging on
You may find it helpful to turn off the pagination. For long backtraces this saves time.
set pagination off
Now run the backtrace:
thread apply all bt full
Now you can type quit to quit the debugger. The mybacktrace.txt file includes backtrace information you can attach to a bug or issue. Or if you’re working with someone in real time, you can upload the text to a pastebin. Either way, you can now provide more assistance to the developer to fix the problem.
ifoolb
Sometimes the system is completely unresponsive, even sysrq doesn’t work, while the display is still on. Any way to generate crash dump in such situations(especially on a laptop, no firewire, serial ports )?
wk3
There is. Use the Magic SysRq key!
When the system has become useless, and you cannot create nor resume an alternate console session (Ctrl+Alt+F2, Ctrl+Alt+F3, etc.), press the keystroke sequence to cause a system crash:
Alt + SysRq c
See https://en.wikipedia.org/wiki/Magic_SysRq_key for all the commands available for the Magic SysRq key.
By the way, if you can switch to an alternate console session, list the highest CPU usage processes with “top”. In Fedora 24 and 25, it has almost always been (for me) caused by gnome-shell. A system hang can perhaps be cleared up by killing it (and for gnome-shell, letting it self restart). Then switch back to GUI mode (with Ctrl+Alt+F1) and see if the problem clears up.
If you run “top” as root (use “su” to get the # prompt), “top” has a command to kill processes. When the offending process is at the top of the list, press “k” and it will prompt with
PID to signal/kill [default pid = 23123]
If it is correct, press Enter; otherwise type in the pid of the suspected errant process then Enter. “Top” prompts with
Send pid 23123 signal [15/sigterm]
Many processes will exit cleanly with SIGTERM, so just press Enter. If that doesn’t kill it, I find that signal 9 (SIGKILL) will do the job (programs cannot ignore or catch SIGKILL). See “man 7 signal” for details.
Leslie Satenstein
Very informative article. Because Fedora is so stable, I do not see many crashes or aborts. Hence, this software is something I would not come across in daily use. It’s good to know for any future situation.
Thank you.