Skip to content

Latest commit

 

History

History
168 lines (125 loc) · 6.04 KB

minidump_to_core.md

File metadata and controls

168 lines (125 loc) · 6.04 KB
Raw

Linux Minidump to Core

On Linux, Chromium can use Breakpad to generate minidump files for crashes. It is possible to convert the minidump files to core files, and examine the core file in gdb, cgdb, or Qtcreator. In the examples below cgdb is assumed but any gdb based debugger can be used.

[TOC]

Creating the core file

Use minidump-2-core to convert the minidump file to a core file. On Linux, one can build the minidump-2-core target in a Chromium checkout, or alternatively, build it in a Google Breakpad checkout.

$ ninja -C out/Release minidump-2-core
$ ./out/Release/minidump-2-core foo.dmp > foo.core

Retrieving Chrome binaries

If the minidump is from a public build then Googlers can find Google Chrome Linux binaries and debugging symbols via https://goto.google.com/chromesymbols. Otherwise, use the locally built chrome files. Google Chrome uses the debug link method to specify the debugging file. Either way be sure to put chrome and chrome.debug (the stripped debug information) in the same directory as the core file so that the debuggers can find them.

For Chrome OS release binaries look for debug-*.tgz files on GoldenEye.

Loading the core file into gdb/cgdb

The recommended syntax for loading a core file into gdb/cgdb is as follows, specifying both the executable and the core file:

$ cgdb chrome foo.core

If the executable is not available then the core file can be loaded on its own but debugging options will be limited:

$ cgdb -c foo.core

If symbols do not seem to work, see Advanced module loading below.

Loading the core file into Qtcreator

Qtcreator is a full GUI wrapper for gdb and it can also load Chrome's core files. From Qtcreator select the Debug menu, Start Debugging, Load Core File... and then enter the paths to the core file and executable. Qtcreator has windows to display the call stack, locals, registers, etc. For more information on debugging with Qtcreator see Getting Started Debugging on Linux.

Source debugging

If you have a Chromium repo that is synchronized to exactly (or even approximately) when the Chrome build was created then you can tell gdb/cgdb/Qtcreator to load source code. Since all source paths in Chrome are relative to the out/Release directory you just need to add that directory to your debugger search path, by adding a line similar to this to ~/.gdbinit:

(gdb) directory /usr/local/chromium/src/out/Release/

Notes

  • Since the core file is created from a minidump, it is incomplete and the debugger may not know values for variables in memory. Minidump files contain thread stacks so local variables and function parameters should be available, subject to the limitations of optimized builds.
  • For gdb's add-symbol-file command to work, the file must have debugging symbols.
    • In case of separate debug files, the gdb manual explains how gdb looks for them.
  • If the stack trace involve system libraries, the Advanced module loading steps shown below need to be repeated for each library.

Advanced module loading

If gdb doesn't find shared objects that are needed you can force it to load them. In gdb, the add-symbol-file command takes a filename and an address. To figure out the address, look near the end of foo.dmp, which contains a copy of /proc/pid/maps from the process that crashed.

One quick way to do this is with grep. For instance, if the executable is /path/to/chrome, one can simply run:

$ grep -a /path/to/chrome$ foo.dmp

7fe749a90000-7fe74d28f000 r-xp 00000000 08:07 289158        /path/to/chrome
7fe74d290000-7fe74d4b7000 r--p 037ff000 08:07 289158        /path/to/chrome
7fe74d4b7000-7fe74d4e0000 rw-p 03a26000 08:07 289158        /path/to/chrome

In this case, 7fe749a90000 is the base address for /path/to/chrome, but gdb takes the start address of the file's text section. To calculate this, one will need a copy of /path/to/chrome, and run:

$ objdump -x /path/to/chrome | grep '\.text' | head -n 1 | tr -s ' ' | cut -d' ' -f 5

00000000005282c0

Now add the two addresses: 7fe749a90000 + 5282c0 = 7fe749fb82c0 and in gdb, run:

(gdb) add-symbol-file /path/to/chrome 0x7fe749fb82c0

Then use gdb as normal.

Using minidump_stackwalk instead of gdb

If all you need is a stack trace, you can use minidump_stackwalk instead of gdb as follows:

  • Compile minidump_stackwalk: autoninja -C out/Default minidump_stackwalk

  • Download breakpad_info.zip for the correct version from go/chromesymbols (Googlers only).

    You can instead generate the .sym files yourself, if you have a build with symbols: out/Default/dump_syms chrome > chrome.sym

  • minidump_stackwalk expects a directory structure like debugfile/BUILDID/debugfile.sym. On Linux and macOS debugfile is just the name of the executable; on Windows, debugfile is that plus a .pdb extension.

    The buildid is in the first line of the breakpad info file:

    $ head -1 chrome.breakpad.x64
MODULE Linux x86_64 0ACBB4D08FB145E1656ADD88DF70B1320 chrome.debug

    So for this specific example, you'd do:

    mkdir -p symbols/chrome/0ACBB4D08FB145E1656ADD88DF70B1320
mv chrome.breakpad.x64 symbols/chrome/0ACBB4D08FB145E1656ADD88DF70B1320

  • Now you can run minidump_stackwalk:

    $ out/Default/minidump_stackwalk 5aeb7476-3200-41ed-8db8-b5d71bea28d3.dmp symbols/
[...]
Thread 0 (crashed)
  0  chrome!content::protocol::FedCmHandler::OnDialogShown() [vector : 1434 + 0x0]
     rax = 0x00003b2401be0720   rdx = 0x000000000000002e

Other resources

For more discussion on this process see [Debugging a Minidump]. This page discusses the same process in the context of Chrome OS and many of the concepts and techniques overlap.

Debugging a Minidump