# Linux Minidump Code Lab [TOC] ## About Minidumps Minidump is a file format for storing parts of a program's state for later inspection. [Microsoft's Documentation](https://docs.microsoft.com/en-us/windows/desktop/api/minidumpapiset/) defines the format though the [Rust Documentation](https://docs.rs/minidump/latest/minidump/format/index.html) is sometimes easier to navigate. The minidump implementation and tools used by Chrome are [Breakpad](https://chromium.googlesource.com/breakpad/breakpad/+/refs/heads/main/README.md) and [Crashpad](https://chromium.googlesource.com/crashpad/crashpad/+/main/README.md). However, the tools of interest here are from the Breakpad project. ## Create a Minidump When Chrome crashes it writes out a minidump file. The minidump file is written under the application product directory. On Linux this is `//Crash Reports`. The default for `XDG_CONFIG_HOME` is `~.config`. Common ``s are `chromium`, `google-chrome`, `google-chrome-beta`, and `google-chrome-unstable`. A typical example is `~.config/google-chrome/Crash Reports`. When a minidump is uploaded it will be moved between the `new`, `pending`, and `completed` subdirectories. The minidump file is named something like `.dmp`. If the minidump is uploaded to the crash reporting system, the `.meta` file will contain the crash report id. Those with access can find the uploaded report at `go/crash/`, where the minidump file will be available with a name like `upload_file_minidump-.dmp`. To create a minidump, you can use a local build of Chromium or a release version of Chrome. Run the browser with the environment variable `CHROME_HEADLESS=1`, which enables crash dumping but prevents crash dumps from being uploaded and deleted. Something like `$ env CHROME_HEADLESS=1 ./out/debug/chrome-wrapper` or `$ env CHROME_HEADLESS=1 /opt/google/chrome/google-chrome`. Navigate to `chrome://crash` to trigger a crash in the renderer process or reproduce your current crash bug. A crash dump file should appear in the `Crash Reports` directory. ## Inspect the Minidump To get an idea about what is in a minidump file, install the Okteta hex editor and add the [Minidump Structure Definition](https://github.com/bungeman/structures/tree/main/okteta-minidump). Open the minidump previously created and explore the information it contains. One quirk to notice is that there is a `ThreadListStream` which contains `MINIDUMP_THREAD`s which contain a `MINIDUMP_THREAD_CONTEXT` and an `ExceptionStream` which also contains a `MINIDUMP_THREAD_CONTEXT`. The thread list contains the thread contexts as they existed when the crash reporter was running. The exception's thread context is the state of the crashing thread at the time that it crashed, which is generally the most interesting thread context. When using the Breakpad tools for Linux (like `minidump_stackwalk` and `minidump-2-core`) the thread context from the exception record is used in place of the thread context associated with the corresponding thread. Each `MINIDUMP_THREAD` contains a `StackMemoryRva` which is a reference to to a copy of the stack on that thread at the time the crash handler was running. Parsing a stack usefully requires additional debug information. `minidump_stackwalk` or a debugger may be used to parse the stack memory to create a usable trace. ## Get the Tools From a Chromium checkout `ninja -C out/release minidump-2-core minidump_stackwalk dump_syms`. From a [Breakpad checkout](https://chromium.googlesource.com/breakpad/breakpad/) `make`. It can be useful to use Breakpad directly on machines where one does not already have a Chromium checkout. When working at this level, one will also want to have `readelf` and `objdump` available, which are available from most distributions. ## Get Executables and Symbols In addition to the minidump, you will need the exact executables of Chromium or Chrome which produced the minidump and those executable's symbols. If the minidump was created locally, you already have the executables. Symbols for Google Chrome's official builds are available from `https://edgedl.me.gvt1.com/chrome/linux/symbols/google-chrome-debug-info-linux64-${VERSION}.zip` where `${VERSION}` is any version of Google Chrome that has recently been served to Stable, Beta, or Unstable (Dev) channels on Linux, like `114.0.5696.0`. Those with access can find both executables and symbols for unreleased builds at `go/chrome-symbols`. For symbols outside of Chrome (like when the crash is happening in a shared library) then symbols for the files of interest must be found. If the minidump was created locally then install the symbol packages from your distribution. If not, you will need to track down the exact symbol files, which can be an interesting exercise. For some distributions using the [debuginfod](https://sourceware.org/elfutils/Debuginfod.html) system can be quite helpful. To ensure the correct binaries and debug symbols are used, the minidump contains the build-id for each loaded module in the `ModuleListStream` in the `CvRecordRva`'s `Signature`. This build-id is matched against a note section of type `NT_GNU_BUILD_ID`, usually named `.note.gnu.build-id` in the executable and symbol files. This note can be inspected with `readelf -n ` like `readelf -n chrome` or `readelf -n chrome.debug` and looking for the `.note.gnu.build-id` section. `readelf` reports the `Build ID` as the flat bytes in the note, but Breakpad binaries like `stackwalk_minidimp` and `dump_syms` will report and expect this truncated to a formatted Type 2 GUID (without dashes). This means `readelf` will output a `` like 33221100554477668899AABBCCDDEEFFXXXXXXXX but crashpad binaries will expect and report this as a `` of 00112233445566778899AABBCCDDEEFF. The `.gnu_debuglink` section states which debug symbol file to use with a striped binary. For example `readelf --string-dump=.gnu_debuglink chrome` produces `chrome.debug`. This can be helpful to know for libraries with interesting debug symbol setup, like libc.so.6. ## Create Symbolized Stack Given a minidump with the name `mini.dmp` `minidump_stackwalk mini.dmp > mini.stackwalk.nosym` This will produce a mostly unsymbolized summary of the crash. To symbolize, look toward the bottom of the output for `WARNING: No symbols, , `. For each `` which is of interest, `mkdir -p symbols//` then `dump_syms > symbols///.sym`. Ensure this output `.sym` contains the expected ``. Then re-run `minidump_stackwalk` but with the symbols directory, like `minidump_stackwalk mini.dmp symbols/ > mini.stackwalk`. The output of `minidump_stackwalk` is often quite useful and enough to track down many issues. However, it does not fully use all of the information from DWARF, so it is possible sometimes to get much better stack traces from a full debugger like gdb. This is particularly true when functions have been aggressively inlined. ## Create Core File `minidump-2-core mini.dmp > mini.core` ## Loading into GDB This works best if the binaries, symbols, and core files are all in different directories to prevent gdb from automatically loading them into the wrong locations. This is also generally necessary when using a system installed version of Chrome. For full details see [the gdb manual](https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html). The easiest way is to rename and move the .debug files to a directory structure like `/debug/.build-id/nn/nnnnnnnn.debug` where `nn` are the first two hex characters of the `build-id`, and `nnnnnnnn` are the rest of the hex characters of the `build-id`. Note that this `build-id` is exactly what is reported by `readelf -n | grep "Build ID"` and not the `build-uuid` used by Breakpad. Then in gdb use `show debug-file-directory` to get the `` and `set debug-file-directory :/debug`. The `offset`s used here are the offsets of the corresponding module from the output of `minidump_stackwalk` or (equivalently) the value of `ModuleListStream::Modules[]::BaseOfImage` from the minidump file (which can be read with the structure definition). ``` $ gdb (gdb) file (gdb) show debug-file-directory (gdb) set debug-file-directory :/debug (gdb) symbol-file -o (gdb) core-file ``` Running the commands in this order avoids needing to load the symbols twice and maps the `` to the expected location. To add an additional shared library it is possible to `(gdb) add-symbol-file -o ` Source paths in Chrome builds are relative to the `out/` directory. If you have a Chromium checkout at or around when the Chrome build was created, it can be added to the debugger search path, like ``` (gdb) directory /chromium/src/out// ```