Collect minidump documentation into one place

Change-Id: I6ecc111cdf78b4f2ab385674dd3e036adcdd5df0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4566687
Reviewed-by: Herb Derby <herb@google.com>
Commit-Queue: Ben Wagner <bungeman@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1164352}

docs/linux/debugging_minidump.md

@@ -0,0 +1,184 @@
+# Linux Minidump Code Lab
+
+[TOC]
+
+## About Minidumps
+
+Minidump is a file format for storing parts of a program's state for later
+inspection. [Microsoft's
+Documentation](https://docs.microsoft.com/en-us/windows/desktop/api/minidumpapiset/)
+defines the format though the [Rust
+Documentation](https://docs.rs/minidump/latest/minidump/format/index.html)
+is sometimes easier to navigate. The minidump implementation and tools used by
+Chrome are
+[Breakpad](https://chromium.googlesource.com/breakpad/breakpad/+/refs/heads/main/README.md)
+and
+[Crashpad](https://chromium.googlesource.com/crashpad/crashpad/+/main/README.md).
+However, the tools of interest here are from the Breakpad project.
+
+## Create a Minidump
+
+When Chrome crashes it writes out a minidump file. The minidump file is written
+under the application product directory. On Linux this is
+`<XDG_CONFIG_HOME>/<app-name>/Crash Reports`. The default for `XDG_CONFIG_HOME`
+is `~.config`. Common `<app-name>`s are `chromium`, `google-chrome`,
+`google-chrome-beta`, and `google-chrome-unstable`. A typical example is
+`~.config/google-chrome/Crash Reports`. When a minidump is uploaded it will be
+moved between the `new`, `pending`, and `completed` subdirectories. The minidump
+file is named something like `<uuid>.dmp`. If the minidump is uploaded to the
+crash reporting system, the `<uuid>.meta` file will contain the crash report id.
+Those with access can find the uploaded report at `go/crash/<report-id>`, where
+the minidump file will be available with a name like
+`upload_file_minidump-<report-id>.dmp`.
+
+To create a minidump, you can use a local build of Chromium or a release
+version of Chrome. Run the browser with the environment variable
+`CHROME_HEADLESS=1`, which enables crash dumping but prevents crash dumps from
+being uploaded and deleted. Something like `$ env CHROME_HEADLESS=1
+./out/debug/chrome-wrapper` or `$ env CHROME_HEADLESS=1
+/opt/google/chrome/google-chrome`. Navigate to `chrome://crash` to trigger a
+crash in the renderer process or reproduce your current crash bug. A crash dump
+file should appear in the `Crash Reports` directory.
+
+## Inspect the Minidump
+
+To get an idea about what is in a minidump file, install the Okteta hex editor
+and add the [Minidump Structure
+Definition](https://github.com/bungeman/structures/tree/main/okteta-minidump).
+Open the minidump previously created and explore the information it contains.
+
+One quirk to notice is that there is a `ThreadListStream` which contains
+`MINIDUMP_THREAD`s which contain a `MINIDUMP_THREAD_CONTEXT` and an
+`ExceptionStream` which also contains a `MINIDUMP_THREAD_CONTEXT`. The thread
+list contains the thread contexts as they existed when the crash reporter was
+running. The exception's thread context is the state of the crashing thread at
+the time that it crashed, which is generally the most interesting thread
+context. When using the Breakpad tools for Linux (like `minidump_stackwalk` and
+`minidump-2-core`) the thread context from the exception record is used in place
+of the thread context associated with the corresponding thread.
+
+Each `MINIDUMP_THREAD` contains a `StackMemoryRva` which is a reference to to a
+copy of the stack on that thread at the time the crash handler was running.
+Parsing a stack usefully requires additional debug information.
+`minidump_stackwalk` or a debugger may be used to parse the stack memory to
+create a usable trace.
+
+## Get the Tools
+
+From a Chromium checkout `ninja -C out/release minidump-2-core
+minidump_stackwalk dump_syms`. From a [Breakpad
+checkout](https://chromium.googlesource.com/breakpad/breakpad/) `make`. It can
+be useful to use Breakpad directly on machines where one does not already have
+a Chromium checkout.
+
+When working at this level, one will also want to have `readelf` and
+`objdump` available, which are available from most distributions.
+
+## Get Executables and Symbols
+
+In addition to the minidump, you will need the exact executables of Chromium or
+Chrome which produced the minidump and those executable's symbols. If the
+minidump was created locally, you already have the executables. Symbols for
+Google Chrome's official builds are available from
+`https://edgedl.me.gvt1.com/chrome/linux/symbols/google-chrome-debug-info-linux64-${VERSION}.zip`
+where `${VERSION}` is any version of Google Chrome that has recently been served
+to Stable, Beta, or Unstable (Dev) channels on Linux, like `114.0.5696.0`. Those
+with access can find both executables and symbols for unreleased builds at
+`go/chrome-symbols`.
+
+For symbols outside of Chrome (like when the crash is happening in a shared
+library) then symbols for the files of interest must be found. If the minidump
+was created locally then install the symbol packages from your distribution.  If
+not, you will need to track down the exact symbol files, which can be an
+interesting exercise. For some distributions using the
+[debuginfod](https://sourceware.org/elfutils/Debuginfod.html) system can be
+quite helpful.
+
+To ensure the correct binaries and debug symbols are used, the minidump contains
+the build-id for each loaded module in the `ModuleListStream` in the
+`CvRecordRva`'s `Signature`. This build-id is matched against a note
+section of type `NT_GNU_BUILD_ID`, usually named `.note.gnu.build-id` in the
+executable and symbol files. This note can be inspected with `readelf -n
+<file>` like `readelf -n chrome` or `readelf -n chrome.debug` and looking for
+the `.note.gnu.build-id` section. `readelf` reports the `Build ID` as the flat
+bytes in the note, but Breakpad binaries like `stackwalk_minidimp` and
+`dump_syms` will report and expect this truncated to a formatted Type 2 GUID
+(without dashes). This means `readelf` will output a `<build-id>` like
+33221100554477668899AABBCCDDEEFFXXXXXXXX but crashpad binaries will expect and
+report this as a `<build-uuid>` of 00112233445566778899AABBCCDDEEFF.
+
+The `.gnu_debuglink` section states which debug symbol file to use with a
+striped binary. For example `readelf --string-dump=.gnu_debuglink chrome`
+produces `chrome.debug`. This can be helpful to know for libraries with
+interesting debug symbol setup, like libc.so.6.
+
+## Create Symbolized Stack
+
+Given a minidump with the name `mini.dmp`
+
+`minidump_stackwalk mini.dmp > mini.stackwalk.nosym`
+
+This will produce a mostly unsymbolized summary of the crash. To symbolize, look
+toward the bottom of the output for `WARNING: No symbols, <file>, <build-uuid>`.
+For each `<file>` which is of interest, `mkdir -p symbols/<file>/<build-uuid>` then
+`dump_syms <file> <directory-with-file.debug> >
+symbols/<file>/<build-uuid>/<file>.sym`. Ensure this output `<file>.sym`
+contains the expected `<build-uuid>`. Then re-run `minidump_stackwalk` but
+with the symbols directory, like `minidump_stackwalk mini.dmp symbols/ >
+mini.stackwalk`.
+
+The output of `minidump_stackwalk` is often quite useful and enough to track
+down many issues. However, it does not fully use all of the information from
+DWARF, so it is possible sometimes to get much better stack traces from a full
+debugger like gdb. This is particularly true when functions have been
+aggressively inlined.
+
+## Create Core File
+
+`minidump-2-core mini.dmp > mini.core`
+
+## Loading into GDB
+
+This works best if the binaries, symbols, and core files are all in different
+directories to prevent gdb from automatically loading them into the wrong
+locations. This is also generally necessary when using a system installed
+version of Chrome. For full details see [the gdb
+manual](https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html).
+The easiest way is to rename and move the .debug files to a directory structure
+like `<debugdir>/debug/.build-id/nn/nnnnnnnn.debug` where `nn` are the first
+two hex characters of the `build-id`, and `nnnnnnnn` are the rest of the hex
+characters of the `build-id`. Note that this `build-id` is exactly what is
+reported by `readelf -n <binary> | grep "Build ID"` and not the `build-uuid`
+used by Breakpad. Then in gdb use `show debug-file-directory` to get the
+`<previous-directories>` and `set debug-file-directory
+<previous-directories>:<debugdir>/debug`.
+
+The `offset`s used here are the offsets of the corresponding module from the
+output of `minidump_stackwalk` or (equivalently) the value of
+`ModuleListStream::Modules[]::BaseOfImage` from the minidump file (which can be
+read with the structure definition).
+
+```
+$ gdb
+(gdb) file <executable>
+(gdb) show debug-file-directory
+<previous directories>
+(gdb) set debug-file-directory <previous directories>:<debugdir>/debug
+(gdb) symbol-file <executable> -o <executable-offset>
+(gdb) core-file <mini.core>
+```
+
+Running the commands in this order avoids needing to load the symbols twice and
+maps the `<executable>` to the expected location.
+
+To add an additional shared library it is possible to
+`(gdb) add-symbol-file <shared-library> -o <shared-library-offset>`
+
+Source paths in Chrome builds are relative to the `out/<build>` directory. If you
+have a Chromium checkout at or around when the Chrome build was created, it can
+be added to the debugger search path, like
+
+```
+(gdb) directory <path-to-chromium>/chromium/src/out/<build>/
+```
+