
This also updates it for the new crash keys v2 API. The documentation previously lived at https://dev.chromium.org/developers/debugging-with-crash-keys. No-Try: True Bug: 598854 Change-Id: I9b672bb40bc6cf7e396644ed3af6de5a90f0478e Reviewed-on: https://chromium-review.googlesource.com/848538 Commit-Queue: Robert Sesek <rsesek@chromium.org> Reviewed-by: Mark Mentovai <mark@chromium.org> Cr-Commit-Position: refs/heads/master@{#527345}
154 lines
6.7 KiB
Markdown
154 lines
6.7 KiB
Markdown
# Debugging with Crash Keys
|
||
|
||
Chrome is client-side software, which means that sometimes there are bugs that
|
||
can occur only on users' machines ("in production") that cannot be reproduced by
|
||
test or software engineering. When this happens, it's often helpful to gather bug-
|
||
specific data from production to help pinpoint the cause of the crash. The crash
|
||
key logging system is a generic method to help do that.
|
||
|
||
[TOC]
|
||
|
||
## High-Level Overview
|
||
|
||
The core of the crash key logging system is in [//components/crash/core/common/crash_key.h](https://cs.chromium.org/chromium/src/components/crash/core/common/crash_key.h),
|
||
which declares a `crash_reporter::CrashKeyString` class. Every crash key has
|
||
an associated value maximum length and a string name to identify it. The maximum
|
||
length is specified as a template parameter in order to allocate that amount of
|
||
space for the value up-front. When a process is crashing, memory corruption can
|
||
make it unsafe to call into the system allocator, so pre-allocating space for
|
||
the value defends against that.
|
||
|
||
When a crash key is set, the specified value is copied to its internal storage.
|
||
And if the process subsequently crashes, the name-value tuple is uploaded as
|
||
POST form-multipart data when the crash report minidump is uploaded to the
|
||
Google crash reporting system. (The data therefore are only accessible to those
|
||
with access to crash reports internally at Google). For platforms that use
|
||
[Crashpad](https://crashpad.chromium.org) as the crash reporting platform, the
|
||
crash keys are also stored in the minidump file itself. For platforms that use
|
||
Breakpad, the keys are only available at upload.
|
||
|
||
The crash key system is used to report some common pieces of data, not just
|
||
things that happen in exceptional cases: the URL of the webpage, command line
|
||
switches, active extension IDs, GPU vendor information, experiment/variations
|
||
information, etc.
|
||
|
||
## Getting Started with a Single Key-Value Pair
|
||
|
||
Imagine you are investigating a crash, and you want to know the value of some
|
||
variable when the crash occurs; the crash key logging system enables you to do
|
||
just that.
|
||
|
||
#### 1. Declare the Crash Key
|
||
|
||
A crash key must be allocated using static storage duration, so that there is
|
||
space for the value to be set. This can be done as a static variable in the
|
||
global or function scope, or in an anonymous namespace:
|
||
|
||
static crash_reporter::CrashKeyString<32> crash_key_one("one");
|
||
|
||
namespace {
|
||
crash_reporter::CrashKeyString<64> crash_key_two("two");
|
||
}
|
||
|
||
void DoSomething(const std::string& arg) {
|
||
static crash_reporter::CrashKeyString<8> three("three");
|
||
crash_key_two.Set(arg);
|
||
three.Set("true");
|
||
}
|
||
|
||
The template argument specifies the maximum length a value can be, and it
|
||
should include space for a trailing NUL byte. Values must be C-strings and
|
||
cannot have embedded NULs. The constructor argument is the name of the
|
||
crash key, and it is what you will use to identify your data in uploaded
|
||
crash reports.
|
||
|
||
If you need to declare an array of crash keys (e.g., for recording N values
|
||
of an array), you can use a constructor tag to avoid warnings about `explicit`:
|
||
|
||
static ArrayItemKey = crash_reporter::CrashKeyString<32>;
|
||
static ArrayItemKey crash_keys[] = {
|
||
{"array-item-1", ArrayItemKey::Tag::kArray},
|
||
{"array-item-2", ArrayItemKey::Tag::kArray},
|
||
{"array-item-3", ArrayItemKey::Tag::kArray},
|
||
{"array-item-4", ArrayItemKey::Tag::kArray},
|
||
};
|
||
|
||
The crash key system will require your target to have a dependency on
|
||
`//components/crash/core/common:crash_key`. If you encounter link errors for
|
||
unresolved symbols to `crashpad::Annotation::SetSize(unsigned int)`, adding
|
||
the dependency will resolve them.
|
||
|
||
#### 2. Set the Crash Key
|
||
|
||
After a key has been allocated, its `Set(base::StringPiece)` and
|
||
`Clear()` methods can be used to record and clear a value. In addition,
|
||
crash_key.h provides a `ScopedCrashKeyString` class to set the value for the
|
||
duration of a scope and clear it upon exiting.
|
||
|
||
#### 3. Seeing the Data
|
||
|
||
Using <http://go/crash> (internal only), find the crash report signature related
|
||
to your bug, and click on the "N of M" reports link to drill down to
|
||
report-specific information. From there, select a report and go to the
|
||
"Product Data" section to view all the crash key-value pairs.
|
||
|
||
## Dealing with DEPS
|
||
|
||
Not all targets in the Chromium source tree are permitted to depend on the
|
||
`//components/crash/core/common:crash_key` target due to DEPS file
|
||
`include_rules`.
|
||
|
||
If the crash key being added is only a temporary debugging aid to track down a
|
||
crash, consider adding the dependency temporarily and removing it when done.
|
||
A specific include rule can be added for crash_key.h:
|
||
|
||
# DEPS
|
||
include_rules = [
|
||
'+components/crash/core/common/crash_key.h',
|
||
]
|
||
|
||
Then simply remove it (and the BUILD.gn dependency) once the crash is resolved
|
||
and the crash key deleted.
|
||
|
||
If this crash key is more permanent, then there is an alternate API in //base
|
||
that can be used. This API is used by the //content module to set its permanent
|
||
crash key information. Note however that the base-level API is more limited in
|
||
terms of features and flexibility. See the header documentation in
|
||
[//base/debug/crash_logging.h](https://cs.chromium.org/chromium/src/base/debug/crash_logging.h)
|
||
for usage examples.
|
||
|
||
## Advanced Topics: Stack Traces
|
||
|
||
Now imagine a scenario where you have a use-after-free. The crash reports coming
|
||
in do not indicate where the object being used was initially freed, however,
|
||
just where it is later being dereferenced. To make debugging easier, it would be
|
||
nice to have the stack trace of the destructor, and the crash key system works
|
||
for that, too.
|
||
|
||
#### 1. Declare the Crash Key
|
||
|
||
Declaring the crash key is no different than written above, though special
|
||
attention should be paid to the maximum size argument, which will affect the
|
||
number of stack frames that are recorded. Typically a value of _1024_ is
|
||
recommended.
|
||
|
||
#### 2. Set the Crash Key
|
||
|
||
To set a stack trace to a crash key, use the `SetCrashKeyStringToStackTrace()`
|
||
function in crash_logging.h:
|
||
|
||
Usemeafterfree::~Usemeafterfree() {
|
||
static crash_reporter::CrashKeyString<1024> trace_key("uaf-dtor-trace");
|
||
crash_reporter::SetCrashKeyStringToStackTrace(&trace_key,
|
||
base::debug::StackTrace());
|
||
}
|
||
|
||
#### 3. Seeing the Data
|
||
|
||
Unlike with the previous example, a stack trace will just be a string of
|
||
hexadecimal addresses. To turn the addresses back into symbols use,
|
||
<http://go/crsym> (internal instance of <https://github.com/chromium/crsym/>).
|
||
Using the **Crash Key** input type, give it a crash report ID and the name of
|
||
your crash key. Crsym will then fetch the symbol data from the internal crash
|
||
processing backends and return a formatted, symbolized stack trace.
|