Rewrite the macOS debugging doc.
This removes a lot of outdated information and provides new content. Change-Id: Ia74840e0a4ee63bd22bcadb4d61436b0574163ba Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2826397 Commit-Queue: Robert Sesek <rsesek@chromium.org> Reviewed-by: Leonard Grey <lgrey@chromium.org> Cr-Commit-Position: refs/heads/master@{#872582}
This commit is contained in:

committed by
Chromium LUCI CQ

parent
2e32361a01
commit
3b731b15e6
@ -2,302 +2,252 @@
|
|||||||
|
|
||||||
[TOC]
|
[TOC]
|
||||||
|
|
||||||
## Resources
|
## Debug vs. Release Builds
|
||||||
|
|
||||||
The [Mac OS X Debugging Magic
|
Debug builds are the default configuration for Chromium and can be explicitly
|
||||||
Technote](http://developer.apple.com/technotes/tn2004/tn2124.html) contains a
|
specified with `is_debug=true` in the `args.gn` file of the out directory. Debug
|
||||||
wealth of information about various debugging options built in to macOS.
|
builds are larger and non-portable because they default to
|
||||||
|
`is_component_build=true`, but they contain full debug information.
|
||||||
|
|
||||||
IMPORTANT: By default, Xcode has the "Load Symbols Lazily" preference set. As a
|
If you set `is_debug=false`, a release build will be created with no symbol
|
||||||
result, any symbols not in the main static library (99% of our code) won't be
|
information, which cannot be used for effective debugging.
|
||||||
visible to set breakpoints. The result is that you set breakpoints in the editor
|
|
||||||
window and they're ignored entirely when you run. The fix, however, is very
|
|
||||||
simple! Uncheck the "Load Symbols Lazily" checkbox in the "Debugging" panel in
|
|
||||||
preferences. Now all your breakpoints will work, at the expense of a little
|
|
||||||
longer load time in gdb. Well worth it, if you ask me.
|
|
||||||
|
|
||||||
ALSO IMPORTANT: If you include `fast_build=1` in your `GYP_DEFINES`, there is an
|
A middle-ground is to set `symbol_level=1`, which will produce a minimal symbol
|
||||||
excellent chance the symbols you'll need for debugging will be stripped! You may
|
table, capable of creating backtraces, but without frame-level local variables.
|
||||||
save yourself a lot of heartache if you remove this, rerun `gyp_chromium` and
|
This is faster to build than a debug build, but it is less useful for debugging.
|
||||||
rebuild before proceeding.
|
|
||||||
|
|
||||||
## Disabling ReportCrash
|
When doing an `is_official_build=true` build (which is meant for producing
|
||||||
|
builds with full compiler optimization suitable for shipping to users),
|
||||||
|
`enable_dsyms` and `enable_stripping` both get set to `true`. The binary itself
|
||||||
|
will be stripped of its symbols, but the debug information will be saved off
|
||||||
|
into a dSYM file. Producing a dSYM is rather slow, so it is uncommon for
|
||||||
|
developers to build with this configuration.
|
||||||
|
|
||||||
macOS helpfully tries to write a crash report every time a binary crashes –
|
### Chrome Builds
|
||||||
which happens for example when a test in unit\_tests fails. Since Chromium's
|
|
||||||
debug binaries are huge, this takes forever. If this happens, "ReportCrash" will
|
|
||||||
be the top cpu consuming process in Activity Monitor. You should disable
|
|
||||||
ReportCrash while you work on Chromium. Run `man ReportCrash` to learn how to do
|
|
||||||
this on your version of macOS. On 10.8, the command is
|
|
||||||
|
|
||||||
launchctl unload -w /System/Library/LaunchAgents/com.apple.ReportCrash.plist
|
The official Google Chrome build has published dSYMs that can be downloaded with
|
||||||
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.ReportCrash.Root.plist
|
the script at `tools/mac/download_symbols.py` or by using the LLDB integration
|
||||||
|
at `tools/lldb/lldb_chrome_symbols.py`.
|
||||||
|
|
||||||
Yes, you need to run this for both the normal user and the admin user.
|
However, the official Chrome build is
|
||||||
|
[codesigned](../../chrome/installer/mac/signing/README.md) with the `restrict`
|
||||||
|
and `runtime` options, which generally prohibit debuggers from attaching.
|
||||||
|
|
||||||
## Processing Apple Crash Reports
|
In order to debug production/released Chrome, you need to do one of two things:
|
||||||
|
|
||||||
If you get a Google Chrome crash report caught by ReportCrash/OS X, it will not
|
1. Disable [System Integrity
|
||||||
have symbols (every frame will be ChromeMain). To get a symbolized stack trace,
|
Protection](https://developer.apple.com/documentation/security/disabling_and_enabling_system_integrity_protection),
|
||||||
use the internal [crsym](http://goto.google.com/crsym) tool by simply pasting
|
by:
|
||||||
the contents of an entire Apple crash report.
|
1. Rebooting into macOS recovery mode
|
||||||
|
2. Launching Terminal
|
||||||
|
3. Running `csrutil enable --without debug`
|
||||||
|
4. Rebooting
|
||||||
|
2. Stripping or force-re-codesigning the binary to not use those options:
|
||||||
|
`codesign --force --sign - path/to/Google\ Chrome.app`
|
||||||
|
|
||||||
## Debugging the renderer process
|
If you will frequently debug official builds, (1) is recommended. Note that
|
||||||
|
disabling SIP reduces the overall security of the system, so your system
|
||||||
|
administrator may frown upon it.
|
||||||
|
|
||||||
Xcode's built in gdb wrapper doesn't allow you to debug more than one process at
|
## The Debugger
|
||||||
once and doesn't deal very well with debugging Chrome's subprocesses directly.
|
|
||||||
There are two different ways around this:
|
|
||||||
|
|
||||||
### (a) Run Chrome in a single process
|
The debugger on macOS is `lldb` and it is included in both a full Xcode install
|
||||||
|
and the Command Line Tools package. There are two ways to use LLDB: either
|
||||||
|
launching Chromium directly in LLDB, or attaching to an existing process:
|
||||||
|
|
||||||
(NOTE: this option is not recommended any more -- Chrome's single-process mode
|
lldb ./out/debug/Chromium.app/Contents/MacOS/Chromium
|
||||||
is neither supported nor tested.)
|
lldb -p <pid>
|
||||||
|
|
||||||
1. Edit the Executable settings for the Chromium app (make it the current
|
LLDB has an extensive help system which you can access by typing `help` at the
|
||||||
executable, then choose Project > Edit Active Executable).
|
`(lldb)` command prompt. The commands are organized into a functional hierarchy,
|
||||||
2. Switch to the Arguments tab and press the "+" button under the arguments
|
and you can explore the subcommands via `(lldb) help breakpoint`, etc. Commands
|
||||||
list
|
can take arguments in a command-line flag style. Many commands also have short
|
||||||
3. Type "`--single-process`" in the list.
|
mnemonics that match the `gdb` equivalents. You can also just use enough letters
|
||||||
|
to form a unique prefix of the command hierarchy. E.g., these are equivalent:
|
||||||
|
|
||||||
From now on Chromium will launch in single-process mode when invoked through
|
(lldb) help breakpoint set
|
||||||
this Xcode project, and the debugger will work fine. This obviously changes the
|
(lldb) h br s
|
||||||
apps behavior slightly, but for most purposes the differences aren't
|
|
||||||
significant. If they are, though, you'll need to…
|
|
||||||
|
|
||||||
### (b) or, Attach Xcode's debugger to a renderer process after launch
|
When the program is running, you can use **Ctrl-C** to interrupt it and pause
|
||||||
|
the debugger.
|
||||||
|
|
||||||
1\. Launch the main executable from the Terminal (**not** through Xcode) and
|
### Passing Arguments
|
||||||
pass in the `--renderer-startup-dialog` flag on the command line. On
|
|
||||||
macOS this causes the renderer to print a message with its PID and then call
|
|
||||||
pause() immediately up on startup. This has the effect of pausing execution of
|
|
||||||
the renderer until the process receives a signal (such as attaching the
|
|
||||||
debugger).
|
|
||||||
|
|
||||||
e.g.
|
To pass arguments to LLDB when starting Chromium, use a `--`:
|
||||||
|
|
||||||
$ ~/dev/chrome//src/xcodebuild/Debug/Chromium.app/Contents/MacOS/Chromium --renderer-startup-dialog
|
lldb ./out/debug/Chromium.app/contents/MacOS/Chromium -- --renderer-startup-dialog
|
||||||
|
|
||||||
the output should look like:
|
### Breakpoints
|
||||||
|
|
||||||
...
|
Simple function-name breakpoints can be specified with a short mnemonic:
|
||||||
[33215:2055:244180145280185:WARNING:/Users/Shared/bla/chrome/src/chrome/renderer/renderer_main.cc(48)]
|
|
||||||
Renderer (33215) paused waiting for debugger to attach @ pid
|
|
||||||
...
|
|
||||||
|
|
||||||
So *33215* is the PID of the renderer process in question.
|
(lldb) b BrowserWindow::Close
|
||||||
|
|
||||||
2\. Open chrome.xcodeproj in Xcode and select Run -> Attach To Process ->
|
But there are a range of other options for setting breakpoints using the, such
|
||||||
Process ID ..
|
as:
|
||||||
|
* `-t` to limit the breakpoint to a specific thread
|
||||||
|
* `-s` to specify a specific shared library, if the same symbol name is exported
|
||||||
|
by multiple libraries
|
||||||
|
* `-o` for a one-shot breakpoint (delete after first hit)
|
||||||
|
|
||||||
3\. Click OK and off you go...
|
See `(lldb) help br set` for full details.
|
||||||
|
|
||||||
## Debugging out-of-process tests:
|
### Navigating the Stack
|
||||||
|
|
||||||
Similar to debugging the renderer process, simply attaching gdb to a
|
When the debugger is paused, you can get a backtrace by typing `bt`. To navigate
|
||||||
|
the stack by-1 type either `up` or `down`. You can also jump to a specific index
|
||||||
|
in the stack by typing `f #` (short for `frame select #`).
|
||||||
|
|
||||||
|
To see local variables, type `v` (short for `frame variable`).
|
||||||
|
|
||||||
|
### Examining Execution
|
||||||
|
|
||||||
|
To step through the program, use:
|
||||||
|
|
||||||
|
* `s` or `si` for step-in
|
||||||
|
* `n` for step-over
|
||||||
|
* `c` to continue (resume or go to next breakpoint)
|
||||||
|
|
||||||
|
### Printing Values
|
||||||
|
|
||||||
|
To print values, use the `p <value>` command, where `<value>` can be a variable,
|
||||||
|
a variable expression like `object->member_->sub_member_.value`, or an address.
|
||||||
|
|
||||||
|
If `<value>` is a pointer to a structure, `p <value>` will usually just print
|
||||||
|
the address. To show the contents of the structure, dereference the value. E.g.:
|
||||||
|
|
||||||
|
(lldb) p item
|
||||||
|
(HistoryMenuBridge::HistoryItem *) $3 = 0x0000000245ef5b30
|
||||||
|
(lldb) p *item
|
||||||
|
(HistoryMenuBridge::HistoryItem) $4 = {
|
||||||
|
title = u"Google"
|
||||||
|
url = {
|
||||||
|
spec_ = "https://www.google.com/"
|
||||||
|
is_valid_ = true
|
||||||
|
…
|
||||||
|
|
||||||
|
Note above that LLDB has also stored the results of the expressions passed to
|
||||||
|
`p` into the variables `$3` and `$4`, which can be referenced in other LLDB
|
||||||
|
expressions.
|
||||||
|
|
||||||
|
Often (and always when printing addresses) there is not type information to
|
||||||
|
enable printing the full structure of the referenced memory. In these cases, use
|
||||||
|
a C-style cast:
|
||||||
|
|
||||||
|
(lldb) p 0x0000000245ef5b30 # Does not have type information
|
||||||
|
(long) $5 = 9763248944
|
||||||
|
(lldb) p (HistoryMenuBridge::HistoryItem*)0x0000000245ef5b30
|
||||||
|
(HistoryMenuBridge::HistoryItem *) $6 = 0x0000000245ef5b30
|
||||||
|
(lldb) p *$6
|
||||||
|
(HistoryMenuBridge::HistoryItem) $7 = {
|
||||||
|
title = u"Google"
|
||||||
|
…
|
||||||
|
|
||||||
|
* For printing Cocoa NSObjects, use the `po` command to get the `-[NSObject description]`.
|
||||||
|
* If `uptr` is a `std::unique_ptr`, the address it wraps is accessible as
|
||||||
|
`uptr.__ptr_.__value_`.
|
||||||
|
* To pretty-print `std::u16string`, follow the instructions [here](../lldbinit.md).
|
||||||
|
|
||||||
|
## Multi-Process Debugging
|
||||||
|
|
||||||
|
Chrome is split into multiple processes, which can mean that the logic you want
|
||||||
|
to debug is in a different process than the main browser/GUI process. There are
|
||||||
|
a few ways to debug the multi-process architecture, discussed below.
|
||||||
|
|
||||||
|
### (a) Attach to a Running Process
|
||||||
|
|
||||||
|
You can use Chrome's Task Manager to associate specific sites with their PID.
|
||||||
|
Then simply attach with LLDB:
|
||||||
|
|
||||||
|
lldb -p <pid>
|
||||||
|
|
||||||
|
Or, if you have already been debugging a Chrome process and want to retain your
|
||||||
|
breakpoints:
|
||||||
|
|
||||||
|
(lldb) attach <pid>
|
||||||
|
|
||||||
|
### (b) Debug Process Startup
|
||||||
|
|
||||||
|
If you need to attach early in the child process's lifetime, you can use one of
|
||||||
|
these startup-dialog switches for the relevant process type:
|
||||||
|
|
||||||
|
* `--renderer-startup-dialog`
|
||||||
|
* `--utility-startup-dialog`
|
||||||
|
* `--renderer-startup-dialog`
|
||||||
|
|
||||||
|
After the process launches, it will print a message like this to standard error:
|
||||||
|
|
||||||
|
[80156:775:0414/130021.862239:ERROR:content_switches_internal.cc(112)] Renderer (80156) paused waiting for debugger to attach. Send SIGUSR1 to unpause.
|
||||||
|
|
||||||
|
Then attach the the process like above in **(a)**, using the PID in parenthesis
|
||||||
|
(e.g. *80156* above).
|
||||||
|
|
||||||
|
### (c) Run Chrome in a single process
|
||||||
|
|
||||||
|
> This option is not recommended. Single-process mode is not tested and is
|
||||||
|
> frequently broken.
|
||||||
|
|
||||||
|
Chrome has an option to run all child processes as threads inside a single
|
||||||
|
process, using the `--single-process` command line flag. This can make debugging
|
||||||
|
easier.
|
||||||
|
|
||||||
|
## Debugging Out-of-Process Tests:
|
||||||
|
|
||||||
|
Similar to debugging the renderer process, simply attaching LLDB to a
|
||||||
out-of-process test like browser\_tests will not hit the test code. In order to
|
out-of-process test like browser\_tests will not hit the test code. In order to
|
||||||
debug a browser test, you need to run the test binary with "`--single_process`"
|
debug a browser test, you need to run the test binary with "`--single_process`"
|
||||||
(note the underscore in `single_process`). Because you can only run one browser
|
(note the underscore in `single_process`). Because you can only run one browser
|
||||||
test in the same process, you're probably going to need to add `--gtest_filter`
|
test in the same process, you're probably going to need to add `--gtest_filter`
|
||||||
as well. So your command will look like this:
|
as well. So your command will look like this:
|
||||||
|
|
||||||
/path/to/src/xcodebuild/Debug/browser_tests --single_process --gtest_filter=GoatTeleporterTest.DontTeleportSheep
|
/path/to/src/out/debug/browser_tests --single_process --gtest_filter=GoatTeleporterTest.DontTeleportSheep
|
||||||
|
|
||||||
## UI Debugging
|
## Working with Xcode
|
||||||
|
|
||||||
For UI Debugging, F-Script Anywhere is very useful.
|
If you'd prefer to use Xcode GUI to use the debugger, there are two options:
|
||||||
Read <https://sites.google.com/a/chromium.org/dev/developers/f-script-anywhere>
|
|
||||||
for more information.
|
|
||||||
|
|
||||||
## Building with Ninja, Debugging with Xcode
|
### (1) Empty Xcode Project
|
||||||
|
|
||||||
See [the
|
This approach creates an empty Xcode project that only provides a GUI debugger:
|
||||||
instructions](https://sites.google.com/a/chromium.org/dev/developers/how-tos/debugging-on-os-x/building-with-ninja-debugging-with-xcode)
|
|
||||||
that discuss that scenario.
|
|
||||||
|
|
||||||
## Temporarily disabling the Sandbox
|
1. Select **File** > **New** > **Project...** and make a new project. Dump it
|
||||||
|
anywhere, call it anything. It doesn't matter.
|
||||||
|
2. Launch Chromium.
|
||||||
|
3. In Xcode, select **Debug** > **Attach to Process** > *Chromium*.
|
||||||
|
4. You can now pause the process and set breakpoints. The debugger will also
|
||||||
|
activate if a crash occurs.
|
||||||
|
|
||||||
Disabling the sandbox can sometimes be useful when debugging, this can be
|
### (2) Use *gn*
|
||||||
achieved by passing the --no-sandbox flag on the command line. This will, for
|
|
||||||
example, allow writing out debugging information to a file from the Renderer
|
|
||||||
Process.
|
|
||||||
e.g.
|
|
||||||
|
|
||||||
$ ~/dev/chrome//src/xcodebuild/Debug/Chromium.app/Contents/MacOS/Chromium --no-sandbox
|
1. Tell `gn` to generate an Xcode project for your out directory:
|
||||||
|
`gn gen --ide=xcode out/debug`
|
||||||
|
2. Open *out/debug/all.xcodeproj*
|
||||||
|
3. Have it automatically generate schemes for you
|
||||||
|
4. You can now build targets from within Xcode, which will simply call out to
|
||||||
|
`ninja` via an Xcode script. But the resulting binaries are available as
|
||||||
|
debuggable targets in Xcode.
|
||||||
|
|
||||||
## Tips on Debugging the Renderer Sandbox
|
Note that any changes to the .xcodeproj will be overwritten; all changes to the
|
||||||
|
build system need to be done in GN.
|
||||||
|
|
||||||
Launch chrome with the `--enable-sandbox-logging` flag. This will cause a message
|
## Debugging the Sandbox
|
||||||
to be printed to /var/log/system.log every time an operation is denied by the
|
|
||||||
Sandbox (you can use Console.app to watch logfiles). This is really useful for
|
|
||||||
debugging and can often provide an explanation for very puzzling problems.
|
|
||||||
|
|
||||||
You can also get the Sandbox to send a *SIGSTOP* to a process when the sandbox
|
See the page on [sandbox debugging](sandbox_debugging.md).
|
||||||
denies functionality. This allows you to attach with a debugger and continue
|
|
||||||
the execution from where it left off:
|
|
||||||
|
|
||||||
$ sandbox-exec -p '(version 1) (allow default) (deny file-write\* (regex "foo") (with send-signal SIGSTOP))' touch foo
|
## System Permission Prompts; Transparency, Consent, and Control (TCC)
|
||||||
|
|
||||||
## Breakpoints Not Getting Hit in gdb
|
When debugging issues with OS-mediated permissions (e.g. Location, Camera,
|
||||||
|
etc.), you need to launch Chromium with LaunchServices rather than through a
|
||||||
|
shell. If you launch Chromium as a subprocess of your terminal shell, the
|
||||||
|
permission requests get attributed to the terminal app rather than Chromium.
|
||||||
|
|
||||||
If a breakpoint you set isn't causing the debugger to stop, try one of these
|
To launch Chromium via launch services, use the `open(1)` command:
|
||||||
solutions:
|
|
||||||
|
|
||||||
- Uncheck "Load symbols lazily" In the Xcode -> Preferences -> Debugging
|
open ./out/debug/Chromium.app
|
||||||
dialog.
|
|
||||||
- Manually insert a call to `Debugger()` in the code, this will forcefully break
|
|
||||||
into the Debugger.
|
|
||||||
|
|
||||||
## Debugging in Release Mode
|
To pass command line arguments:
|
||||||
|
|
||||||
See "Preserving symbols in Release builds" below.
|
open ./out/debug/Chromium.app -- --enable-features=MyCoolFeature
|
||||||
|
|
||||||
### Preserving symbols in Release builds
|
|
||||||
|
|
||||||
Profiling tools like Shark and 'sample' expect to find symbol names in the
|
|
||||||
binary, but in Release builds most symbols are stripped out. You can preserve
|
|
||||||
symbols by temporarily changing the build process, by adding
|
|
||||||
`mac_strip_release=0` to your GYP\_DEFINES, rerunning gclient runhooks, and
|
|
||||||
rebuilding (changing this define only relinks the main binary, it doesn't
|
|
||||||
recompile everything).
|
|
||||||
|
|
||||||
*(The above "Debugging in Release Mode" trick with the .dSYM file might work for
|
|
||||||
Shark/sample too; I haven't tried it yet. —snej)*
|
|
||||||
|
|
||||||
## Using DTrace
|
|
||||||
|
|
||||||
jgm's awesome introductory article:
|
|
||||||
[http://www.mactech.com/articles/mactech/Vol.23/23.11/ExploringLeopardwithDTrace/index.html
|
|
||||||
](http://www.mactech.com/articles/mactech/Vol.23/23.11/ExploringLeopardwithDTrace/index.html)
|
|
||||||
Defining static probes on macOS:
|
|
||||||
[http://www.macresearch.org/tuning-cocoa-applications-using-dtrace-custom-static-probes-and-instruments
|
|
||||||
](http://www.macresearch.org/tuning-cocoa-applications-using-dtrace-custom-static-probes-and-instruments)
|
|
||||||
|
|
||||||
[http://www.brendangregg.com/dtrace.html\#Examples
|
|
||||||
](http://www.brendangregg.com/dtrace.html#Examples)[http://blogs.sun.com/bmc/resource/dtrace\_tips.pdf
|
|
||||||
](http://blogs.sun.com/bmc/resource/dtrace_tips.pdf)
|
|
||||||
|
|
||||||
DTrace examples on macOS: /usr/share/examples/DTTk
|
|
||||||
|
|
||||||
To get truss on macOS, use dtruss. That requires root, so I often sudo dtruss -p
|
|
||||||
and attach to a running nonroot program.
|
|
||||||
|
|
||||||
## Testing other locales
|
|
||||||
|
|
||||||
To test Chrome in a different locale, change your system locale via the System
|
|
||||||
Preferences. (Keep the preferences window open so that you can change the
|
|
||||||
locale back without needing to navigate through menus in a language you may not
|
|
||||||
know.)
|
|
||||||
|
|
||||||
## Memory/Heap Inspection
|
|
||||||
|
|
||||||
There are several low-level command-line tools that can be used to inspect
|
|
||||||
what's going on with memory inside a process.
|
|
||||||
|
|
||||||
'**heap**' summarizes what's currently in the malloc heap(s) of a process. (It
|
|
||||||
only works with regular malloc, of course, but Mac Chrome still uses that.) It
|
|
||||||
shows a number of useful things:
|
|
||||||
|
|
||||||
- How much of the heap is used or free
|
|
||||||
- The distribution of block sizes
|
|
||||||
- A listing of every C++, Objective-C and CoreFoundation class found in the
|
|
||||||
heap, with the number of instances, total size and average size.
|
|
||||||
|
|
||||||
It identifies C++ objects by their vtables, so it can't identify vtable-less
|
|
||||||
classes, including a lot of the lower-level WebCore ones like StringImpl. To
|
|
||||||
work around this I temporarily added the 'virtual' keyword to
|
|
||||||
WebCore::RefCounted's destructor method, which forces every ref-counted object
|
|
||||||
to include a vtable pointer identifying its class.
|
|
||||||
|
|
||||||
'**malloc\_history**' identifies the stack backtrace that allocated every malloc
|
|
||||||
block in the heap. It lists every unique backtrace together with its number of
|
|
||||||
blocks and their total size. It requires that the process use malloc stack
|
|
||||||
logging, which is enabled if the environment variable MallocStackLogging is set
|
|
||||||
when it launches. The 'env' command is handy for this:
|
|
||||||
|
|
||||||
$ env MallocStackLogging=1 Chromium.app/Contents/MacOS/Chromium
|
|
||||||
|
|
||||||
Then in another shell you run
|
|
||||||
|
|
||||||
$ malloc_history <pid> -all_by_size
|
|
||||||
|
|
||||||
Watch out: the output is *big*. I ran malloc\_history on a fairly bloated heap
|
|
||||||
and got 60MB of text.
|
|
||||||
|
|
||||||
'**leaks**' finds malloc blocks that have no pointers to them and are probably
|
|
||||||
leaked. It doesn't require MallocStackLogging, but it's more useful if it's on
|
|
||||||
because it can then show the backtrace that allocated each leaked block. (So far
|
|
||||||
I've seen only trivial leakage in Chrome.)
|
|
||||||
|
|
||||||
'**vmmap**' shows all the virtual-memory regions in the process's address space.
|
|
||||||
This is less useful since it doesn't say anything about individual malloc blocks
|
|
||||||
(except huge ones) but it can be useful for looking at things like static data
|
|
||||||
size, mapped files, and how much memory is paged out. I recommend the
|
|
||||||
"-resident" flag, which shows how much of each allocation is currently paged
|
|
||||||
into RAM. See the man page for details.
|
|
||||||
|
|
||||||
|
|
||||||
Notes:
|
|
||||||
|
|
||||||
- These are not going to be very useful on stripped binaries, and they're less
|
|
||||||
useful in release builds.
|
|
||||||
- All of these except vmmap take several *minutes* to run, apparently because
|
|
||||||
of the number of symbols in Chrome. They spend most of their time pegging
|
|
||||||
one CPU down inside system code that's reading symbol tables from the
|
|
||||||
binary. Be patient.
|
|
||||||
- There are GUI apps in /Developer that do a lot of the same things, such as
|
|
||||||
Instruments, MallocDebug and Shark. I (snej) personally find the
|
|
||||||
command-line tools easier to understand, but YMMV.
|
|
||||||
|
|
||||||
## **Working with minidumps**
|
|
||||||
|
|
||||||
[See this
|
|
||||||
page.](https://sites.google.com/a/chromium.org/dev/developers/crash-reports)
|
|
||||||
|
|
||||||
## CrMallocErrorBreak
|
|
||||||
|
|
||||||
If you are looking at a crash report that ends in `CrMallocErrorBreak`, then
|
|
||||||
either a `malloc` or `free` call has failed with the given stacktrace. Chromium
|
|
||||||
overrides the empty function `malloc_error_break` in macOS's Libc
|
|
||||||
with `CrMallocErrorBreak`. The system calls this function as a debugging aide
|
|
||||||
that we've made fatal because it catches useful memory errors. Specifically,
|
|
||||||
`CrMallocErrorBreak` will be called (resulting in a crash) under the following
|
|
||||||
circumstances:
|
|
||||||
|
|
||||||
- Attempting to free a pointer that was not allocated.
|
|
||||||
- Attempting to free the same pointer more than once.
|
|
||||||
- Freeing a pointer of size 0.
|
|
||||||
- Freeing an unaligned pointer.
|
|
||||||
- An internal checksum of the object being freed does not match. This usually
|
|
||||||
indicates heap corruption!
|
|
||||||
- Invalid checksums on the small or tiny free list. The system maintains a
|
|
||||||
list of small allocations that it reuses to speed up things like allocations
|
|
||||||
in a loop. A checksum mismatch usually indicates a use-after-free,
|
|
||||||
double-free, or heap corruption.
|
|
||||||
- Extra-large allocation failures. Normally all failures to allocate go
|
|
||||||
through `CrMallocErrorBreak` but are
|
|
||||||
not fatal because that is the job of Chromium's OOM killer. Extra-large
|
|
||||||
allocations go through a different path and are sometimes killed here
|
|
||||||
instead.
|
|
||||||
|
|
||||||
If you get a crash report that that ends in `CrMallocErrorBreak`,
|
|
||||||
it is likely not an issue with this feature. It is instead surfacing a
|
|
||||||
(sometimes serious) bug in your code or other code that is stomping on your
|
|
||||||
code's memory. Using Chromium's memory tools (ASan, HeapCheck, and Valgrind) is
|
|
||||||
a good start, if you can reproduce the problem.
|
|
||||||
|
|
||||||
## Enabling high-DPI (aka "HiDPI" or "Retina") modes on standard-DPI hardware.
|
|
||||||
|
|
||||||
Under macOS 10.7 and above it's possible to fake "HiDPI" modes on standard-DPI
|
|
||||||
hardware. This can be useful in testing up-scaling codepaths or high-DPI
|
|
||||||
resources.
|
|
||||||
|
|
||||||
1. Configure the OS to offer HiDPI modes:
|
|
||||||
- EITHER [follow Apple's instructions to enable high resolution
|
|
||||||
modes](http://developer.apple.com/library/mac/#documentation/GraphicsAnimation/Conceptual/HighResolutionOSX/Testing/Testing.html)
|
|
||||||
- OR run the command-line: `sudo defaults write /Library/Preferences/com.apple.windowserver DisplayResolutionEnabled -bool YES`
|
|
||||||
2. Open the System Preferences -> Displays panel, select Scaled mode and scroll to the bottom to see modes marked "(HiDPI)".
|
|
||||||
|
|
||||||
Looking for `gdb`? It's been replaced with `lldb`. Use that it instead.
|
|
||||||
|
|
||||||
## Taking CPU Samples
|
## Taking CPU Samples
|
||||||
|
|
||||||
@ -305,10 +255,11 @@ A quick and easy way to investigate slow or hung processes is to use the sample
|
|||||||
facility, which will generate a CPU sample trace. This can be done either in the
|
facility, which will generate a CPU sample trace. This can be done either in the
|
||||||
Terminal with the sample(1) command or by using Activity Monitor:
|
Terminal with the sample(1) command or by using Activity Monitor:
|
||||||
|
|
||||||
1. Open Activity Monitor
|
1. Open Activity Monitor
|
||||||
2. Find the process you want to sample (for "Helper" processes, you may want to consult the Chrome Task Manager)
|
2. Find the process you want to sample (for "Helper" processes, you may want to
|
||||||
3. Double-click on the row
|
consult the Chrome Task Manager)
|
||||||
4. Click the **Sample** button in the process's information window
|
3. Double-click on the row
|
||||||
|
4. Click the **Sample** button in the process's information window
|
||||||
|
|
||||||
After a few seconds, the sample will be completed. For official Google Chrome
|
After a few seconds, the sample will be completed. For official Google Chrome
|
||||||
builds, the sample should be symbolized using
|
builds, the sample should be symbolized using
|
||||||
@ -318,3 +269,127 @@ analysis.
|
|||||||
|
|
||||||
See also [How to Obtain a Heap
|
See also [How to Obtain a Heap
|
||||||
Dump](../memory-infra/heap_profiler.md#how-to-obtain-a-heap-dump-m66_linux_macos_windows).
|
Dump](../memory-infra/heap_profiler.md#how-to-obtain-a-heap-dump-m66_linux_macos_windows).
|
||||||
|
|
||||||
|
## Working with Minidumps
|
||||||
|
|
||||||
|
[See this
|
||||||
|
page.](https://sites.google.com/a/chromium.org/dev/developers/crash-reports)
|
||||||
|
|
||||||
|
## Disabling ReportCrash
|
||||||
|
|
||||||
|
macOS helpfully tries to write a crash report every time a binary crashes –
|
||||||
|
which happens for example when a test in unit\_tests fails. Since Chromium's
|
||||||
|
debug binaries are huge, this takes forever. If this happens, "ReportCrash" will
|
||||||
|
be the top cpu consuming process in Activity Monitor. You should disable
|
||||||
|
ReportCrash while you work on Chromium. Run `man ReportCrash` to learn how to do
|
||||||
|
this on your version of macOS. On 10.15, the command is
|
||||||
|
|
||||||
|
launchctl unload -w /System/Library/LaunchAgents/com.apple.ReportCrash.plist
|
||||||
|
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.ReportCrash.Root.plist
|
||||||
|
|
||||||
|
Yes, you need to run this for both the normal user and the admin user.
|
||||||
|
|
||||||
|
## Processing Apple Crash Reports
|
||||||
|
|
||||||
|
If you get a Google Chrome crash report caught by ReportCrash/macOS, it will not
|
||||||
|
have symbols (every frame will be ChromeMain). To get a symbolized stack trace,
|
||||||
|
use the internal [crsym](httsp://goto.google.com/crsym) tool by simply pasting
|
||||||
|
the contents of an entire Apple crash report.
|
||||||
|
|
||||||
|
## Testing Other Locales
|
||||||
|
|
||||||
|
To test Chrome in a different locale, change your system locale via the System
|
||||||
|
Preferences. (Keep the preferences window open so that you can change the
|
||||||
|
locale back without needing to navigate through menus in a language you may not
|
||||||
|
know.)
|
||||||
|
|
||||||
|
## Using DTrace
|
||||||
|
|
||||||
|
DTrace is a powerful, kernel-level profiling and dynamic tracing utility. In
|
||||||
|
order to use DTrace, you need to (at least partially) disable System Integrity
|
||||||
|
Protection with ([see above](#Chrome-Builds)):
|
||||||
|
|
||||||
|
csrutil enable --without dtrace
|
||||||
|
|
||||||
|
Using DTrace is beyond the scope of this document, but the following resources
|
||||||
|
are useful:
|
||||||
|
|
||||||
|
* [jgm's awesome introductory article](http://www.mactech.com/articles/mactech/Vol.23/23.11/ExploringLeopardwithDTrace/index.html)
|
||||||
|
* [Big Nerd Ranch's four-part series](https://www.bignerdranch.com/blog/hooked-on-dtrace-part-1/)
|
||||||
|
* [Defining static probes on macOS](http://www.macresearch.org/tuning-cocoa-applications-using-dtrace-custom-static-probes-and-instruments)
|
||||||
|
* [Examples](http://www.brendangregg.com/dtrace.html#Examples)
|
||||||
|
* [Tips from Sun](http://blogs.sun.com/bmc/resource/dtrace_tips.pdf)
|
||||||
|
|
||||||
|
DTrace examples on macOS: `/usr/share/examples/DTTk`
|
||||||
|
|
||||||
|
To get truss on macOS, use dtruss. That requires root, so use `sudo dtruss -p`
|
||||||
|
and to attach to a running non-root program.
|
||||||
|
|
||||||
|
## Memory/Heap Inspection
|
||||||
|
|
||||||
|
Chrome has [built-in memory instrumentation](../memory-infra/README.md) that can
|
||||||
|
be used to identify allocations and potential leaks.
|
||||||
|
|
||||||
|
MacOS also provides several low-level command-line tools that can be used to
|
||||||
|
inspect what's going on with memory inside a process.
|
||||||
|
|
||||||
|
**`heap`** summarizes what's currently in the malloc heap(s) of a process. (It
|
||||||
|
only works with regular malloc, of course, but Mac Chrome still uses that.) It
|
||||||
|
shows a number of useful things:
|
||||||
|
|
||||||
|
* How much of the heap is used or free
|
||||||
|
* The distribution of block sizes
|
||||||
|
* A listing of every C++, Objective-C and CoreFoundation class found in the
|
||||||
|
heap, with the number of instances, total size and average size.
|
||||||
|
|
||||||
|
It identifies C++ objects by their vtables, so it can't identify vtable-less
|
||||||
|
classes, including a lot of the lower-level WebCore ones like StringImpl. To
|
||||||
|
work around, temporarily added the `virtual` keyword to `WTF::RefCounted`'s
|
||||||
|
destructor method, which forces every ref-counted object to include a vtable
|
||||||
|
pointer identifying its class.
|
||||||
|
|
||||||
|
**`malloc_history`** identifies the stack backtrace that allocated every malloc
|
||||||
|
block in the heap. It lists every unique backtrace together with its number of
|
||||||
|
blocks and their total size. It requires that the process use malloc stack
|
||||||
|
logging, which is enabled if the environment variable MallocStackLogging is set
|
||||||
|
when it launches. The `env` command is handy for this:
|
||||||
|
|
||||||
|
$ env MallocStackLogging=1 Chromium.app/Contents/MacOS/Chromium
|
||||||
|
|
||||||
|
Then in another shell you run
|
||||||
|
|
||||||
|
$ malloc_history <pid> -all_by_size
|
||||||
|
|
||||||
|
Watch out: the output is *big*.
|
||||||
|
|
||||||
|
**`leaks`** finds malloc blocks that have no pointers to them and are probably
|
||||||
|
leaked. It doesn't require MallocStackLogging, but it's more useful if it's on
|
||||||
|
because it can then show the backtrace that allocated each leaked block.
|
||||||
|
|
||||||
|
**`vmmap`** shows all the virtual-memory regions in the process's address space.
|
||||||
|
This is less useful since it doesn't say anything about individual malloc blocks
|
||||||
|
(except huge ones) but it can be useful for looking at things like static data
|
||||||
|
size, mapped files, and how much memory is paged out. The "-resident" flag shows
|
||||||
|
how much of each allocation is currently paged into RAM. See the man page for
|
||||||
|
details.
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
* These are not going to be very useful on stripped binaries, and they're less
|
||||||
|
useful in release builds.
|
||||||
|
* All of these except vmmap take several *minutes* to run, apparently because
|
||||||
|
of the number of symbols in Chrome. They spend most of their time pegging
|
||||||
|
one CPU down inside system code that's reading symbol tables from the
|
||||||
|
binary. Be patient.
|
||||||
|
* Instruments is an application bundled with Xcode that provides GUI interfaces
|
||||||
|
for many of these tools, including `sample` and `leaks`. Most Chromies prefer
|
||||||
|
the command line tools, but Instruments can be useful. If running Instruments
|
||||||
|
on a local build, expect to wait a few minutes for it to load symbols before
|
||||||
|
it starts recording useful data
|
||||||
|
|
||||||
|
## Resources
|
||||||
|
|
||||||
|
The [Mac OS X Debugging Magic
|
||||||
|
Technote](https://developer.apple.com/technotes/tn2004/tn2124.html) contains a
|
||||||
|
wealth of (mostly outdated) information about various debugging options built in
|
||||||
|
to macOS.
|
||||||
|
@ -45,3 +45,13 @@ so:
|
|||||||
|
|
||||||
Adjust the `--start` (and potentially add an `--end` date/time in the same
|
Adjust the `--start` (and potentially add an `--end` date/time in the same
|
||||||
format) to limit the amount of output.
|
format) to limit the amount of output.
|
||||||
|
|
||||||
|
## Breaking on Sandbox Violations
|
||||||
|
|
||||||
|
In order to determine what is causing a sandbox violation, it can be helpful to
|
||||||
|
use the `send-signal` action on a sandbox rule, like so:
|
||||||
|
|
||||||
|
(deny file-write (path "/foo/bar") (with send-signal SIGSTOP))
|
||||||
|
|
||||||
|
That will cause the process to stop until a debugger is attached or *SIGCONT* is
|
||||||
|
sent to the process.
|
||||||
|
@ -200,16 +200,7 @@ You can find out more about GoogleTest at its
|
|||||||
|
|
||||||
## Debugging
|
## Debugging
|
||||||
|
|
||||||
Good debugging tips can be found
|
Good debugging tips can be found [here](mac/debugging.md).
|
||||||
[here](https://dev.chromium.org/developers/how-tos/debugging-on-os-x). If you
|
|
||||||
would like to debug in a graphical environment, rather than using `lldb` at the
|
|
||||||
command line, that is possible without building in Xcode (see
|
|
||||||
[Debugging in Xcode](https://www.chromium.org/developers/how-tos/debugging-on-os-x/building-with-ninja-debugging-with-xcode)).
|
|
||||||
|
|
||||||
Tips for printing variables from `lldb` prompt (both in Xcode or in terminal):
|
|
||||||
* If `uptr` is a `std::unique_ptr`, the address it wraps is accessible as
|
|
||||||
`uptr.__ptr_.__value_`.
|
|
||||||
* To pretty-print `std::u16string`, follow the instructions [here](lldbinit.md).
|
|
||||||
|
|
||||||
## Update your checkout
|
## Update your checkout
|
||||||
|
|
||||||
|
@ -140,4 +140,4 @@ See
|
|||||||
https://chromium.googlesource.com/chromium/src/+/master/docs/linux/debugging.md
|
https://chromium.googlesource.com/chromium/src/+/master/docs/linux/debugging.md
|
||||||
|
|
||||||
macOS :
|
macOS :
|
||||||
https://dev.chromium.org/developers/how-tos/debugging-on-os-x
|
https://chromium.googlesource.com/chromium/src/+/main/docs/mac/debugging.md
|
||||||
|
Reference in New Issue
Block a user