2097a1d966
Benchmark has been superseded by system_health.memory_mobile. Also take the chance to update and fix some broken links in memory benchmark docs. TBR=crouleau@chromium.org Bug: 761014 Change-Id: I8bd548bff0bc35cbba751ce2745e5162fbbcbec2 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1688785 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org> Cr-Commit-Position: refs/heads/master@{#674997}
204 lines
8.7 KiB
Markdown
204 lines
8.7 KiB
Markdown
# Memory Benchmarks
|
|
|
|
This document describes benchmarks available to track Chrome's and
|
|
WebView's memory usage, where they live, what they measure, how to run them,
|
|
and on how to diagnose regressions.
|
|
|
|
[TOC]
|
|
|
|
## Glossary
|
|
|
|
* **User story:** a set of actions to perform on a browser or device (e.g.
|
|
open google homepage, type "foo", click search, scroll down, visit first
|
|
result, etc.).
|
|
* **Metric:** a data aggregation process that takes a Chrome trace as input
|
|
(produced by a [Telemetry][] run) and produces a set of summary numbers as
|
|
output (e.g. total GPU memory used).
|
|
* **Benchmark:** a combination of (one or more) user stories and (one or
|
|
more) metrics.
|
|
|
|
[Telemetry]: https://github.com/catapult-project/catapult/blob/master/telemetry/README.md
|
|
|
|
## System Health
|
|
|
|
*System health* is an effort to unify top-level benchmarks (as opposite to
|
|
micro-benchmarks and regression tests) that are suitable to capture
|
|
representative user stories.
|
|
|
|
### Benchmarks
|
|
|
|
System health memory benchmarks are:
|
|
|
|
* [system_health.memory_mobile][system_health] -
|
|
user stories running on Android devices.
|
|
* [system_health.memory_desktop][system_health] -
|
|
user stories running on desktop platforms.
|
|
|
|
These benchmarks are run continuously on the [chrome.perf][] waterfall,
|
|
collecting and reporting results on the
|
|
[Chrome Performance Dashboard][chromeperf].
|
|
|
|
[system_health]: https://chromium.googlesource.com/chromium/src/+/master/tools/perf/page_sets/system_health/
|
|
[chrome.perf]: https://ci.chromium.org/p/chrome/g/chrome.perf/console
|
|
[chromeperf]: https://chromeperf.appspot.com/report
|
|
|
|
### User stories
|
|
|
|
System health user stories are classified by the kind of interactions they
|
|
perform with the browser:
|
|
|
|
* `browse` stories navigate to a URL and interact with the page; e.g.
|
|
scroll, click on elements, navigate to subpages, navigate back.
|
|
* `load` stories just navigate to a URL and wait for the page to
|
|
load.
|
|
* `background` stories navigate to a URL, possibly interact with the
|
|
page, and then bring another app to the foreground (thus pushing the
|
|
browser to the background).
|
|
* `long_running` stories interact with a page for a longer period
|
|
of time (~5 mins).
|
|
* `blank` has a single story that just navigates to **about:blank**.
|
|
|
|
The full name of a story has the form `{interaction}:{category}:{site}` where:
|
|
|
|
* `interaction` is one the labels given above;
|
|
* `category` is used to group together sites with a similar purpose,
|
|
e.g. `news`, `social`, `tools`;
|
|
* `site` is a short name identifying the website in which the story mostly
|
|
takes place, e.g. `cnn`, `facebook`, `gmail`.
|
|
|
|
For example `browse:news:cnn` and `background:social:facebook` are two system
|
|
health user stories.
|
|
|
|
Today, for most stories a garbage collection is forced at the end of the
|
|
story and a memory dump is then triggered. Metrics report the values
|
|
obtained from this single measurement.
|
|
|
|
## Continuous monitoring
|
|
|
|
|
|
|
|
To view data from one of the benchmarks on the
|
|
[Chrome Performance Dashboard][chromeperf] you should select:
|
|
|
|
* **Test suite:** The name of a *[benchmark](#Benchmarks)*.
|
|
* **Bot:** The name of a *platform or device configuration*. Sign in to also
|
|
see internal bots.
|
|
* **Subtest (1):** The name of a *[metric](#Understanding-memory-metrics)*.
|
|
* **Subtest (2):** The name of a *story group*; these have the form
|
|
`{interaction}_{category}` for system health stories.
|
|
* **Subtest (3):** The name of a *[user story](#User-stories)*
|
|
(with `:` replaced by `_`).
|
|
|
|
If you are investigating a Perf dashboard alert and would like to see the
|
|
details, you can click on any point of the graph. It gives you the commit range,
|
|
buildbot output and a link to the trace file taken during the buildbot run.
|
|
(More information about reading trace files [here][memory-infra])
|
|
|
|
[memory-infra]: /docs/memory-infra/README.md
|
|
|
|
|
|
|
|
## How to run the benchmarks
|
|
|
|
Benchmarks may be run on a local platform/device or remotely on a pinpoint
|
|
try job.
|
|
|
|
### How to run a pinpoint try job
|
|
|
|
Given a patch already uploaded to code review, try jobs provide a convenient
|
|
way to evaluate its memory implications on devices or platforms which
|
|
may not be immediately available to developers.
|
|
|
|
|
|
|
|
To start a try job go to the [pinpoint][] website, click on the `+` button to
|
|
create a new job, and fill in the required details:
|
|
|
|
[pinpoint]: https://pinpoint-dot-chromeperf.appspot.com/
|
|
|
|
* **Bug ID** (optional): The id of a crbug.com issue where pinpoint can post
|
|
updates when the job finishes.
|
|
* **Gerrit URL**: URL to the patch you want to test. Note that your patch can
|
|
live in chromium or any of its sub-repositories!
|
|
* **Bot**: Select a suitable device/platform from the drop-down menu on which
|
|
to run your job.
|
|
* **Benchmark**: The name of the benchmark to run. If you are interested in
|
|
memory try `system_health.memory_mobile` or `system_health.memory_desktop`
|
|
as appropriate.
|
|
* **Story** (optional): A pattern passed to Telemetry's `--story-filter`
|
|
option to only run stories that match the pattern.
|
|
* **Extra Test Arguments** (optional): Additional command line arguments for
|
|
Telemetry's `run_benchmark`. Of note, if you are interested in running a
|
|
small but representative sample of system health stories you can pass
|
|
`--story-tag-filter health_check`.
|
|
|
|
If you have more specific needs, or need to automate the creation of jobs, you
|
|
can also consider using [pinpoint_cli][].
|
|
|
|
[pinpoint_cli]: https://cs.chromium.org/chromium/src/tools/perf/pinpoint_cli
|
|
|
|
### How to run locally
|
|
|
|
After building, e.g. `ChromePublic.apk`, you can run a specific system health
|
|
story with the command:
|
|
|
|
```
|
|
$SRC/tools/perf/run_benchmark run system_health.memory_mobile \
|
|
--browser android-chromium --story-filter load:search:google
|
|
```
|
|
|
|
This will run the story with a default of 3 repetitions and produce a
|
|
`results.html` file comparing results from this and any previous benchmark
|
|
runs. In addition, you'll also get individual [trace files][memory-infra]
|
|
for each story run by the benchmark. **Note:** by default only high level
|
|
metrics are shown, you may need to tick the "Show all" check box in order to
|
|
view some of the lower level memory metrics.
|
|
|
|
|
|
|
|
Other useful options for this command are:
|
|
|
|
* `--pageset-repeat [n]` - override the default number of repetitions
|
|
* `--reset-results` - clear results from any previous benchmark runs in the
|
|
`results.html` file.
|
|
* `--results-label [label]` - give meaningful names to your benchmark runs,
|
|
this way it is easier to compare them.
|
|
|
|
For WebView make sure to [replace the system WebView][webview_install]
|
|
on your device and use `--browser android-webview`.
|
|
|
|
[memory-infra]: /docs/memory-infra/README.md
|
|
[webview_install]: https://www.chromium.org/developers/how-tos/build-instructions-android-webview
|
|
|
|
## Understanding memory metrics
|
|
|
|
There is a large number of [memory-infra][] metrics, breaking down usage
|
|
attributed to different components and processes.
|
|
|
|
|
|
|
|
Most memory metrics have the form
|
|
`memory:{browser}:{processes}:{source}:{component}:{kind}`
|
|
where:
|
|
|
|
* **browser:** One of `chrome` or `webview`.
|
|
* **processess:** One of `browser_process`, `renderer_processess`,
|
|
`gpu_process`, or `all_processess`.
|
|
* **source:** One of `reported_by_chrome` or `reported_by_os`
|
|
* **component:** May be a Chrome component, e.g. `skia` or `sqlite`;
|
|
details about a specific component, e.g. `v8:heap`; or a class of memory
|
|
as seen by the OS, e.g. `system_memory:native_heap` or `gpu_memory`. If
|
|
reported by chrome, the metrics are gathered by `MemoryDumpProvider`s,
|
|
probes placed in the specific components' codebase. For example, in
|
|
"memory:chrome:all_processes:reported_by_chrome:net:effective_size_avg,"
|
|
the component is "net" which is Chrome's network stack and
|
|
"reported_by_chrome" means that this metric is gathered via probes in
|
|
the network stack.
|
|
* **kind:** The kind of memory being reported. For metrics reported by
|
|
Chrome this usually is `effective_size` (others are `locked_size`
|
|
and `allocated_objects_size`); for metrics by the OS this usually is
|
|
`proportional_resident_size` (others are `peak_resident_size` and
|
|
`private_dirty_size`).
|
|
|
|
[memory-infra]: /docs/memory-infra/README.md
|