docs: Cut mb.py & run-swarmed.py mentions in debugging-with-swarming.md
With the development and use of the UTR as far long as it is, these old tools likely serve little use. So this begins trying to shift usage to the much more supported UTR by updating docs. Future CLs will add deprecation notices to the old tools, and eventually delete them. Bug: 386167803 Change-Id: If5265b7fac7959038d9da817cf241db50d6fe56b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6124971 Reviewed-by: Brian Sheedy <bsheedy@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Reviewed-by: Struan Shrimpton <sshrimp@google.com> Cr-Commit-Position: refs/heads/main@{#1401544}
This commit is contained in:
Ben Pastene
committed by
Chromium LUCI CQ
Chromium LUCI CQ
parent
9e20dd1307
commit
dd94f632a3
@ -1,8 +1,11 @@
|
||||
# Debugging with Swarming
|
||||
|
||||
This document outlines how to debug a test failure on a specific builder
|
||||
configuration without needing to repeatedly upload new CL revisions or do CQ dry
|
||||
runs.
|
||||
configuration on Swarming using the [UTR tool](../../tools/utr/README.md)
|
||||
without needing to repeatedly upload new CL revisions or do CQ dry runs. This
|
||||
tool will automatically handle steps like replicating the right GN args,
|
||||
building & uploading the test isolate, triggering & collecting the swarming test
|
||||
tasks.
|
||||
|
||||
[TOC]
|
||||
|
||||
@ -18,7 +21,9 @@ on the swarming infrastructure. It contains binaries as well as any libraries
|
||||
they link against or support data. An isolate can be thought of like a tarball,
|
||||
but held by the CAS server and identified by a digest of its contents. The
|
||||
isolate also includes the command(s) to run, which is why the command is
|
||||
specified when building the isolate, not when executing it.
|
||||
specified when building the isolate, not when executing it. See the
|
||||
[infra glossay](../infra/glossary.md) for the definitions of these terms and
|
||||
more.
|
||||
|
||||
Normally, when you do a CQ dry run, something like this happens:
|
||||
|
||||
@ -54,204 +59,23 @@ or perhaps:
|
||||
use_swarming_to_run(type, isolate)
|
||||
```
|
||||
|
||||
## The easy way
|
||||
|
||||
A lot of the steps described in this doc have been bundled up into 2
|
||||
tools. Before using either of these you will need to
|
||||
[authenticate](#authenticating).
|
||||
|
||||
### run-swarmed.py
|
||||
|
||||
A lot of the logic below is wrapped up in `tools/run-swarmed.py`, which you can run
|
||||
like this:
|
||||
|
||||
```
|
||||
$ tools/run-swarmed.py $outdir $target [-- --gtest_filter=...]
|
||||
```
|
||||
|
||||
See the `--help` option of `run-swarmed.py` for more details about that script.
|
||||
Many flags are converted into dimensions to pass to `mb.py`; see
|
||||
[Bot selection criteria](#bot-selection-criteria) for possible values. (Most
|
||||
dimensions have the same name as the flag, but `--swarming-os` is just the `os`
|
||||
dimension.)
|
||||
|
||||
Note you might need `--swarming-os Ubuntu-14.04` if you get an error like,
|
||||
`UnboundLocalError: local variable 'dbus_pid' referenced before assignment`.
|
||||
|
||||
Web tests can be also run on swarmed. However:
|
||||
- the only supported output folders are `Debug` and `Release`
|
||||
- `--no-test-flags` must be specified.
|
||||
|
||||
For example, you can run all Web Platform Tests inside `<some-wpt-folder>` like this:
|
||||
```
|
||||
$ autoninja -C out/Release blink_tests
|
||||
$ tools/run-swarmed.py --no-test-flags out/Release blink_wpt_tests <some-wpt-folder>
|
||||
```
|
||||
|
||||
### mb.py run
|
||||
|
||||
Similar to `tools/run-swarmed.py`, `mb.py run` bundles much of the logic into a
|
||||
single command line. Unlike `tools/run-swarmed.py`, `mb.py run` allows the user
|
||||
to specify extra arguments to pass to the test, but has a messier command line.
|
||||
|
||||
To use it, run:
|
||||
```
|
||||
$ tools/mb/mb.py run \
|
||||
-s --no-default-dimensions \
|
||||
-d pool $pool \
|
||||
$criteria \
|
||||
$outdir $target \
|
||||
-- $extra_args
|
||||
```
|
||||
|
||||
## A concrete example
|
||||
|
||||
Here's how to run `chrome_public_test_apk` on a bot with a Nexus 5 running KitKat.
|
||||
Here's how to run `chrome_public_unit_test_apk` on Android devices. By using the
|
||||
config of the `android-arm64-rel` trybot, we can run it on Pixel 3 XLs running
|
||||
Android Pie.
|
||||
|
||||
```sh
|
||||
$ tools/mb/mb.py run \
|
||||
-s --no-default-dimensions \
|
||||
-d pool chromium.tests \
|
||||
-d device_os_type userdebug -d device_os KTU84P -d device_type hammerhead \
|
||||
out/Android-arm-dbg chrome_public_test_apk
|
||||
$ vpython3 tools/utr \
|
||||
-p chromium \
|
||||
-B try \
|
||||
-b android-arm64-rel \
|
||||
-t "chrome_public_unit_test_apk on Android device Pixel 3 XL" \
|
||||
compile-and-test
|
||||
```
|
||||
|
||||
This assumes you have an `out/Android-arm-dbg/args.gn` like
|
||||
|
||||
```
|
||||
ffmpeg_branding = "Chrome"
|
||||
is_component_build = false
|
||||
is_debug = true
|
||||
proprietary_codecs = true
|
||||
strip_absolute_paths_from_debug_symbols = true
|
||||
symbol_level = 1
|
||||
system_webview_package_name = "com.google.android.webview"
|
||||
target_os = "android"
|
||||
use_remoteexec = true
|
||||
```
|
||||
|
||||
## Bot selection criteria
|
||||
|
||||
The examples in this doc use `$criteria`. To figure out what values to use, you
|
||||
can go to an existing swarming run
|
||||
([recent tasks page](https://chromium-swarm.appspot.com/tasklist)) and
|
||||
look at the `Dimensions` section. Each of these becomes a `-d dimension_name
|
||||
dimension_value` in your `$criteria`. Click on `bots` (or go
|
||||
[here](https://chromium-swarm.appspot.com/botlist)) to be taken to a UI that
|
||||
allows you to try out the criteria interactively, so that you can be sure that
|
||||
there are bots matching your criteria. Sometimes the web page shows a
|
||||
human-friendly name rather than the name required on the commandline. [This
|
||||
file](https://cs.chromium.org/chromium/infra/luci/appengine/swarming/ui2/modules/alias.js)
|
||||
contains the mapping to human-friendly names. You can test your commandline by
|
||||
entering `dimension_name:dimension_value` in the interactive UI.
|
||||
|
||||
## Building an isolate
|
||||
|
||||
At the moment, you can only build an isolate locally, like so (commands you type
|
||||
begin with `$`):
|
||||
|
||||
```
|
||||
$ tools/mb/mb.py isolate //$outdir $target
|
||||
```
|
||||
|
||||
This will produce some files in $outdir. The most pertinent two are
|
||||
`$outdir/$target.isolate` and `$outdir/target.isolated`. If you've already built
|
||||
$target, you can save some CPU time and run `tools/mb/mb.py` with `--no-build`:
|
||||
|
||||
```
|
||||
$ tools/mb/mb.py isolate --no-build //$outdir $target
|
||||
```
|
||||
|
||||
Support for building an isolate using swarming, which would allow you to build
|
||||
for a platform you can't build for locally, does not yet exist.
|
||||
|
||||
## Authenticating
|
||||
|
||||
You may need to log in to `https://chromium-swarm.appspot.com` to do this
|
||||
|
||||
```
|
||||
$ tools/luci-go/isolate login
|
||||
```
|
||||
|
||||
Use your google.com account for this. On Windows the command would be:
|
||||
|
||||
```
|
||||
$ tools\luci-go\isolate.exe login
|
||||
```
|
||||
|
||||
## Uploading an isolate
|
||||
|
||||
You can then upload the resulting isolate to the CAS server:
|
||||
|
||||
```
|
||||
$ tools/luci-go/isolate archive \
|
||||
-cas-instance chromium-swarm \
|
||||
-i $outdir/$target.isolate \
|
||||
-dump-json $outdir/$target.archive.json
|
||||
```
|
||||
|
||||
The archive json looks like this:
|
||||
|
||||
```
|
||||
{
|
||||
"unit_tests": "5bee0815d2ddd2b876b49d4cce8aaa23de8a6f9e2dbf134ec409fbdc224e8c64/398"
|
||||
}
|
||||
```
|
||||
|
||||
Do not ctrl-c it after it does this, even if it seems to be hanging for a
|
||||
minute - just let it finish.
|
||||
|
||||
## Running an isolate
|
||||
|
||||
Now that the isolate is on the CAS server with digest `$digest` from the
|
||||
previous step, you can run on bots of your choice:
|
||||
|
||||
```
|
||||
$ tools/luci-go/swarming trigger \
|
||||
-server https://chromium-swarm.appspot.com \
|
||||
-dimension pool=$pool \
|
||||
$criteria \
|
||||
-digest $digest
|
||||
```
|
||||
|
||||
There are two more things you need to fill in here. The first is the pool name;
|
||||
you should pick "chromium.tests" unless you know otherwise. The pool is the
|
||||
collection of hosts from which swarming will try to pick bots to run your tasks.
|
||||
|
||||
The second is the criteria, which is how you specify which bot(s) you want your
|
||||
task scheduled on. These are specified via "dimensions", which are specified
|
||||
with `-dimension key=val`. In fact, the `-dimension pool=$pool` in the
|
||||
command above is selecting based on the "pool" dimension. There are a lot of
|
||||
possible dimensions; one useful one is "os", like `-dimension os=Linux`. Examples of
|
||||
other dimensions include:
|
||||
|
||||
* `-dimension os=Mac10.13.6` to select a specific OS version
|
||||
* `-dimension device_type="Pixel 3"` to select a specific Android device type
|
||||
* `-dimension gpu=8086:1912` to select a specific GPU
|
||||
|
||||
The [swarming bot list] allows you to see all the dimensions and the values they
|
||||
can take on.
|
||||
|
||||
And you can pass shard number via env flags, e.g.
|
||||
|
||||
* `-env GTEST_SHARD_INDEX=0` to specify which shard to run.
|
||||
* `-env GTEST_TOTAL_SHARDS=1` to specify total number of shards.
|
||||
|
||||
If you need to pass additional arguments to the test, simply add
|
||||
`-- $extra_args` to the end of the `swarming trigger` command line - anything
|
||||
after the `--` will be passed directly to the test.
|
||||
|
||||
When you invoke `swarming trigger`, it will emit a piece of information: a
|
||||
command you can run to collect the results of that task. For example:
|
||||
|
||||
```
|
||||
To collect results use:
|
||||
swarming collect -server https://chromium-swarm.appspot.com 4a0e739053fddd10
|
||||
```
|
||||
|
||||
The 'collect' command given there will block until the task is complete, then
|
||||
produce the task's results, or you can load that URL and watch the task's
|
||||
progress.
|
||||
You can find the UTR invocation for any test on the build UI under the step's
|
||||
"reproduction instructions" (displayed by clicking the page icon in the UI).
|
||||
|
||||
## Other notes
|
||||
|
||||
|
||||
@ -62,6 +62,10 @@ you're interested in. Some basic builders you can use are listed below:
|
||||
runs basic functional test suites on CrOS VMs running on intel. Use `-B ci -b
|
||||
chromeos-amd64-generic-rel-gtest` on the UTR cmd line.
|
||||
|
||||
Note that the names of the above builders may change over time and the links
|
||||
may eventually break. Navigate the build console to find the current list of
|
||||
running builders.
|
||||
|
||||
## Command-line Examples
|
||||
|
||||
Below are some example invocations of the UTR:
|
||||
@ -100,6 +104,62 @@ with additional test cmd-line flags:
|
||||
vpython3 run.py -B try -b linux-chromeos-rel -t interactive_ui_tests test -- --gtest_filter=TestClass.TestCase --gtest_repeat=100
|
||||
```
|
||||
|
||||
## Custom builder configs
|
||||
|
||||
For the specified builder, the UTR will reuse its precise configs. This includes
|
||||
things like GN args, test targets, and test dimensions. These configs are
|
||||
encoded as starlark in the checkout, and are read at UTR invocation-time.
|
||||
Consequently, you can make local modifications to these configs and they will be
|
||||
reflected in any subsequent UTR invocation for that builder.
|
||||
|
||||
Example: at time of writing, the
|
||||
[Linux Tests](https://ci.chromium.org/ui/p/chromium/builders/ci/Linux%20Tests)
|
||||
builder runs tests on Ubuntu-22.04 (jammy). To test out running its tests on
|
||||
Ubuntu-24.04 (noble), you would find its builder config in the
|
||||
[infra starlark dir](https://source.chromium.org/chromium/chromium/src/+/main:infra/config/subprojects/)
|
||||
in the repo. It should look something like:
|
||||
```
|
||||
ci.thin_tester(
|
||||
name = "Linux Tests",
|
||||
triggered_by = ["ci/Linux Builder"],
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
Find the line that controls the dimensions its tests target:
|
||||
```
|
||||
...
|
||||
targets = targets.bundle(
|
||||
...
|
||||
mixins = [
|
||||
"isolate_profile_data",
|
||||
"linux-jammy",
|
||||
],
|
||||
...
|
||||
```
|
||||
|
||||
And change the Jammy mixin to Noble:
|
||||
```
|
||||
...
|
||||
targets = targets.bundle(
|
||||
...
|
||||
mixins = [
|
||||
"isolate_profile_data",
|
||||
"linux-noble",
|
||||
],
|
||||
...
|
||||
```
|
||||
|
||||
Then regen the infra configs, and re-run the UTR. Within the UTR invocation,
|
||||
it should now run base_unittests on Ubuntu-24.04 rather than Ubuntu-22.04:
|
||||
```sh
|
||||
$ ./infra/config/main.star
|
||||
$ vpython3 run.py -B ci -b 'Linux Tests' -t base_unittests compile-and-test
|
||||
```
|
||||
|
||||
See [here](../../infra/config/targets/README.md) for more info about how a
|
||||
builder's tests are configured in starlark.
|
||||
|
||||
## Questions/feedback
|
||||
|
||||
If you have any usage questions or suggestions for feedback, feel free to file
|
||||
|
||||
Reference in New Issue
Block a user