Also,
* Add a disclaimer that `run_wpt_tests.py` Android support is
experimental.
* Update "Known Issues" section with new hotlist and template URL.
Bug: None
Change-Id: I221ce2b6622e53eb90e8905efcc094b3ca259379
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5851602
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1353984}
----
Add a section to explain the data format of the code coverage report.
Bug: b:360923880
Change-Id: I94d5ff02ae63584a0bf1d8f052aeacafdbd95e68
----
Update the code coverage documentation.
Change-Id: I94d5ff02ae63584a0bf1d8f052aeacafdbd95e68
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5817355
Reviewed-by: Peter Wen <wnwen@chromium.org>
Commit-Queue: Arthur Wang <wuwang@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1347571}
This is a reland of commit 8964d40d2b
Changes since original:
Added a missing expectation file for
external/wpt/wai-aria/role/contextual-roles.html. The try builds for
the original CL passed but became stale because they predated
https://crrev.com/c/5769811, which introduced the new failure.
Original change's description:
> [wptrunner] Remove `chrome` port and expectations
>
> Currently, the WPT infrastructure supports both content shell and chrome
> coverage on Linux simultaneously. The plan now is to open up chrome as
> an option on macOS and Windows, with the caveat that each WPT can only
> target one binary or the other. This will keep the number of
> expectations manageable and minimize the amount of redundant coverage.
>
> This CL:
> * Removes harness support for the `linux-chrome` option, since this will
> be decided at the test level.
> * Renames `*-expected.txt` files out of the obsolete `linux-chrome`
> directory as needed.
> * Removes ChromeTestExpectations, with any necessary wdspec test
> expectations moved to TestExpectations.
>
> Bug: 328079854
> Change-Id: I205197b038e6441773b2d7307a83e8d5da3837a0
> Cq-Include-Trybots: luci.chromium.try:linux-blink-rel
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5747069
> Reviewed-by: Weizhong Xia <weizhong@google.com>
> Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
> Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
> Cr-Commit-Position: refs/heads/main@{#1338757}
Bug: 328079854
Fixed: 41484800
Change-Id: Ida08b37b629021d99fa3a92246ebcb48846fe284
Cq-Include-Trybots: luci.chromium.try:mac11.0-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac11.0.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac12.0-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac12.0.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac13-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac13.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac14-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac14.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac15-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac15.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:win11-blink-rel
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5771704
Reviewed-by: Weizhong Xia <weizhong@google.com>
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Commit-Queue: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#1339144}
This reverts commit 8964d40d2b.
Reason for revert:
external/wpt/wai-aria/role/contextual-roles.html is missing an
expectation file:
https://ci.chromium.org/ui/p/chromium/builders/ci/mac11-arm64-rel-tests/43434/overviewhttps://ci.chromium.org/ui/p/chromium/builders/ci/mac12-arm64-rel-tests/29051/overview
Original change's description:
> [wptrunner] Remove `chrome` port and expectations
>
> Currently, the WPT infrastructure supports both content shell and chrome
> coverage on Linux simultaneously. The plan now is to open up chrome as
> an option on macOS and Windows, with the caveat that each WPT can only
> target one binary or the other. This will keep the number of
> expectations manageable and minimize the amount of redundant coverage.
>
> This CL:
> * Removes harness support for the `linux-chrome` option, since this will
> be decided at the test level.
> * Renames `*-expected.txt` files out of the obsolete `linux-chrome`
> directory as needed.
> * Removes ChromeTestExpectations, with any necessary wdspec test
> expectations moved to TestExpectations.
>
> Bug: 328079854
> Change-Id: I205197b038e6441773b2d7307a83e8d5da3837a0
> Cq-Include-Trybots: luci.chromium.try:linux-blink-rel
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5747069
> Reviewed-by: Weizhong Xia <weizhong@google.com>
> Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
> Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
> Cr-Commit-Position: refs/heads/main@{#1338757}
Bug: 328079854
Change-Id: I9deb2c2bd012b28f80ed2c78d394917c6d413be1
Cq-Include-Trybots: luci.chromium.try:linux-blink-rel
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5770702
Reviewed-by: Orin Jaworski <orinj@chromium.org>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Owners-Override: Orin Jaworski <orinj@google.com>
Reviewed-by: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#1338796}
Currently, the WPT infrastructure supports both content shell and chrome
coverage on Linux simultaneously. The plan now is to open up chrome as
an option on macOS and Windows, with the caveat that each WPT can only
target one binary or the other. This will keep the number of
expectations manageable and minimize the amount of redundant coverage.
This CL:
* Removes harness support for the `linux-chrome` option, since this will
be decided at the test level.
* Renames `*-expected.txt` files out of the obsolete `linux-chrome`
directory as needed.
* Removes ChromeTestExpectations, with any necessary wdspec test
expectations moved to TestExpectations.
Bug: 328079854
Change-Id: I205197b038e6441773b2d7307a83e8d5da3837a0
Cq-Include-Trybots: luci.chromium.try:linux-blink-rel
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5747069
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1338757}
1. Add mac15/arm64 to builders.json
2. Add references to port configurations
3. Add flag to Expectation files
4. Create README file in .../platform/mac-mac14/ and .../platform/mac-mac15-arm64/
Change-Id: Ia4b225440acf45e0f6ed348cedbb509c2b7457cb
Bug: 355448109
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5749682
Reviewed-by: Reid Kleckner <rnk@chromium.org>
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: An Sung <ansung@google.com>
Reviewed-by: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1338068}
The @DoNotBatch is likely to be copied-pasted into tests so
make it conform to the formatting standards to avoid having
and extra `git cl format` loop after fixing the test.
Bug: None
Change-Id: Id49142c27e808acfcfefffae073df1d108b4ed91
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5511384
Auto-Submit: Finnur Thorarinsson <finnur@chromium.org>
Reviewed-by: Andrew Grieve <agrieve@chromium.org>
Commit-Queue: Andrew Grieve <agrieve@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1295628}
The canonical bug format is TODO(crbug.com/<id>). TODOs of the
following forms will all be migrated to the new format:
- TODO(crbug.com/<old id>)
- TODO(https://crbug.com/<old id>)
- TODO(crbug/<old id>)
- TODO(crbug/monorail/<old id>)
- TODO(<old id>)
- TODO(issues.chromium.org/<old id>)
- TODO(https://issues.chromium.org/<old id>)
- TODO(https://issues.chromium.org/u/1/issues/<old id>)
- TODO(bugs.chromium.org/<old id>)
Bug id mapping is sourced from go/chrome-on-buganizer-prod-issues.
See go/crbug-todo-migration for details.
#crbug-todo-migration
Bug: b/321899722
Change-Id: Iee14d10d544e9f0ec046117cc4ec8a55c427adc0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5469947
Reviewed-by: Darryl James <dljames@chromium.org>
Owners-Override: Alison Gale <agale@chromium.org>
Commit-Queue: Alison Gale <agale@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1290838}
This CL implements a flag called `--inspector-protocol-log` that accepts
a path to a Chrome DevTools Protocol message log. If specified, the test
runner would replay the log mocking the actual browser. The purpose of
this flag is to allow reproducing test flakiness locally and it is not
meant to be used on the bots for now.
Bug: 327140253
Change-Id: I871442d568878b3a1a0d71a18eb2eb721b457e76
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5331573
Reviewed-by: danakj <danakj@chromium.org>
Commit-Queue: Alex Rudenko <alexrudenko@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1274969}
As we are planning to run the headful mode on CI to get performance
data, running in headful mode does not necessary mean debugging.
Add '--no-xvfb' to control if we do not want to start xvfb.
Bug: 1517496
Change-Id: Ia00b2ee0de380c2c8a835d9bce5c9eb71993f57b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5252105
Reviewed-by: Jonathan Lee <jonathanjlee@google.com>
Commit-Queue: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#1257627}
Monorail is read-only now. Activate the Buganizer code path, update the
WPT notification instructions, and add `buganizer_public.component_id`
to `external/DIR_METADATA` to collect all bugs for now. According to the
FAQ [0], the remaining `DIR_METADATA` will receive Buganizer component
IDs "soon after migration".
[0]: go/chrome-on-buganizer-faq
Bug: b/40257571
Change-Id: I9b0b89be29964ec46c7e09d7d39eadbf5ee23030
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5270234
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1256359}
This is a reland of commit 0265879ffa
The original change was reverted because this enabled the continuous
mode on PGO bots, causing them to fail. The continuous mode was enabled
everywhere by the change made in the original CL to test_env.py. It is
believed that the continuous mode is incompatible with pgo builds
because PGO builds make manual calls to compiler-rt to flush the
counters. To address this, this reland wil omit the changes made to
test_env.py. Instead, continuous mode will be enabled on the bots later
by updating the recipe, which is in a separate repository.
Original change's description:
> Enable continuous mode/runtime counter relocation for coverage builds
>
> Enabling the continuous mode for coverage builds allows us to recover
> coverage data for test processes run in sandboxes while avoiding the CQ
> failures that we see with the noncontinuous mode that flushes all the
> counters to disk at process shutdown. This should also be expected to
> recover coverage data for tests that exit abnormally.
>
> Enabling the continuous mode is also the first step in landing several
> changes that will potentially improve instrumented test performance.
>
>
> Cq-Include-Trybots: luci.chromium.try:chromeos-amd64-generic-siso-rel
> Cq-Include-Trybots: luci.chromium.try:ios-simulator-siso
> Cq-Include-Trybots: luci.chromium.try:linux_chromium_asan_siso_rel_ng
> Cq-Include-Trybots: luci.chromium.try:mac-siso-rel
> Change-Id: Iab1177b2e8f5a24887a33417d1556c90f1002363
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4953786
> Reviewed-by: Junji Watanabe <jwata@google.com>
> Reviewed-by: danakj <danakj@chromium.org>
> Reviewed-by: Ben Pastene <bpastene@chromium.org>
> Reviewed-by: Hans Wennborg <hans@chromium.org>
> Commit-Queue: Alan Zhao <ayzhao@google.com>
> Reviewed-by: Dominic Farolino <dom@chromium.org>
> Reviewed-by: Prakhar Asthana <pasthana@google.com>
> Cr-Commit-Position: refs/heads/main@{#1235959}
Bug: 1462187,1468343,1510916
Change-Id: I8a065f0a5886a53f5509491796501dd5b288c99a
Cq-Include-Trybots: luci.chromium.try:chromeos-amd64-generic-siso-rel
Cq-Include-Trybots: luci.chromium.try:ios-simulator-siso
Cq-Include-Trybots: luci.chromium.try:linux_chromium_asan_siso_rel_ng
Cq-Include-Trybots: luci.chromium.try:mac-siso-rel
Cq-Include-Trybots: chrome/try:linux-pgo,mac-pgo,win64-pgo
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5122344
Commit-Queue: Prakhar Asthana <pasthana@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Prakhar Asthana <pasthana@google.com>
Reviewed-by: Dominic Farolino <dom@chromium.org>
Reviewed-by: danakj <danakj@chromium.org>
Reviewed-by: Hans Wennborg <hans@chromium.org>
Commit-Queue: Alan Zhao <ayzhao@google.com>
Cr-Commit-Position: refs/heads/main@{#1245683}
Adds documentation on the expectation file format used by GPU and Blink
tests. Some aspects of the format, particularly various edge cases,
do not seem to currently have documentation anywhere.
Bug: 1499971
Change-Id: I0942e3afd8b935bf813cfb4acb9fb162aa49191d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5014876
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Brian Sheedy <bsheedy@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1244250}
1. Describe `ChromeTestExpectations`.
2. Update outdated information about flag-specific rebaselining.
3. Remove outdated information about `WontFix`, `Rebaseline`, and
`Missing`; no expectation files use these anymore.
4. Remove references to the old flakiness dashboard.
5. Describe the virtual -> base TestExpectations fallback behavior.
6. Repair some dead links.
Bug: 1474702, 1429690
Change-Id: Id25d92a0200954f586edf11e2932675a53df0b1f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5063123
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1229574}
This allows us to easily override test expectations from
TestExpectations. Otherwise we will have to split tags to
avoid conflicts.
Bug: 1499775
Cq-Include-Trybots: luci.chromium.try:linux-wpt-fyi-rel
Change-Id: Ib3c559a6b3b95b72178b8947c1749394fb963b90
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5021829
Commit-Queue: Weizhong Xia <weizhong@google.com>
Reviewed-by: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1224078}
No-op renames of `webdriver_{tests_suite -> wpt_tests}` and
`wpt_tests_suite` (for Chrome) to `chrome_wpt_tests`.
All WPT suite names now follow the pattern `*_wpt_tests`. Also, `wpt` in
`webdriver_wpt_tests` distinguishes that suite from chromedriver's
non-WPT suites.
Bug: 1474702
Change-Id: Ic075372693bf3c270c4fb00c17b2905c3f1cb932
Cq-Include-Trybots: luci.chromium.try:linux-blink-rel
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5014293
Commit-Queue: Brian Sheedy <bsheedy@chromium.org>
Auto-Submit: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#1221982}
The previous wording relied too much on the screenshot and did not point
out exactly what the user should look for (background color of line
number). Spelling it out to save future readers time.
Bug: None
Change-Id: Id447caae7116124f2ac5a31fb5346afa6a07f7c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5013947
Reviewed-by: Tibor Goldschwendt <tiborg@chromium.org>
Commit-Queue: Demetrios Papadopoulos <dpapad@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1221797}
It is agreed that wpt metadata are too complicated to handle, and we
have decided to continue use test expectations and baselines for
run_wpt_tests.py, and convert that back to metadata when running
the test. Update the document to explain how to run and rebaseline
webdriver tests.
Bug: 1474702
Change-Id: I46a9077800c55d7a8cf6ad208b6886b7799ce6db
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4998687
Commit-Queue: Weizhong Xia <weizhong@google.com>
Reviewed-by: Thiago Perrotta <tperrotta@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1219149}
