As WPT migration is mostly done and headless shell is now the default
target for WPTs, make this clear in document and naming to make it
less likely a new WPT accidentally added to the exception list.
Renamed headless_shell.filter to content_shell.filter to make it
obvious that tests listed there are intended to be run with content
shell and is something unusual.
Validate-Test-Flakiness: skip
Bug: 40927590
Change-Id: I787f7c4757aba3f869c4edeb7f43c5aeff79c41b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6154686
Reviewed-by: Rick Byers <rbyers@chromium.org>
Commit-Queue: Weizhong Xia <weizhong@google.com>
Reviewed-by: Struan Shrimpton <sshrimp@google.com>
Reviewed-by: Mason Freed <masonf@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1404578}
Also,
* Add a disclaimer that `run_wpt_tests.py` Android support is
experimental.
* Update "Known Issues" section with new hotlist and template URL.
Bug: None
Change-Id: I221ce2b6622e53eb90e8905efcc094b3ca259379
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5851602
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1353984}
----
Add a section to explain the data format of the code coverage report.
Bug: b:360923880
Change-Id: I94d5ff02ae63584a0bf1d8f052aeacafdbd95e68
----
Update the code coverage documentation.
Change-Id: I94d5ff02ae63584a0bf1d8f052aeacafdbd95e68
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5817355
Reviewed-by: Peter Wen <wnwen@chromium.org>
Commit-Queue: Arthur Wang <wuwang@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1347571}
This is a reland of commit 8964d40d2b
Changes since original:
Added a missing expectation file for
external/wpt/wai-aria/role/contextual-roles.html. The try builds for
the original CL passed but became stale because they predated
https://crrev.com/c/5769811, which introduced the new failure.
Original change's description:
> [wptrunner] Remove `chrome` port and expectations
>
> Currently, the WPT infrastructure supports both content shell and chrome
> coverage on Linux simultaneously. The plan now is to open up chrome as
> an option on macOS and Windows, with the caveat that each WPT can only
> target one binary or the other. This will keep the number of
> expectations manageable and minimize the amount of redundant coverage.
>
> This CL:
> * Removes harness support for the `linux-chrome` option, since this will
> be decided at the test level.
> * Renames `*-expected.txt` files out of the obsolete `linux-chrome`
> directory as needed.
> * Removes ChromeTestExpectations, with any necessary wdspec test
> expectations moved to TestExpectations.
>
> Bug: 328079854
> Change-Id: I205197b038e6441773b2d7307a83e8d5da3837a0
> Cq-Include-Trybots: luci.chromium.try:linux-blink-rel
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5747069
> Reviewed-by: Weizhong Xia <weizhong@google.com>
> Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
> Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
> Cr-Commit-Position: refs/heads/main@{#1338757}
Bug: 328079854
Fixed: 41484800
Change-Id: Ida08b37b629021d99fa3a92246ebcb48846fe284
Cq-Include-Trybots: luci.chromium.try:mac11.0-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac11.0.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac12.0-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac12.0.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac13-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac13.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac14-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac14.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac15-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac15.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:win11-blink-rel
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5771704
Reviewed-by: Weizhong Xia <weizhong@google.com>
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Commit-Queue: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#1339144}
This reverts commit 8964d40d2b.
Reason for revert:
external/wpt/wai-aria/role/contextual-roles.html is missing an
expectation file:
https://ci.chromium.org/ui/p/chromium/builders/ci/mac11-arm64-rel-tests/43434/overviewhttps://ci.chromium.org/ui/p/chromium/builders/ci/mac12-arm64-rel-tests/29051/overview
Original change's description:
> [wptrunner] Remove `chrome` port and expectations
>
> Currently, the WPT infrastructure supports both content shell and chrome
> coverage on Linux simultaneously. The plan now is to open up chrome as
> an option on macOS and Windows, with the caveat that each WPT can only
> target one binary or the other. This will keep the number of
> expectations manageable and minimize the amount of redundant coverage.
>
> This CL:
> * Removes harness support for the `linux-chrome` option, since this will
> be decided at the test level.
> * Renames `*-expected.txt` files out of the obsolete `linux-chrome`
> directory as needed.
> * Removes ChromeTestExpectations, with any necessary wdspec test
> expectations moved to TestExpectations.
>
> Bug: 328079854
> Change-Id: I205197b038e6441773b2d7307a83e8d5da3837a0
> Cq-Include-Trybots: luci.chromium.try:linux-blink-rel
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5747069
> Reviewed-by: Weizhong Xia <weizhong@google.com>
> Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
> Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
> Cr-Commit-Position: refs/heads/main@{#1338757}
Bug: 328079854
Change-Id: I9deb2c2bd012b28f80ed2c78d394917c6d413be1
Cq-Include-Trybots: luci.chromium.try:linux-blink-rel
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5770702
Reviewed-by: Orin Jaworski <orinj@chromium.org>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Owners-Override: Orin Jaworski <orinj@google.com>
Reviewed-by: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#1338796}
Currently, the WPT infrastructure supports both content shell and chrome
coverage on Linux simultaneously. The plan now is to open up chrome as
an option on macOS and Windows, with the caveat that each WPT can only
target one binary or the other. This will keep the number of
expectations manageable and minimize the amount of redundant coverage.
This CL:
* Removes harness support for the `linux-chrome` option, since this will
be decided at the test level.
* Renames `*-expected.txt` files out of the obsolete `linux-chrome`
directory as needed.
* Removes ChromeTestExpectations, with any necessary wdspec test
expectations moved to TestExpectations.
Bug: 328079854
Change-Id: I205197b038e6441773b2d7307a83e8d5da3837a0
Cq-Include-Trybots: luci.chromium.try:linux-blink-rel
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5747069
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1338757}
1. Add mac15/arm64 to builders.json
2. Add references to port configurations
3. Add flag to Expectation files
4. Create README file in .../platform/mac-mac14/ and .../platform/mac-mac15-arm64/
Change-Id: Ia4b225440acf45e0f6ed348cedbb509c2b7457cb
Bug: 355448109
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5749682
Reviewed-by: Reid Kleckner <rnk@chromium.org>
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: An Sung <ansung@google.com>
Reviewed-by: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1338068}
The @DoNotBatch is likely to be copied-pasted into tests so
make it conform to the formatting standards to avoid having
and extra `git cl format` loop after fixing the test.
Bug: None
Change-Id: Id49142c27e808acfcfefffae073df1d108b4ed91
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5511384
Auto-Submit: Finnur Thorarinsson <finnur@chromium.org>
Reviewed-by: Andrew Grieve <agrieve@chromium.org>
Commit-Queue: Andrew Grieve <agrieve@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1295628}
The canonical bug format is TODO(crbug.com/<id>). TODOs of the
following forms will all be migrated to the new format:
- TODO(crbug.com/<old id>)
- TODO(https://crbug.com/<old id>)
- TODO(crbug/<old id>)
- TODO(crbug/monorail/<old id>)
- TODO(<old id>)
- TODO(issues.chromium.org/<old id>)
- TODO(https://issues.chromium.org/<old id>)
- TODO(https://issues.chromium.org/u/1/issues/<old id>)
- TODO(bugs.chromium.org/<old id>)
Bug id mapping is sourced from go/chrome-on-buganizer-prod-issues.
See go/crbug-todo-migration for details.
#crbug-todo-migration
Bug: b/321899722
Change-Id: Iee14d10d544e9f0ec046117cc4ec8a55c427adc0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5469947
Reviewed-by: Darryl James <dljames@chromium.org>
Owners-Override: Alison Gale <agale@chromium.org>
Commit-Queue: Alison Gale <agale@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1290838}
This CL implements a flag called `--inspector-protocol-log` that accepts
a path to a Chrome DevTools Protocol message log. If specified, the test
runner would replay the log mocking the actual browser. The purpose of
this flag is to allow reproducing test flakiness locally and it is not
meant to be used on the bots for now.
Bug: 327140253
Change-Id: I871442d568878b3a1a0d71a18eb2eb721b457e76
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5331573
Reviewed-by: danakj <danakj@chromium.org>
Commit-Queue: Alex Rudenko <alexrudenko@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1274969}
As we are planning to run the headful mode on CI to get performance
data, running in headful mode does not necessary mean debugging.
Add '--no-xvfb' to control if we do not want to start xvfb.
Bug: 1517496
Change-Id: Ia00b2ee0de380c2c8a835d9bce5c9eb71993f57b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5252105
Reviewed-by: Jonathan Lee <jonathanjlee@google.com>
Commit-Queue: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#1257627}
Monorail is read-only now. Activate the Buganizer code path, update the
WPT notification instructions, and add `buganizer_public.component_id`
to `external/DIR_METADATA` to collect all bugs for now. According to the
FAQ [0], the remaining `DIR_METADATA` will receive Buganizer component
IDs "soon after migration".
[0]: go/chrome-on-buganizer-faq
Bug: b/40257571
Change-Id: I9b0b89be29964ec46c7e09d7d39eadbf5ee23030
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5270234
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1256359}
This is a reland of commit 0265879ffa
The original change was reverted because this enabled the continuous
mode on PGO bots, causing them to fail. The continuous mode was enabled
everywhere by the change made in the original CL to test_env.py. It is
believed that the continuous mode is incompatible with pgo builds
because PGO builds make manual calls to compiler-rt to flush the
counters. To address this, this reland wil omit the changes made to
test_env.py. Instead, continuous mode will be enabled on the bots later
by updating the recipe, which is in a separate repository.
Original change's description:
> Enable continuous mode/runtime counter relocation for coverage builds
>
> Enabling the continuous mode for coverage builds allows us to recover
> coverage data for test processes run in sandboxes while avoiding the CQ
> failures that we see with the noncontinuous mode that flushes all the
> counters to disk at process shutdown. This should also be expected to
> recover coverage data for tests that exit abnormally.
>
> Enabling the continuous mode is also the first step in landing several
> changes that will potentially improve instrumented test performance.
>
>
> Cq-Include-Trybots: luci.chromium.try:chromeos-amd64-generic-siso-rel
> Cq-Include-Trybots: luci.chromium.try:ios-simulator-siso
> Cq-Include-Trybots: luci.chromium.try:linux_chromium_asan_siso_rel_ng
> Cq-Include-Trybots: luci.chromium.try:mac-siso-rel
> Change-Id: Iab1177b2e8f5a24887a33417d1556c90f1002363
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4953786
> Reviewed-by: Junji Watanabe <jwata@google.com>
> Reviewed-by: danakj <danakj@chromium.org>
> Reviewed-by: Ben Pastene <bpastene@chromium.org>
> Reviewed-by: Hans Wennborg <hans@chromium.org>
> Commit-Queue: Alan Zhao <ayzhao@google.com>
> Reviewed-by: Dominic Farolino <dom@chromium.org>
> Reviewed-by: Prakhar Asthana <pasthana@google.com>
> Cr-Commit-Position: refs/heads/main@{#1235959}
Bug: 1462187,1468343,1510916
Change-Id: I8a065f0a5886a53f5509491796501dd5b288c99a
Cq-Include-Trybots: luci.chromium.try:chromeos-amd64-generic-siso-rel
Cq-Include-Trybots: luci.chromium.try:ios-simulator-siso
Cq-Include-Trybots: luci.chromium.try:linux_chromium_asan_siso_rel_ng
Cq-Include-Trybots: luci.chromium.try:mac-siso-rel
Cq-Include-Trybots: chrome/try:linux-pgo,mac-pgo,win64-pgo
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5122344
Commit-Queue: Prakhar Asthana <pasthana@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Prakhar Asthana <pasthana@google.com>
Reviewed-by: Dominic Farolino <dom@chromium.org>
Reviewed-by: danakj <danakj@chromium.org>
Reviewed-by: Hans Wennborg <hans@chromium.org>
Commit-Queue: Alan Zhao <ayzhao@google.com>
Cr-Commit-Position: refs/heads/main@{#1245683}
Adds documentation on the expectation file format used by GPU and Blink
tests. Some aspects of the format, particularly various edge cases,
do not seem to currently have documentation anywhere.
Bug: 1499971
Change-Id: I0942e3afd8b935bf813cfb4acb9fb162aa49191d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5014876
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Brian Sheedy <bsheedy@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1244250}
1. Describe `ChromeTestExpectations`.
2. Update outdated information about flag-specific rebaselining.
3. Remove outdated information about `WontFix`, `Rebaseline`, and
`Missing`; no expectation files use these anymore.
4. Remove references to the old flakiness dashboard.
5. Describe the virtual -> base TestExpectations fallback behavior.
6. Repair some dead links.
Bug: 1474702, 1429690
Change-Id: Id25d92a0200954f586edf11e2932675a53df0b1f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5063123
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1229574}
