1. Describe `ChromeTestExpectations`.
2. Update outdated information about flag-specific rebaselining.
3. Remove outdated information about `WontFix`, `Rebaseline`, and
`Missing`; no expectation files use these anymore.
4. Remove references to the old flakiness dashboard.
5. Describe the virtual -> base TestExpectations fallback behavior.
6. Repair some dead links.
Bug: 1474702, 1429690
Change-Id: Id25d92a0200954f586edf11e2932675a53df0b1f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5063123
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1229574}
This allows us to easily override test expectations from
TestExpectations. Otherwise we will have to split tags to
avoid conflicts.
Bug: 1499775
Cq-Include-Trybots: luci.chromium.try:linux-wpt-fyi-rel
Change-Id: Ib3c559a6b3b95b72178b8947c1749394fb963b90
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5021829
Commit-Queue: Weizhong Xia <weizhong@google.com>
Reviewed-by: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1224078}
No-op renames of `webdriver_{tests_suite -> wpt_tests}` and
`wpt_tests_suite` (for Chrome) to `chrome_wpt_tests`.
All WPT suite names now follow the pattern `*_wpt_tests`. Also, `wpt` in
`webdriver_wpt_tests` distinguishes that suite from chromedriver's
non-WPT suites.
Bug: 1474702
Change-Id: Ic075372693bf3c270c4fb00c17b2905c3f1cb932
Cq-Include-Trybots: luci.chromium.try:linux-blink-rel
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5014293
Commit-Queue: Brian Sheedy <bsheedy@chromium.org>
Auto-Submit: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#1221982}
The previous wording relied too much on the screenshot and did not point
out exactly what the user should look for (background color of line
number). Spelling it out to save future readers time.
Bug: None
Change-Id: Id447caae7116124f2ac5a31fb5346afa6a07f7c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5013947
Reviewed-by: Tibor Goldschwendt <tiborg@chromium.org>
Commit-Queue: Demetrios Papadopoulos <dpapad@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1221797}
It is agreed that wpt metadata are too complicated to handle, and we
have decided to continue use test expectations and baselines for
run_wpt_tests.py, and convert that back to metadata when running
the test. Update the document to explain how to run and rebaseline
webdriver tests.
Bug: 1474702
Change-Id: I46a9077800c55d7a8cf6ad208b6886b7799ce6db
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4998687
Commit-Queue: Weizhong Xia <weizhong@google.com>
Reviewed-by: Thiago Perrotta <tperrotta@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1219149}
Add support for interactively debugging WPTs with upstream's `wpt run`.
`run_wpt_tests.py --verbose` ensures browser logs are dumped to stdout
so that `debug_renderer` can find the PID to attach to.
Bug: 1440021
Test: ./debug_renderer ./run_wpt_tests.py -t Default
Test: external/wpt/badging/badge-success.https.html
Change-Id: I58323aea67f6232bc28b198b8f18d3d7bb7d8a7b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4893148
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Robert Flack <flackr@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1201810}
Also remove some docs. The last changes will be the gn which requires
different owners so those will be in separate CLs
Bug: 1469110
Change-Id: Ifa255da0742ff543e289b481af7c80f8f7dd3103
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4857198
Commit-Queue: Struan Shrimpton <sshrimp@google.com>
Reviewed-by: Garrett Beaty <gbeaty@google.com>
Cr-Commit-Position: refs/heads/main@{#1195423}
Currently, in the rewriting rule of the test runners, "~NOTFOUND" is
used as an invalid URL. However, '~' is NOT a forbidden character
according to the URL standard.
Chromium is addressing [1] this non-compliant behavior and will permit
'~' character in URL hosts. Thus, we can no longer use "~NOTFOUND" as
it will become a valid URL.
As an alternative, any invalid character is acceptable. Let's use '^'
because '^' is a forbidden character both before and after the CL [1].
The PR for WPT Chrome runner is:
https://github.com/web-platform-tests/wpt/pull/41840/
[1]: https://crrev.com/c/4823237
Bug: 1416013
Change-Id: Id8767ab49b54e778da53113aa4995e098bdbee59
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4845589
Reviewed-by: Yoav Weiss <yoavweiss@chromium.org>
Commit-Queue: Hayato Ito <hayato@chromium.org>
Reviewed-by: Mathias Bynens <mathias@chromium.org>
Reviewed-by: Adam Rice <ricea@chromium.org>
Reviewed-by: Chidera Olibie <colibie@google.com>
Reviewed-by: danakj <danakj@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1194658}
In the iOS Blink port, running web tests is essential
to assess the level of support for various web features
and functionalities. This change includes necessary
additions to enable the execution of web tests on the
iOS Blink port.
co-author: mkim@ solved the timeout issue by adding
a custom UIApplication that overrides IsRunningTests.
Bug: 1421239
Change-Id: I8835097f628280c76d1213a16a29d6cdddd19ecb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4712370
Reviewed-by: Stephen McGruer <smcgruer@chromium.org>
Reviewed-by: Dave Tapuska <dtapuska@chromium.org>
Reviewed-by: Rohit Rao <rohitrao@chromium.org>
Commit-Queue: Gyuyoung Kim <gyuyoung@igalia.com>
Reviewed-by: Peter Beverloo <peter@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1191108}
Based on discussion in crbug.com/1475247, `rebaseline-cl --fill-missing`
still seems useful for downloading platform-agnostic expectations
without the round-trip of retrying infrastructure failures, but it's not
a feature users actively seek out. Therefore:
* Hide the `--{no-,}fill-missing` options with a deprecation message
(i.e., the feature is only available for infrastructure failures).
There's no automation that uses these options.
* Remove a redundant user prompt. It's no longer possible to continue
with incomplete results.
Bug: 1383284
Change-Id: Icdedba8e406252041da4fa7ccaf4c6f60ef1ce8f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4824713
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1190222}
Having a doc for repro bot failures locally is helpful and
I've got many questions in the past that can be answered by
such a doc. Hope this doc can be helpful for any new Chromium
developers.
Bug: 1467350
Change-Id: I9d4dc839c24004685b5b6113ce288cd47fdc7703
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4711854
Commit-Queue: Sven Zheng <svenzheng@chromium.org>
Reviewed-by: Jenny Zhang <jennyz@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1176748}
* Add new section on how to view the test results and what they are.
* Add new section on testing flag-specific, platform-specific, and
virtual configurations.
* Update builder list.
* Reference wptrunner support from the web test/WPT documents.
* Remove staleness warning because the project has stabilized.
* Replace image that is too large in the debugging section with a
smaller lower-quality version.
* General rewording.
Bug: 1442392
Change-Id: If659ea0195d73060b2da961538a8225480c81aeb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4654034
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1174325}
This change will allow us to create the virtual suite for SingleThreaded
tests and only run the single threaded version of those tests, without
affecting other virtual suites.
This will allow to start running most WPT tests with threaded
compositing while keep running a small amount of the tests
single threaded.
Bug: 1463799
Change-Id: Ifa885d23fffcb1277726aecfc7b66bf2edc39442
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4698657
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Yotam Hacohen <yotha@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1172584}
Currently, failing imported tests are grouped by most specific OWNERS
file for notification. However, some WPT directories only have
DIR_METADATA and not OWNERS; failures in these directories will not get
any notifications, even if the DIR_METADATA has opted in.
This change groups failures by most specific DIR_METADATA instead, since
DIR_METADATA are generally better maintained and hold information for
filing bugs. The team email is now CC'ed (if available), but continue
CC'ing OWNERS from the same directory (if available). For most top-level
directories with both DIR_METADATA and OWNERS, this notification
behavior represents no change.
Update the documentation accordingly.
Bug: 1456825
Change-Id: Idcade096908f58ac9d214297627d614231d68969
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4662112
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#1168480}
To help users understand WPT metadata, this change adds equivalent
examples in TestExpectations/*-expected.txt format.
Also:
* Update the finalized table of properties.
* General rewording.
Bug: 1442392
Change-Id: Ifcf856fc9fe2d48f7e658e19ce5db596d1a78d9b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4649951
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1163272}
This change restores and documents a `run_wpt_tests.py --no-headless`
flag that is plumbed through to `wpt run`. `--no-xvfb` has been removed,
since the need for Xvfb is directly tied to headless mode.
Bug: 1440021, 1442392
Change-Id: I938a64c5539b86042260b0ecd87d120b1202aa94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4650693
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#1163076}
This change finalizes targets the user guide will teach developers to
build. The names were picked to be:
* Short: Including `tests` or `isolate` is redundant.
* Descriptive: Make it obvious `wpt_tests_isolate` tests against chrome.
* Consistent: Other targets follow a `<product>_<test-type>` convention:
* `system_webview_wpt`
* `content_shell_crash_test`
* `android_webview_junit_tests`
Bug: 1442392
Change-Id: Id227c981538b624d3ad5e8c829e67c0152920b5a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4629897
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Andrew Grieve <agrieve@chromium.org>
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: Ali Juma <ajuma@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1161514}
And disable the test_shared_library_unittests test on Mac, as it
fails on the bot due to the DLL not being present, just like on
Windows component builds.
R=lukasza@chromium.org
Bug: 1386212, 1442273
Change-Id: I73ca099d28dd69a13cc013b69e4ff73d412fd3b2
Cq-Include-Trybots: luci.chromium.try:android-rust-arm32-rel,android-rust-arm64-dbg,android-rust-arm64-rel,linux-rust-x64-dbg,linux-rust-x64-rel,mac-rust-x64-rel,win-rust-x64-dbg,win-rust-x64-rel
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4544956
Commit-Queue: Dirk Pranke <dpranke@google.com>
Auto-Submit: danakj <danakj@chromium.org>
Commit-Queue: danakj <danakj@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Cr-Commit-Position: refs/heads/main@{#1146235}
The warning about clobbering all-pass statuses (crrev.com/c/4334979) is
not useful when the affected builder/test passes expectedly (so it
generates no `actual_text` to download). In this change, `rebaseline-cl`
instead automatically generates a dummy baseline that `run_web_tests.py`
and `optimize-baselines` recognize as all-pass. This preserves subtest
-level coverage for failing platforms.
These dummy baselines should rarely be necessary, so this change's
impact on checkout size should be negligible. The dummy baselines are
self-documenting and link to the relevant web test documentation, which
this change updates.
Bug: 1324638
Test: To test the `run_web_tests.py` side:
Test: For `external/wpt/badging/badge-success.https.html`, create an
Test: abbreviated all-pass baseline for linux (host platform) and a
Test: failing baseline for win.
Test: `./run_web_tests.py -t Default external/wpt/badging` succeeds
Change-Id: I4a94fb4fda5c566ab0b5838c8ac913c6ee6cccce
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4385729
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1124532}
While not really about histograms, this seems like the best spot to put this information.
I don't have much direct information here in terms of how this is configured, but not sure how necessary it is (and honestly, how this is configured, maybe the test doc reviewer may know more).
Bug: 1414862
Change-Id: Idf39a4b9ee3ac85668da8c13b18427ef60dd156b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4240979
Commit-Queue: Robert Kaplow <rkaplow@chromium.org>
Reviewed-by: Alexei Svitkine <asvitkine@chromium.org>
Reviewed-by: David Jacobo <djacobo@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1104509}
Note to sheriffs: If this CL causes a few new failures,
please fix the failures by adding new
lines to TestExpectations rather than reverting.
In this CL we are adding mac13 and mac13-arm64 builders. In order to
achieve this the CL also adds the new baselines for the new builders
by running the rebaseline tool on the new builds.
Also add the ports to the necessary files in order for the builders
to properly run on Mac13 and Mac13-arm64.
Change-Id: Ifc230fc805ce1b44c438d1de21f08e7fe95a901f
Bug: 1372533
Cq-Include-Trybots: luci.chromium.try:linux-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac10.13-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac10.14-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac10.15-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac11.0-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac11.0.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac12.0-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac12.0.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac13-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac13.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:win10.20h2-blink-rel
Cq-Include-Trybots: luci.chromium.try:win11-blink-rel
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4197702
Reviewed-by: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Nihar Damarasingu <nihardamar@google.com>
Cr-Commit-Position: refs/heads/main@{#1101843}
This page is now pointed to by builds using RTS to explain what's
going on. Added information on what's happening and how to disable
Bug: 1344885
Change-Id: I2588ca4fb2c01cf989f4015aaff6dddfa515e525
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4157733
Commit-Queue: Struan Shrimpton <sshrimp@google.com>
Reviewed-by: Stephanie Kim <kimstephanie@google.com>
Cr-Commit-Position: refs/heads/main@{#1091551}
