Add support for interactively debugging WPTs with upstream's `wpt run`.
`run_wpt_tests.py --verbose` ensures browser logs are dumped to stdout
so that `debug_renderer` can find the PID to attach to.
Bug: 1440021
Test: ./debug_renderer ./run_wpt_tests.py -t Default
Test: external/wpt/badging/badge-success.https.html
Change-Id: I58323aea67f6232bc28b198b8f18d3d7bb7d8a7b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4893148
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Robert Flack <flackr@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1201810}
Also remove some docs. The last changes will be the gn which requires
different owners so those will be in separate CLs
Bug: 1469110
Change-Id: Ifa255da0742ff543e289b481af7c80f8f7dd3103
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4857198
Commit-Queue: Struan Shrimpton <sshrimp@google.com>
Reviewed-by: Garrett Beaty <gbeaty@google.com>
Cr-Commit-Position: refs/heads/main@{#1195423}
Currently, in the rewriting rule of the test runners, "~NOTFOUND" is
used as an invalid URL. However, '~' is NOT a forbidden character
according to the URL standard.
Chromium is addressing [1] this non-compliant behavior and will permit
'~' character in URL hosts. Thus, we can no longer use "~NOTFOUND" as
it will become a valid URL.
As an alternative, any invalid character is acceptable. Let's use '^'
because '^' is a forbidden character both before and after the CL [1].
The PR for WPT Chrome runner is:
https://github.com/web-platform-tests/wpt/pull/41840/
[1]: https://crrev.com/c/4823237
Bug: 1416013
Change-Id: Id8767ab49b54e778da53113aa4995e098bdbee59
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4845589
Reviewed-by: Yoav Weiss <yoavweiss@chromium.org>
Commit-Queue: Hayato Ito <hayato@chromium.org>
Reviewed-by: Mathias Bynens <mathias@chromium.org>
Reviewed-by: Adam Rice <ricea@chromium.org>
Reviewed-by: Chidera Olibie <colibie@google.com>
Reviewed-by: danakj <danakj@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1194658}
In the iOS Blink port, running web tests is essential
to assess the level of support for various web features
and functionalities. This change includes necessary
additions to enable the execution of web tests on the
iOS Blink port.
co-author: mkim@ solved the timeout issue by adding
a custom UIApplication that overrides IsRunningTests.
Bug: 1421239
Change-Id: I8835097f628280c76d1213a16a29d6cdddd19ecb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4712370
Reviewed-by: Stephen McGruer <smcgruer@chromium.org>
Reviewed-by: Dave Tapuska <dtapuska@chromium.org>
Reviewed-by: Rohit Rao <rohitrao@chromium.org>
Commit-Queue: Gyuyoung Kim <gyuyoung@igalia.com>
Reviewed-by: Peter Beverloo <peter@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1191108}
Based on discussion in crbug.com/1475247, `rebaseline-cl --fill-missing`
still seems useful for downloading platform-agnostic expectations
without the round-trip of retrying infrastructure failures, but it's not
a feature users actively seek out. Therefore:
* Hide the `--{no-,}fill-missing` options with a deprecation message
(i.e., the feature is only available for infrastructure failures).
There's no automation that uses these options.
* Remove a redundant user prompt. It's no longer possible to continue
with incomplete results.
Bug: 1383284
Change-Id: Icdedba8e406252041da4fa7ccaf4c6f60ef1ce8f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4824713
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1190222}
Having a doc for repro bot failures locally is helpful and
I've got many questions in the past that can be answered by
such a doc. Hope this doc can be helpful for any new Chromium
developers.
Bug: 1467350
Change-Id: I9d4dc839c24004685b5b6113ce288cd47fdc7703
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4711854
Commit-Queue: Sven Zheng <svenzheng@chromium.org>
Reviewed-by: Jenny Zhang <jennyz@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1176748}
* Add new section on how to view the test results and what they are.
* Add new section on testing flag-specific, platform-specific, and
virtual configurations.
* Update builder list.
* Reference wptrunner support from the web test/WPT documents.
* Remove staleness warning because the project has stabilized.
* Replace image that is too large in the debugging section with a
smaller lower-quality version.
* General rewording.
Bug: 1442392
Change-Id: If659ea0195d73060b2da961538a8225480c81aeb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4654034
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1174325}
This change will allow us to create the virtual suite for SingleThreaded
tests and only run the single threaded version of those tests, without
affecting other virtual suites.
This will allow to start running most WPT tests with threaded
compositing while keep running a small amount of the tests
single threaded.
Bug: 1463799
Change-Id: Ifa885d23fffcb1277726aecfc7b66bf2edc39442
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4698657
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Yotam Hacohen <yotha@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1172584}
Currently, failing imported tests are grouped by most specific OWNERS
file for notification. However, some WPT directories only have
DIR_METADATA and not OWNERS; failures in these directories will not get
any notifications, even if the DIR_METADATA has opted in.
This change groups failures by most specific DIR_METADATA instead, since
DIR_METADATA are generally better maintained and hold information for
filing bugs. The team email is now CC'ed (if available), but continue
CC'ing OWNERS from the same directory (if available). For most top-level
directories with both DIR_METADATA and OWNERS, this notification
behavior represents no change.
Update the documentation accordingly.
Bug: 1456825
Change-Id: Idcade096908f58ac9d214297627d614231d68969
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4662112
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#1168480}
To help users understand WPT metadata, this change adds equivalent
examples in TestExpectations/*-expected.txt format.
Also:
* Update the finalized table of properties.
* General rewording.
Bug: 1442392
Change-Id: Ifcf856fc9fe2d48f7e658e19ce5db596d1a78d9b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4649951
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1163272}
This change restores and documents a `run_wpt_tests.py --no-headless`
flag that is plumbed through to `wpt run`. `--no-xvfb` has been removed,
since the need for Xvfb is directly tied to headless mode.
Bug: 1440021, 1442392
Change-Id: I938a64c5539b86042260b0ecd87d120b1202aa94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4650693
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#1163076}
This change finalizes targets the user guide will teach developers to
build. The names were picked to be:
* Short: Including `tests` or `isolate` is redundant.
* Descriptive: Make it obvious `wpt_tests_isolate` tests against chrome.
* Consistent: Other targets follow a `<product>_<test-type>` convention:
* `system_webview_wpt`
* `content_shell_crash_test`
* `android_webview_junit_tests`
Bug: 1442392
Change-Id: Id227c981538b624d3ad5e8c829e67c0152920b5a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4629897
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Andrew Grieve <agrieve@chromium.org>
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: Ali Juma <ajuma@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1161514}
And disable the test_shared_library_unittests test on Mac, as it
fails on the bot due to the DLL not being present, just like on
Windows component builds.
R=lukasza@chromium.org
Bug: 1386212, 1442273
Change-Id: I73ca099d28dd69a13cc013b69e4ff73d412fd3b2
Cq-Include-Trybots: luci.chromium.try:android-rust-arm32-rel,android-rust-arm64-dbg,android-rust-arm64-rel,linux-rust-x64-dbg,linux-rust-x64-rel,mac-rust-x64-rel,win-rust-x64-dbg,win-rust-x64-rel
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4544956
Commit-Queue: Dirk Pranke <dpranke@google.com>
Auto-Submit: danakj <danakj@chromium.org>
Commit-Queue: danakj <danakj@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Cr-Commit-Position: refs/heads/main@{#1146235}
The warning about clobbering all-pass statuses (crrev.com/c/4334979) is
not useful when the affected builder/test passes expectedly (so it
generates no `actual_text` to download). In this change, `rebaseline-cl`
instead automatically generates a dummy baseline that `run_web_tests.py`
and `optimize-baselines` recognize as all-pass. This preserves subtest
-level coverage for failing platforms.
These dummy baselines should rarely be necessary, so this change's
impact on checkout size should be negligible. The dummy baselines are
self-documenting and link to the relevant web test documentation, which
this change updates.
Bug: 1324638
Test: To test the `run_web_tests.py` side:
Test: For `external/wpt/badging/badge-success.https.html`, create an
Test: abbreviated all-pass baseline for linux (host platform) and a
Test: failing baseline for win.
Test: `./run_web_tests.py -t Default external/wpt/badging` succeeds
Change-Id: I4a94fb4fda5c566ab0b5838c8ac913c6ee6cccce
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4385729
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Jonathan Lee <jonathanjlee@google.com>
Cr-Commit-Position: refs/heads/main@{#1124532}
While not really about histograms, this seems like the best spot to put this information.
I don't have much direct information here in terms of how this is configured, but not sure how necessary it is (and honestly, how this is configured, maybe the test doc reviewer may know more).
Bug: 1414862
Change-Id: Idf39a4b9ee3ac85668da8c13b18427ef60dd156b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4240979
Commit-Queue: Robert Kaplow <rkaplow@chromium.org>
Reviewed-by: Alexei Svitkine <asvitkine@chromium.org>
Reviewed-by: David Jacobo <djacobo@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1104509}
Note to sheriffs: If this CL causes a few new failures,
please fix the failures by adding new
lines to TestExpectations rather than reverting.
In this CL we are adding mac13 and mac13-arm64 builders. In order to
achieve this the CL also adds the new baselines for the new builders
by running the rebaseline tool on the new builds.
Also add the ports to the necessary files in order for the builders
to properly run on Mac13 and Mac13-arm64.
Change-Id: Ifc230fc805ce1b44c438d1de21f08e7fe95a901f
Bug: 1372533
Cq-Include-Trybots: luci.chromium.try:linux-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac10.13-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac10.14-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac10.15-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac11.0-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac11.0.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac12.0-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac12.0.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac13-blink-rel
Cq-Include-Trybots: luci.chromium.try:mac13.arm64-blink-rel
Cq-Include-Trybots: luci.chromium.try:win10.20h2-blink-rel
Cq-Include-Trybots: luci.chromium.try:win11-blink-rel
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4197702
Reviewed-by: Jonathan Lee <jonathanjlee@google.com>
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Nihar Damarasingu <nihardamar@google.com>
Cr-Commit-Position: refs/heads/main@{#1101843}
This page is now pointed to by builds using RTS to explain what's
going on. Added information on what's happening and how to disable
Bug: 1344885
Change-Id: I2588ca4fb2c01cf989f4015aaff6dddfa515e525
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4157733
Commit-Queue: Struan Shrimpton <sshrimp@google.com>
Reviewed-by: Stephanie Kim <kimstephanie@google.com>
Cr-Commit-Position: refs/heads/main@{#1091551}
Running full Chromium with code coverage no longer requires disabling
the sandbox. It was fixed in 2019.
Found while working on crbug.com/1399354
Bug: None, 972232
No-Try: True
Change-Id: Ieeb86f162ebc0f955ace8d511ee3bf7c9b61cf92
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4129804
Reviewed-by: Zhaoyang Li <zhaoyangli@chromium.org>
Commit-Queue: Kai Ninomiya <kainino@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1089024}
This is a follow up change for crbug/1381274.
After this change, wpt-importer and rebaseline tool should trigger
win-rel, instead of win10_chromium_x64_rel_ng.
Cleaned up some lint error at the same time.
Bug: 1383534
Change-Id: I824490600f7e032afd38c8b8f132b6d78e165de3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4024484
Reviewed-by: Xianzhu Wang <wangxianzhu@chromium.org>
Commit-Queue: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#1070619}
Previously for a virtual test to skip the base tests, we needed to add
the following entries in NeverFixTests:
base/tests/* [ Skip ]
virtual/suite/base/tests/* [ Pass ]
and we needed to keep them in sync with VirtualTestSuites.
With the "exclusive_tests" field, we can define a virtual suite and
skip all or selected base tests at one place, which is easier to keep
consistency.
Bug: 1379095
Change-Id: Ic76a76eadcca3b7786906a925e5011c43ed59ce0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4010728
Reviewed-by: Weizhong Xia <weizhong@google.com>
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1069366}
Flake portal has been turned off (a little ahead of schedule), make sure
all links point to LUCI Analysis instead.
gtest_flake_tips.md still needs to be updated, but needs more than just
changing links, so will be done in a separate CL.
Bug: 1379496
Change-Id: If0d0ac4572dde7c4dc55eb39abaf31f222dc7489
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3995864
Auto-Submit: Matthew Warton <mwarton@google.com>
Reviewed-by: Struan Shrimpton <sshrimp@google.com>
Commit-Queue: Struan Shrimpton <sshrimp@google.com>
Cr-Commit-Position: refs/heads/main@{#1065995}
