Added platform config to allow devs to specify on which platforms
to run a virtual test suites, in the hope we can reduce the number
of tests on some platforms.
Update the documentation also.
Bug: 1234863
Change-Id: I2470671b5efb774275bca83da72d5ef6b698e0aa
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3530131
Reviewed-by: Xianzhu Wang <wangxianzhu@chromium.org>
Commit-Queue: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#984521}
This CL sets testing/android/docs/README.md as a master document
for Android chrome test-related information, and completes it
with links to various resources in the repo.
Bug: 1303119
Change-Id: I0439472d3224d4450f50ff0ed59ef3516b8ca0cb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3450015
Reviewed-by: Henrique Nakashima <hnakashima@chromium.org>
Reviewed-by: Andrew Grieve <agrieve@chromium.org>
Commit-Queue: Jinsuk Kim <jinsukkim@chromium.org>
Cr-Commit-Position: refs/heads/main@{#982857}
This CL adds a description for how to debug particular
WPT tests with debuggers in web_tests_in_content_shell.md
and put the link to it from web_platform_tests.md.
Investigating and fixing failures of WPT tests would be
one of the best ways for new contributors to start their
contributions. And using debuggers would be one of the
best ways to understand the inside of Chromium for them.
Adding a clear description for how to debug particular
WPT tests with debuggers in the document would help them
start their contributions.
Change-Id: If08409c1cc2f8e7d6e786e58852129e3cb6950dc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3371519
Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Cr-Commit-Position: refs/heads/main@{#960705}
Change this to --timeout-ms to align with other parameter names,
e.g. --exit-after-n-crashes-or-timeouts, --skip-timeouts,
--initialize-webgpu-adapter-at-startup-timeout-ms.
Bug: 1095123
Change-Id: I039c342249c7392ffdb850eebbbf63a51fe7cd8a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3366217
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Xianzhu Wang <wangxianzhu@chromium.org>
Commit-Queue: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/main@{#955755}
Update web_tests.md to fix some problems in Test Harness Options section
- Remove --renderer-startup-dialog option description from the table
because run_web_tests.py seems to no longer have
--renderer-startup-dialog option.
- Let the Meaning of --reset-results display correctly. Vertical bar
in the inline code block inside the table isn't displayed correctly. Let
it out from the inline code block and use HTML character code instead.
Update AUTHORS
- Add a new individual contributor Takahiro because this is his first
patch
Change-Id: Ie374a0a77e27fabf1d7b522845fcf0d9407ddd63
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3357156
Reviewed-by: Takashi Toyoshima <toyoshim@chromium.org>
Commit-Queue: Takashi Toyoshima <toyoshim@chromium.org>
Cr-Commit-Position: refs/heads/main@{#955567}
This summarizes the state of how test results are fetched and used on
chromium bots after the work done in crbug.com/1135718.
This doc should also provide pointers/advice for folks introducing a
new test type or test harness to Chromium.
Bug: 1135718
Change-Id: Ia39a3a4101c71bb8591f7875cb812ef56b53a823
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3293843
Reviewed-by: Chan Li <chanli@chromium.org>
Reviewed-by: Erik Staab <estaab@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/main@{#955439}
This was switched to vpython3 because run_web_tests.py
specifies vpython3.
It turns out the other scripts specify vpython and
while vpython3 almost works, it fails with some other
messages such as (for run_blink_wptserve.py):
ALPN negotiated protocol h3
Connection close sent (code 0x0, reason )
<repeatedly>
Bug: crbug.com/1280472
Change-Id: Id455ea0f2fea860f76a4b0e17de9490634779c40
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3344086
Auto-Submit: Johann Koenig <johannkoenig@google.com>
Reviewed-by: Kent Tamura <tkent@chromium.org>
Commit-Queue: Kent Tamura <tkent@chromium.org>
Cr-Commit-Position: refs/heads/main@{#952206}
vpython3 is specified in run_web_tests.py
using python or python3 leads to errors such as:
or
ImportError: No module named requests
ModuleNotFoundError: No module named 'aioquic'
Change-Id: I53660e5222020d9266c0ff3ac088e1ff99ad7e27
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3341132
Auto-Submit: Johann Koenig <johannkoenig@google.com>
Reviewed-by: Kent Tamura <tkent@chromium.org>
Commit-Queue: Kent Tamura <tkent@chromium.org>
Cr-Commit-Position: refs/heads/main@{#951859}
Although web platform tests are different from web tests, the document
does not clearly mentions they are different. I misunderstood
web test as web platform tests. I have updated web_tests and
writing_web_tests to clarify they are different.
Also, I have added the way to debug WPTs in writing_web_tests.
This is something like my brain dump of what I have learned and used
recently. That includes the way to run test one by one, logging,
and the way to see HTTP headers for WPTs.
Change-Id: I1673e15b2c8ec7f8b17665aeb87b16d81de0d62a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3319983
Reviewed-by: Kouhei Ueno <kouhei@chromium.org>
Reviewed-by: Takashi Toyoshima <toyoshim@chromium.org>
Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Cr-Commit-Position: refs/heads/main@{#949943}
Plain old python fails with:
ImportError: No module named six.moves.queue
Using vpython should fix that.
Also replace "0" in the output dir with the name of a task that it
would really look like.
Bug: None
Change-Id: I51138442a368edc0e4c88f2f8ab24262a6d233e0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3152067
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/main@{#919996}
This is a reland of 8f26bc7ead
Original change's description:
> Optimize data dependency for blink_web_tests
>
> Some baselines are only useful for certain platforms. Explicitly list
> such dependency so that we don't need download unnecessary baselines to
> swarming bots. Added a presubmit check to ensure we do not miss any
> folders under web_tests.
>
> This should be able to reduce the overhead on the swarming bots. After
> this change, we saved 38524 files or directories on Windows.
> This is 20.8% of the web_tests folder.
>
> Bug: 1230939
> Change-Id: I805f03f9f002a05b0fb1456271aa9b383c9bd3c2
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3139556
> Commit-Queue: Weizhong Xia <weizhong@google.com>
> Reviewed-by: Xianzhu Wang <wangxianzhu@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Cr-Commit-Position: refs/heads/main@{#918838}
Bug: 1230939, 1247545
Change-Id: Ia930728c00bc97f44e790b5beaebae701772bf80
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3149438
Commit-Queue: Weizhong Xia <weizhong@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Cr-Commit-Position: refs/heads/main@{#919846}
This reverts commit 8f26bc7ead.
Reason for revert: https://ci.chromium.org/ui/p/chromium/builders/try/chromium_presubmit/1376442/overview
Original change's description:
> Optimize data dependency for blink_web_tests
>
> Some baselines are only useful for certain platforms. Explicitly list
> such dependency so that we don't need download unnecessary baselines to
> swarming bots. Added a presubmit check to ensure we do not miss any
> folders under web_tests.
>
> This should be able to reduce the overhead on the swarming bots. After
> this change, we saved 38524 files or directories on Windows.
> This is 20.8% of the web_tests folder.
>
> Bug: 1230939
> Change-Id: I805f03f9f002a05b0fb1456271aa9b383c9bd3c2
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3139556
> Commit-Queue: Weizhong Xia <weizhong@google.com>
> Reviewed-by: Xianzhu Wang <wangxianzhu@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Cr-Commit-Position: refs/heads/main@{#918838}
Bug: 1230939
Change-Id: Ice2c5d057c1bb325b9f14d8e1b5224be00955ec2
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3147151
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Jiewei Qian <qjw@chromium.org>
Reviewed-by: Jiewei Qian <qjw@chromium.org>
Owners-Override: Jiewei Qian <qjw@chromium.org>
Cr-Commit-Position: refs/heads/main@{#919121}
Some baselines are only useful for certain platforms. Explicitly list
such dependency so that we don't need download unnecessary baselines to
swarming bots. Added a presubmit check to ensure we do not miss any
folders under web_tests.
This should be able to reduce the overhead on the swarming bots. After
this change, we saved 38524 files or directories on Windows.
This is 20.8% of the web_tests folder.
Bug: 1230939
Change-Id: I805f03f9f002a05b0fb1456271aa9b383c9bd3c2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3139556
Commit-Queue: Weizhong Xia <weizhong@google.com>
Reviewed-by: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Cr-Commit-Position: refs/heads/main@{#918838}
There was a stray (really old!) reference to `xcodebuild` instead
of //out for build directories, and I changed a couple places from
talking about //out/Debug or //out/Release to //out/Default just
to be more consistent with how we talk about build directory names
elsewhere.
Change-Id: I5410c5abc7a57bb4c71d42839a3efc142ab152c4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3138433
Reviewed-by: Christian Biesinger <cbiesinger@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Cr-Commit-Position: refs/heads/main@{#917368}
1. Use pushd/popd when switching directories, and popd at the end
to switch directory to the original directory (instead of in a deep
subdirectory containing the last rebaselined test).
2. Emit blink_tool.py optmize-baselines to remove redundant baselines.
Change-Id: I03916a41d7c31f306757a55d767dbfb90dc1eebd
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3123088
Reviewed-by: Xiaocheng Hu <xiaochengh@chromium.org>
Reviewed-by: Aleks Totic <atotic@chromium.org>
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Cr-Commit-Position: refs/heads/main@{#916101}
This reverts commit 93e16e0560.
Reason for revert: <People have concern on this, revert first>
Original change's description:
> Create AllTestsByDirectories.json for blink_web_tests
>
> So that we can speed up collecting test cases. This json file is
> a dictionary with directory names as the key, and test cases under
> the directory in a list as the value. Both keys and values are
> sorted.
>
> Added a presubmit check to ensure this file is updated when
> there is change to test cases.
>
> Bug: 982208
> Change-Id: Ibacaee4b5159a82806d210ce7ec42a84e5f9c531
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3067372
> Commit-Queue: Weizhong Xia <weizhong@google.com>
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Reviewed-by: Xianzhu Wang <wangxianzhu@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#912423}
Bug: 982208
Change-Id: I0c32c81ab4e520ee10630a8ac70e7989ef066465
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3100379
Commit-Queue: Weizhong Xia <weizhong@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Cr-Commit-Position: refs/heads/master@{#912665}
So that we can speed up collecting test cases. This json file is
a dictionary with directory names as the key, and test cases under
the directory in a list as the value. Both keys and values are
sorted.
Added a presubmit check to ensure this file is updated when
there is change to test cases.
Bug: 982208
Change-Id: Ibacaee4b5159a82806d210ce7ec42a84e5f9c531
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3067372
Commit-Queue: Weizhong Xia <weizhong@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Xianzhu Wang <wangxianzhu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#912423}
- Combine rebaseline documentation in web_tests.md into
web_test_expectations.md
- Format and add more details in #Testing-Runtime-Flags in
web_tests.md, and add a link to it from web_test_expectationd.md.
- Prefer --flag-specific in some cases.
- Add rebaseline script as a rebaseline method.
- Mention the glob syntax of test expectations, and add a link to the
more complete and up-to-date documentation of the syntax.
Bug: 1234863
Change-Id: Ia8a1dd140db3855fce391974a64ac742c6f8dfc1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3064798
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Weizhong Xia <weizhong@google.com>
Cr-Commit-Position: refs/heads/master@{#907385}
This patch adds some instructions for getting started with gtest unit
tests (and fixes the broken link “Run Web Tests locally”).
I’m a gtest novice, but hopefully I’ve found the right terminology.
Change-Id: I68cba9dac943a95d6593e6f089f5290a5aaa61f1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2868185
Commit-Queue: Delan Azabani <dazabani@igalia.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Cr-Commit-Position: refs/heads/master@{#886648}
Added the option --flag-specific=<foo>.
Currently supporting only 'highdpi' for flag-specific.
New experimental highdpi builder is added which gets triggered with this option.
Results from the new builder are fetched and rebaselined under
third_party/blink/web_tests/flag-specific/<foo>/. Other builders are ignored with this option.
Rebaselined the highdpi tests using these changes.
Bug: 1068400
Change-Id: I5638eb225398159caa06963c9ef5f3991b1a7a3b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2545863
Reviewed-by: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Stephen McGruer <smcgruer@chromium.org>
Commit-Queue: Preethi Mohan <preethim@google.com>
Cr-Commit-Position: refs/heads/master@{#859222}
commit 2389045 (Move content/shell/$process/web_test
to content/web_test/$process) has updated the
path to test_info_extractor.h
The current patch fixes the link to "protocol mode"
Bug: 1174454
Change-Id: Ib22507db10f75d0e5cdabb53aa13bd38ef52bb53
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2673426
Reviewed-by: Kentaro Hara <haraken@chromium.org>
Commit-Queue: Arnaud Mandy <arnaud.mandy@intel.com>
Cr-Commit-Position: refs/heads/master@{#850567}
This reverts commit 8da79b7118.
Reason for revert: We're looking into other methods of test selection because our filtering of source files conflicts with state sharing between test files.
Original change's description:
> Proof of concept for test selection via GN
>
> This CL implements gn based unittest filtering. The files in the rts_exclude_file are manually selected for this example, but will later be populated using the smart logic in rts-chromium.
> These two cases (source_set and test template) cover most of the test files. I haven't seen any other templates with unittests, but even if there are some this would be a great start. This also doesn't cover webtests, but I think there's enough exclusion power here to save a lot of compute.
> This also adds a linux-rts builder to the mb config to prepare the experimental builder.
>
> Bug: 1145216
> Change-Id: I7343249f3df0739e39cfe351e5714d7978059d86
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2551276
> Commit-Queue: Gregory Guterman <guterman@google.com>
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Reviewed-by: Nodir Turakulov <nodir@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#841248}
TBR=dpranke@google.com,nodir@chromium.org,chromium-scoped@luci-project-accounts.iam.gserviceaccount.com,guterman@google.com
Bug: 1145216
Change-Id: I96ec160c5f2e08e8be7faed7c6126aff3f82c053
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2667128
Reviewed-by: Gregory Guterman <guterman@google.com>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Gregory Guterman <guterman@google.com>
Cr-Commit-Position: refs/heads/master@{#849942}
