- The protocol mode
- Running HTTP web tests in Content Shell
- Running WPT tests in Content Shell
- Debugging with --single-process
- Debugging web tests
- Some corrections
Change-Id: I1d23d0f8a75b9d597edab153e4ec0ea9cdb7cb67
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1541863
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#645002}
Before crrev.com/c/1277654 when we created text results for ref tests
most of which don't need text baselines, we let reftest's text baseline
be optional, so that most reftests can ignore the text results, while
a few of them for which the text result is significant can still check
the text results.
After crrev.com/1277654, reftests no longer produce text results thus
don't need text baseline, unless a test explicitly produces text result.
In this case, we can just let the test determine if the text baseline
is needed, instead of run_web_tests.py checking existance of the
text baselines.
About existing reftests producing text results but without text
baselines:
1. for tests that don't actually need text results, modify them so that
they no longer produce text results. This includes some tests
previously calling testRunner.dumpAsTextWithPixelResults() just
to dump pixel results.
2. for tests that produce meaningful text results, create text
baselines.
3. for tests that don't produce text results intentionally, but the
text results are created because of some console messages, etc.
also create text baselines. There are several cases:
a) the message shows the purpose of the test, e.g. intentionally
testing with some invalid values. New text baselines are created
in this CL.
b) the message shows some problem of the test that doesn't need to
fix for now, e.g. the test is using some deprecating API. New
baselines are created in this CL.
c) the message indicates some fatal bug in the test, e.g. using
undefined variable, invalid JavaScript grammar, etc. The bugs are
fixed in this CL.
Bug: 703899
Change-Id: I82a6b25c61d1fb794840fdaa0da248909a690592
Reviewed-on: https://chromium-review.googlesource.com/c/1298120
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#604116}
SHERIFFS: PLEASE DO NOT REVERT THIS CL BECAUSE OF A SMALL AMOUNT OF LAYOUT
TEST FLAKINESS. If a few layout tests begin to show flakiness after this CL
lands, please add them to TestExpectations and email me (masonfreed@) to
investigate. Though I tried to identify all potentially flaky tests, a few
may have slipped through.
With this CL, the --enable-display-compositor-pixel-dump flag becomes the default
for content_shell. With this flag in place, layout test pixel dumps are performed
from the browser side, instead of from the renderer side. Note that to avoid a
significant amount of layout test flakiness, another change was also made to
not add the --run-all-compositor-stages-before-draw flag by default. There is
a bug (crbug.com/894613) tracking that problem separately.
With the flip of this switch, several modifications had to be made to the
TestExpectations file. First, there are a number of tests that change their
appearance slightly when being captured from the browser, and these tests need
to be rebaselined. These are summarized below, and will be rebaselined as a
separate CL, once this one lands and has had time to stabilize.
These bugs track the items added to TestExpectations:
- crbug.com/887140: HDR support
- crbug.com/881040: Media controls now contain an overflow menu.
- crbug.com/667551: A bunch of tests are listed under this bug, and just
require rebaselining to fix non-material single-pixel
antialiasing failures.
- crbug.com/891427: These either start failing, or become flaky, when the
--enable-display-compositor-pixel-dump flag is enabled.
They need to be debugged prior to re-enabling.
- crbug.com/895556: These tests double their background size when the flag
is enabled. They need to be fixed or rebaselined.
Bug: 667551, 891427, 881040, 887140, 894613, 895556
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel
Change-Id: I46946e6377f659c9dedc0dfaa20e7658e8cc519d
Reviewed-on: https://chromium-review.googlesource.com/c/1213864
Commit-Queue: Mason Freed <masonfreed@chromium.org>
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Reviewed-by: Peter Beverloo <peter@chromium.org>
Reviewed-by: Stephen Chenney <schenney@chromium.org>
Cr-Commit-Position: refs/heads/master@{#603127}
Now for tests that don't call any dumpAsXXX methods, they by default
generate pixel results only, and run_web_tests.py just check their
pixel results against the corresponding pixel baselines only.
Provide testRunner.dumpAsLayout() and dumpAsLayoutWithPixelResults()
for tests that do need to dump the layout tree.
We still dump layout tree when running such tests in content_shell
--run-web-tests command line (instead of running from run_web_tests.py
in protocol mode).
Changes to layout tests and their expectations:
- Use testRunner.dumpAsLayoutWithPixelResults() or dumpAsLayout()
in some tests that still requires the layout dump:
- editing/ tests need layout dump to show the caret position.
- two printing/ tests need layout dump to show how the bug is
fixed.
In the future, we may just get rid of layout dumps if it's
feasible.
- Remove testRunner.dumpAsTextWithPixelResults() from some tests
which used the function just to suppress layout output. They didn't
actually want text output but we hadn't had a way to dump image only
until we had the ImageFirst feature.
- Previous platform-specific -expected.txt for tests under previous
ImageFirst directories are removed and the common baselines are
created because the baselines are no longer platform-specific without
the layout dump.
Extra -expected.txt baselines are not deleted in this CL. Will
follow-up in https://chromium-review.googlesource.com/c/chromium/src/+/1286894.
Bug: 703899
Cq-Include-Trybots: luci.chromium.try:linux-blink-gen-property-trees;luci.chromium.try:linux_layout_tests_slimming_paint_v2;luci.chromium.try:linux_layout_tests_layout_ng
Change-Id: Ifcf7fea90684e133b4222909ee508469d0c985e0
Reviewed-on: https://chromium-review.googlesource.com/c/1277654
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Reviewed-by: Mike West <mkwst@chromium.org>
Cr-Commit-Position: refs/heads/master@{#602161}
Previous to this CL, these autoscroll/animation tests passed only because there
is a default surface deadline (typically 4 frames) in place. Surface activation
does not happen automatically, because in layout test mode, rasterization is not
performed until the very end of the test, for image capture. For the tests in
question, that deadline was previously always being hit, causing the surface to
be activated, triggering the monitored scroll events and passing the test.
When the --enable-display-compositor-pixel-dump flag is enabled, however, the
default surface activation deadline is changed to be infinite (no deadline). As
a result, all such autoscroll/animation tests fail.
With this CL, those tests that looked at animation behavior have been modified
to call a new continuouslyRunAnimations() javascript method, which sets a flag
that causes full rasterization to be included in the lifecycle update that was
already taking place. This triggers surface activation, and runs the
animations, allowing these tests to pass even with an infinite deadline.
Bug: 883547, 874162
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_layout_tests_slimming_paint_v2;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: I0d9e1ae7d10fab186207cdfbc8e2f85f60d99f35
Reviewed-on: https://chromium-review.googlesource.com/c/1229354
Commit-Queue: Mason Freed <masonfreed@chromium.org>
Reviewed-by: Kent Tamura <tkent@chromium.org>
Reviewed-by: Philip Rogers <pdr@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Reviewed-by: Timothy Dresser <tdresser@chromium.org>
Cr-Commit-Position: refs/heads/master@{#595898}
The test runner is `run_web_tests.py` (and not `run_webkit_tests.py`).
This patch also makes smaller tweaks and improvements to the doc.
Change-Id: I559cf54e98135f1fe90986e1cf06d140ca911918
Reviewed-on: https://chromium-review.googlesource.com/1203991
Commit-Queue: Mathias Bynens <mathias@chromium.org>
Reviewed-by: Kent Tamura <tkent@chromium.org>
Reviewed-by: Victor Costan <pwnall@chromium.org>
Cr-Commit-Position: refs/heads/master@{#588805}
This CL adds the `is_regression` per-test field (though it won't
yet be set by anything), and adds a bunch of text clarifying the
differences between failures, unexpected results, and regressions.
It also does some general editorial cleanup, fixing typos and
re-sorting some of the enumerated values.
BUG=822078, 837047
Change-Id: If78f6ef66c01d7c2924bfd98589832fe9299968f
Reviewed-on: https://chromium-review.googlesource.com/1096486
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Chan Li <chanli@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#566615}
This doesn't include those in 'if' statements.
Bug: 703768
TBR: already agreed in blink-dev; too many files.
Cq-Include-Trybots: master.tryserver.chromium.linux:linux_layout_tests_layout_ng
Change-Id: I499b91695e177d2a498d96cf20537ee146e22d06
Reviewed-on: https://chromium-review.googlesource.com/1057947
Reviewed-by: Xianzhu Wang <wangxianzhu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#558467}
All code in third_party/WebKit/Tools/Scripts was moved to
third_party/blink/tools.
This CL also fixes references to old names of scripts in Tools/Scripts.
* third_party/WebKit/PRESUBMIT.py:
Remove dead code. This PRESUBMIT.py doesn't use
|audit_non_blink_usage|.
Bug: 829697
Change-Id: I6c482a795708fa207124cb2030a4fdaa8ceac343
Reviewed-on: https://chromium-review.googlesource.com/1038863
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Kent Tamura <tkent@chromium.org>
Cr-Commit-Position: refs/heads/master@{#555366}
which is the new name of run-webkit-tests.
This CL also replaces references to "run-webkit-tests" with
"run_web_tests.py".
* //blink/tools/run_layout_tests.py
Show a message, and run run_web_tests.py.
* //BUILD.gn
"webkit_layout_tests" doesn't need //third_party/WebKit/Tools/Scripts
any longer.
* //testing/buildbot/gn_isolate_map.pyl
Replace run-webkit-tests with run_web_tests.py.
* //third_party/blink/tools/run_web_tests.{bat,py}
Added.
* //third_party/WebKit/Tools/Scripts/run-webkit-tests
Show a message.
* Other files: Really trivial changes. In many files, this CL
replaces "run-webkit-tests" with "run_web_tests.py" in comments or
strings.
Note that we decided to rename LayoutTests to web_tests.
https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/KKNbuzj-3HY/H8FWgtKrBgAJ
Bug: 829697
Change-Id: I899457cb3bcdb306aa47843a4c767c5980d9edc0
Reviewed-on: https://chromium-review.googlesource.com/1025543
Commit-Queue: Kent Tamura <tkent@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553452}
It will supersedes //third_party/WebKit/Tools/Scripts/lint-test-expectations.
lint-test-expectaions isn't removed yet due to dependency from
build/scripts/slave/chromium/.
Bug: 829697
Change-Id: I213b413304240e2f8672a1d2209a2800331b1ade
Reviewed-on: https://chromium-review.googlesource.com/1025437
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Kent Tamura <tkent@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553357}
The suffix part of the new name is 'tool' instead of 'patch' because
this command doesn't handle patches so much.
Bug: 829697
Change-Id: I90f34b6260032a7bbaf562d9b58ffc1a55c3922f
Reviewed-on: https://chromium-review.googlesource.com/1018700
Commit-Queue: Kent Tamura <tkent@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#552382}
A fresh WPT import checkout is very likely to fail when no GitHub
credentials are used due to the number of API requests we make vs.
GitHub's rate limits for unauthenticated requests.
Make things more obvious by:
* Allowing the credentials to be specified as environment variables in
addition to a JSON file (when both are set, the JSON file specified in
the command-line has precedence).
* Printing a warning in test_importer.py when no credentials are set and
pointing to the documentation in //docs/testing.
* Expanding the error message in test_exporter.py to point to the same
location.
Bug: 816390
Change-Id: I5508e024e8a13e6f597f027cd157d7abbe9b2fe1
Reviewed-on: https://chromium-review.googlesource.com/937462
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Commit-Queue: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Cr-Commit-Position: refs/heads/master@{#539164}
--allow-local-commits was removed in 4a301c5 ("Remove the
--allow-local-commits flag from wpt-import") so stop recommending its usage.
Additionally, given the amount of export commits we have in
web-platform-tests it is likely that unauthenticated runs will fail, so
explain how to use the --credentials-json option in both wpt-export and
wpt-import.
Bug: 816390
Change-Id: I72d7bc6dcb80d0db372e9dac1176f68d1b812bcb
Reviewed-on: https://chromium-review.googlesource.com/937241
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Commit-Queue: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Cr-Commit-Position: refs/heads/master@{#539116}
Replaces mentions of --interactive with --test-launcher-interactive and
one stale --dialog with --ui.
Bug: None
Change-Id: I6d2f80aae8b4282fdb1bd57bee2dd52d86b43250
Reviewed-on: https://chromium-review.googlesource.com/929342
Commit-Queue: Peter Kasting <pkasting@chromium.org>
Reviewed-by: Peter Kasting <pkasting@chromium.org>
Cr-Commit-Position: refs/heads/master@{#538221}
This introduces a new base class, TestBrowserUI, and reorganizes the existing
test code into a series of virtual functions so tests can provide the necessary
implementation for non-dialog cases.
This also adds a single consumer of this new base class, InfoBarUITest, to
verify it works.
Bug: 686285
Change-Id: I2b5a7c86347f5042ac5218c9fa90e94f68adf577
Reviewed-on: https://chromium-review.googlesource.com/804953
Commit-Queue: Peter Kasting <pkasting@chromium.org>
Reviewed-by: Scott Violet <sky@chromium.org>
Reviewed-by: Trent Apted <tapted@chromium.org>
Cr-Commit-Position: refs/heads/master@{#524855}
This clarifies the meaning of forward slashes in artifact file paths,
and also clarifies the directory these paths are relative to, when no
artifact permanent location is set.
Bug: 775619
Change-Id: I1f41d86273455123823350a39b239cc04b6a0216
Reviewed-on: https://chromium-review.googlesource.com/802092
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#521525}
which is a more logical and intuitive place than Rebaseline.
Besides, we add support for removing redundant all-PASS testharness.js
baselines that are not at the root. This may happen when a platfrom
result becomes all-PASS, when its fallback platforms are already
all-PASS, in which case we only download a new all-PASS baseline to the
platform. The previous optimization only looked at the root.
Bug: 768525
Change-Id: Icf4fa98850f5a72541a20d54217a64daef8cb389
Reviewed-on: https://chromium-review.googlesource.com/803795
Commit-Queue: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#521173}
If --additional-driver-flag is specified, it was weird to still let
--reset-results reset the existing baselines. There seems no use case
for that.
Now let --reset-results reset flag-specific baselines if --additional-
driver-flag is specified.
Bug: 660231
Change-Id: I6ee5a5f563a7f972fcece763256d7f99f3e97cbe
Reviewed-on: https://chromium-review.googlesource.com/740841
Reviewed-by: Chris Harrelson <chrishtr@chromium.org>
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#512061}
--copy-baselines:
If the actual result is different from the current baseline,
copy the current baseline into the *most-specific-platform*
directory, or the flag-specific generic-platform directory if
--additional-driver-flag is specified. See --reset-results.
--reset-results:
Reset expectations to the generated results in their existing location.
If --copy-baselines is specified, the copied baselines will be reset.
--new-flag-specific-baseline:
Deprecated. Replaced by --copy-baselines --reset-results
--add-platform-exceptions:
Deprecated.
The separation of --new-flag-specific-baseline into --copy-baselines and
--reset-results (existing) is to ease code review of flag-specific
rebaseline CLs. The developer can use --copy-baselines to generate the
first patch set, then use --reset-results to generate the second patch
set. The reviewer compares the first and the second patch sets to see
the difference of the new baselines.
Bug: 660231
Change-Id: Ia91a92ddb9e260b20566e49705f551df13ccacc3
Reviewed-on: https://chromium-review.googlesource.com/713760
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Steve Kobes <skobes@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#509077}
