This CL adds the `is_regression` per-test field (though it won't
yet be set by anything), and adds a bunch of text clarifying the
differences between failures, unexpected results, and regressions.
It also does some general editorial cleanup, fixing typos and
re-sorting some of the enumerated values.
BUG=822078, 837047
Change-Id: If78f6ef66c01d7c2924bfd98589832fe9299968f
Reviewed-on: https://chromium-review.googlesource.com/1096486
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Chan Li <chanli@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#566615}
This doesn't include those in 'if' statements.
Bug: 703768
TBR: already agreed in blink-dev; too many files.
Cq-Include-Trybots: master.tryserver.chromium.linux:linux_layout_tests_layout_ng
Change-Id: I499b91695e177d2a498d96cf20537ee146e22d06
Reviewed-on: https://chromium-review.googlesource.com/1057947
Reviewed-by: Xianzhu Wang <wangxianzhu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#558467}
All code in third_party/WebKit/Tools/Scripts was moved to
third_party/blink/tools.
This CL also fixes references to old names of scripts in Tools/Scripts.
* third_party/WebKit/PRESUBMIT.py:
Remove dead code. This PRESUBMIT.py doesn't use
|audit_non_blink_usage|.
Bug: 829697
Change-Id: I6c482a795708fa207124cb2030a4fdaa8ceac343
Reviewed-on: https://chromium-review.googlesource.com/1038863
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Kent Tamura <tkent@chromium.org>
Cr-Commit-Position: refs/heads/master@{#555366}
which is the new name of run-webkit-tests.
This CL also replaces references to "run-webkit-tests" with
"run_web_tests.py".
* //blink/tools/run_layout_tests.py
Show a message, and run run_web_tests.py.
* //BUILD.gn
"webkit_layout_tests" doesn't need //third_party/WebKit/Tools/Scripts
any longer.
* //testing/buildbot/gn_isolate_map.pyl
Replace run-webkit-tests with run_web_tests.py.
* //third_party/blink/tools/run_web_tests.{bat,py}
Added.
* //third_party/WebKit/Tools/Scripts/run-webkit-tests
Show a message.
* Other files: Really trivial changes. In many files, this CL
replaces "run-webkit-tests" with "run_web_tests.py" in comments or
strings.
Note that we decided to rename LayoutTests to web_tests.
https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/KKNbuzj-3HY/H8FWgtKrBgAJ
Bug: 829697
Change-Id: I899457cb3bcdb306aa47843a4c767c5980d9edc0
Reviewed-on: https://chromium-review.googlesource.com/1025543
Commit-Queue: Kent Tamura <tkent@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553452}
It will supersedes //third_party/WebKit/Tools/Scripts/lint-test-expectations.
lint-test-expectaions isn't removed yet due to dependency from
build/scripts/slave/chromium/.
Bug: 829697
Change-Id: I213b413304240e2f8672a1d2209a2800331b1ade
Reviewed-on: https://chromium-review.googlesource.com/1025437
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Kent Tamura <tkent@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553357}
The suffix part of the new name is 'tool' instead of 'patch' because
this command doesn't handle patches so much.
Bug: 829697
Change-Id: I90f34b6260032a7bbaf562d9b58ffc1a55c3922f
Reviewed-on: https://chromium-review.googlesource.com/1018700
Commit-Queue: Kent Tamura <tkent@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#552382}
A fresh WPT import checkout is very likely to fail when no GitHub
credentials are used due to the number of API requests we make vs.
GitHub's rate limits for unauthenticated requests.
Make things more obvious by:
* Allowing the credentials to be specified as environment variables in
addition to a JSON file (when both are set, the JSON file specified in
the command-line has precedence).
* Printing a warning in test_importer.py when no credentials are set and
pointing to the documentation in //docs/testing.
* Expanding the error message in test_exporter.py to point to the same
location.
Bug: 816390
Change-Id: I5508e024e8a13e6f597f027cd157d7abbe9b2fe1
Reviewed-on: https://chromium-review.googlesource.com/937462
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Commit-Queue: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Cr-Commit-Position: refs/heads/master@{#539164}
--allow-local-commits was removed in 4a301c5 ("Remove the
--allow-local-commits flag from wpt-import") so stop recommending its usage.
Additionally, given the amount of export commits we have in
web-platform-tests it is likely that unauthenticated runs will fail, so
explain how to use the --credentials-json option in both wpt-export and
wpt-import.
Bug: 816390
Change-Id: I72d7bc6dcb80d0db372e9dac1176f68d1b812bcb
Reviewed-on: https://chromium-review.googlesource.com/937241
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Commit-Queue: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Cr-Commit-Position: refs/heads/master@{#539116}
Replaces mentions of --interactive with --test-launcher-interactive and
one stale --dialog with --ui.
Bug: None
Change-Id: I6d2f80aae8b4282fdb1bd57bee2dd52d86b43250
Reviewed-on: https://chromium-review.googlesource.com/929342
Commit-Queue: Peter Kasting <pkasting@chromium.org>
Reviewed-by: Peter Kasting <pkasting@chromium.org>
Cr-Commit-Position: refs/heads/master@{#538221}
This introduces a new base class, TestBrowserUI, and reorganizes the existing
test code into a series of virtual functions so tests can provide the necessary
implementation for non-dialog cases.
This also adds a single consumer of this new base class, InfoBarUITest, to
verify it works.
Bug: 686285
Change-Id: I2b5a7c86347f5042ac5218c9fa90e94f68adf577
Reviewed-on: https://chromium-review.googlesource.com/804953
Commit-Queue: Peter Kasting <pkasting@chromium.org>
Reviewed-by: Scott Violet <sky@chromium.org>
Reviewed-by: Trent Apted <tapted@chromium.org>
Cr-Commit-Position: refs/heads/master@{#524855}
This clarifies the meaning of forward slashes in artifact file paths,
and also clarifies the directory these paths are relative to, when no
artifact permanent location is set.
Bug: 775619
Change-Id: I1f41d86273455123823350a39b239cc04b6a0216
Reviewed-on: https://chromium-review.googlesource.com/802092
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#521525}
which is a more logical and intuitive place than Rebaseline.
Besides, we add support for removing redundant all-PASS testharness.js
baselines that are not at the root. This may happen when a platfrom
result becomes all-PASS, when its fallback platforms are already
all-PASS, in which case we only download a new all-PASS baseline to the
platform. The previous optimization only looked at the root.
Bug: 768525
Change-Id: Icf4fa98850f5a72541a20d54217a64daef8cb389
Reviewed-on: https://chromium-review.googlesource.com/803795
Commit-Queue: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#521173}
If --additional-driver-flag is specified, it was weird to still let
--reset-results reset the existing baselines. There seems no use case
for that.
Now let --reset-results reset flag-specific baselines if --additional-
driver-flag is specified.
Bug: 660231
Change-Id: I6ee5a5f563a7f972fcece763256d7f99f3e97cbe
Reviewed-on: https://chromium-review.googlesource.com/740841
Reviewed-by: Chris Harrelson <chrishtr@chromium.org>
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#512061}
--copy-baselines:
If the actual result is different from the current baseline,
copy the current baseline into the *most-specific-platform*
directory, or the flag-specific generic-platform directory if
--additional-driver-flag is specified. See --reset-results.
--reset-results:
Reset expectations to the generated results in their existing location.
If --copy-baselines is specified, the copied baselines will be reset.
--new-flag-specific-baseline:
Deprecated. Replaced by --copy-baselines --reset-results
--add-platform-exceptions:
Deprecated.
The separation of --new-flag-specific-baseline into --copy-baselines and
--reset-results (existing) is to ease code review of flag-specific
rebaseline CLs. The developer can use --copy-baselines to generate the
first patch set, then use --reset-results to generate the second patch
set. The reviewer compares the first and the second patch sets to see
the difference of the new baselines.
Bug: 660231
Change-Id: Ia91a92ddb9e260b20566e49705f551df13ccacc3
Reviewed-on: https://chromium-review.googlesource.com/713760
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Steve Kobes <skobes@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#509077}
This removes mention of NeedsRebaseline/rebaseline-o-matic, which have
now been removed.
Bug: 692811
Change-Id: I9893eef07c7380186398a89a7cfdbcd524627193
Reviewed-on: https://chromium-review.googlesource.com/584941
Reviewed-by: Jeff Carpenter <jeffcarp@chromium.org>
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#489350}
- Update instructions for new Gerrit provisional PR flow
- Split out necessary information from additional information
- Add note that CLs with >1000 files will not be exported
- Rephrase a lot of things
Bug:
Change-Id: Ie40bdd4ca39b5efaf63fa7341d9df861d16e84dc
Reviewed-on: https://chromium-review.googlesource.com/567592
Commit-Queue: Jeff Carpenter <jeffcarp@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#486904}
Purpose: The purpose of this commit is to try to offer
clearer info/suggestions to sheriffs when import CLs
cause failures.
Change-Id: I90711d6bfe2f04ff0ec2a9d8c2f9caa92637f688
Bug: 726572
Reviewed-on: https://chromium-review.googlesource.com/516203
Reviewed-by: Matt Giuca <mgiuca@chromium.org>
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#477522}
The new option will create new flag-specific baselines when
--additional-driver-flag is specified. For example, we can create
new baselines under LayoutTests/flag-specific/enable-slimming-paint-v2
for --additional-driver-flag=--enable-slimming-paint-v2.
Change-Id: I159f59b70538807154cd2ecd7faa324091246478
Reviewed-on: https://chromium-review.googlesource.com/522916
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Walter Korman <wkorman@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#477101}
Use --reset-results instead of --new-test-results.
Use "webkit-patch rebaseline-cl" or "--reset-results --add-platform-exceptions"
instead of --new-baseline.
Still keep "--add-platform-exceptions" to create platform-version-specific
baselines. May remove it in the future if no one wants it.
BUG=660231
Change-Id: I9671c69a806e5cba1ccd838944e757a364a947a2
Reviewed-on: https://chromium-review.googlesource.com/523386
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Walter Korman <wkorman@chromium.org>
Cr-Commit-Position: refs/heads/master@{#477075}
