The Monorail component for code coverage was renamed from:
Tools>CodeCoverage to Infra>Test>CodeCoverage, and this CL updates the
references.
Change-Id: I886422c5b8f839570ade415354fe2c122ee35ff7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1726371
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Yuke Liao <liaoyuke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#682373}
- add support for resolving shared library deps on android
- add work around for missing local build id on android
- use devil.py to access the android device
- hook script up to swarming on android
Bug: 688737, 849904
Change-Id: I9ff2e3ae1fed279fef83a44b07c693c338b8ee13
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1583715
Commit-Queue: Jochen Eisinger <jochen@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#657138}
This field has been moved from the top level to the metadata field
in the JSON results. The field has also been removed from the JSON
results documentation.
Bug: None
Change-Id: I7da08bcaf71c48d52b49f9cc7595cf551ab94f3c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1565371
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Cr-Commit-Position: refs/heads/master@{#650159}
This CL had to be reverted because GPU bots were failing because the
results_merger.py script could not handle the test_name_prefix field
that was still getting populated in the JSON results. This CL will
add the metadata field along side the test_name_prefix field in the
Optional fields list in the results_merger.py script. It also removes
the checking of test_name_prefix in the gpu unit tests because the
test_name_prefix field will be removed from the top level of the
results in a future CL.
Bug: chromium:698902
Change-Id: I4867e60ca5a97d426d5fd1a4de08e26c14d326ae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1559492
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#649665}
This reverts commit 7c7d23256a.
Reason for revert: We reverted the removal of the test_name_prefix
top level field in crrev.com/c/1558856. So for the JSON results to be properly
merged, we need to re-add the test_name_prefix field to the
results_merger.py script.
Original change's description:
> Adds the metadata optional field to the standard JSON results
>
> A test runner may have several pieces of metadata. Instead of putting all
> of them in the top level, they can be put in the metadata field. I
> also removed the test name prefix, tags and expectations files fields
> from the top level. These fields will be put in the metadata field.
>
> Bug: chromium:698902
> Change-Id: I97973e4aed4cb2fe3761e047e48ce455422d300e
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1552490
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Reviewed-by: Stephen Martinis <martiniss@chromium.org>
> Commit-Queue: Rakib Hasan <rmhasan@google.com>
> Cr-Commit-Position: refs/heads/master@{#648900}
TBR=kbr@chromium.org,martiniss@chromium.org,rmhasan@google.com
Change-Id: Ic1ccc87851fb10540ac505789fd2d2a0be3ecb24
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: chromium:698902
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1559246
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#648943}
A test runner may have several pieces of metadata. Instead of putting all
of them in the top level, they can be put in the metadata field. I
also removed the test name prefix, tags and expectations files fields
from the top level. These fields will be put in the metadata field.
Bug: chromium:698902
Change-Id: I97973e4aed4cb2fe3761e047e48ce455422d300e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1552490
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Cr-Commit-Position: refs/heads/master@{#648900}
This is a split of a previous review: https://chromium-review.googlesource.com/c/chromium/src/+/1500784
There we discussed both the non-headless content_shell, as well as fuzzy pixel
diffs. However that review also had testing infrastructure changes which we are
blocked on landing.
This change pulls out the portion required for running SkiaRenderer and Vulkan
blink_web_tests. So that the team can run them locally.
content_shell defaults to a headless state when running the blink_web_tests.
Which forces Chrome to perform software rendering. There is currently no Swift
Shader equivalent for Vulkan. So we cannot use this for testing.
This change introduces a new Chrome switch "--disable-headless-mode" which
prevents the content_shell from triggering the headless state.
Actual GPU output introduces a variance from the baseline expectation files. So
this change also updates image_diff to have a new flag "--fuzzy-diff" which
allows it to have a tolerance in pixel differences. A corresponding
"--fuzzy-diff" flag has been added for blink_web_tests to be able to
trigger this.
I've also updated documentation of blink_web_tests based on findings
while writting this.
A follow up change will land tests on the Chromium GPU.FYI bots.
Bug: 925161
Change-Id: Ifd3837ec48a32bbdc776433e8c543138dde7418d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1553280
Commit-Queue: Jonathan Ross <jonross@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#648767}
These new fields can be useful for debugging regressions due to
invalid test expectations. It also updates the JSON results documentation
to include the new fields as well as the optional test_name_prefix field
which was added in a previous CL.
Bug: chromium:698902
Change-Id: Icc0edd59a2a3b585fb793a70355ee585cac28fad
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1546257
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Cr-Commit-Position: refs/heads/master@{#646000}
- The protocol mode
- Running HTTP web tests in Content Shell
- Running WPT tests in Content Shell
- Debugging with --single-process
- Debugging web tests
- Some corrections
Change-Id: I1d23d0f8a75b9d597edab153e4ec0ea9cdb7cb67
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1541863
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#645002}
Before crrev.com/c/1277654 when we created text results for ref tests
most of which don't need text baselines, we let reftest's text baseline
be optional, so that most reftests can ignore the text results, while
a few of them for which the text result is significant can still check
the text results.
After crrev.com/1277654, reftests no longer produce text results thus
don't need text baseline, unless a test explicitly produces text result.
In this case, we can just let the test determine if the text baseline
is needed, instead of run_web_tests.py checking existance of the
text baselines.
About existing reftests producing text results but without text
baselines:
1. for tests that don't actually need text results, modify them so that
they no longer produce text results. This includes some tests
previously calling testRunner.dumpAsTextWithPixelResults() just
to dump pixel results.
2. for tests that produce meaningful text results, create text
baselines.
3. for tests that don't produce text results intentionally, but the
text results are created because of some console messages, etc.
also create text baselines. There are several cases:
a) the message shows the purpose of the test, e.g. intentionally
testing with some invalid values. New text baselines are created
in this CL.
b) the message shows some problem of the test that doesn't need to
fix for now, e.g. the test is using some deprecating API. New
baselines are created in this CL.
c) the message indicates some fatal bug in the test, e.g. using
undefined variable, invalid JavaScript grammar, etc. The bugs are
fixed in this CL.
Bug: 703899
Change-Id: I82a6b25c61d1fb794840fdaa0da248909a690592
Reviewed-on: https://chromium-review.googlesource.com/c/1298120
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#604116}
SHERIFFS: PLEASE DO NOT REVERT THIS CL BECAUSE OF A SMALL AMOUNT OF LAYOUT
TEST FLAKINESS. If a few layout tests begin to show flakiness after this CL
lands, please add them to TestExpectations and email me (masonfreed@) to
investigate. Though I tried to identify all potentially flaky tests, a few
may have slipped through.
With this CL, the --enable-display-compositor-pixel-dump flag becomes the default
for content_shell. With this flag in place, layout test pixel dumps are performed
from the browser side, instead of from the renderer side. Note that to avoid a
significant amount of layout test flakiness, another change was also made to
not add the --run-all-compositor-stages-before-draw flag by default. There is
a bug (crbug.com/894613) tracking that problem separately.
With the flip of this switch, several modifications had to be made to the
TestExpectations file. First, there are a number of tests that change their
appearance slightly when being captured from the browser, and these tests need
to be rebaselined. These are summarized below, and will be rebaselined as a
separate CL, once this one lands and has had time to stabilize.
These bugs track the items added to TestExpectations:
- crbug.com/887140: HDR support
- crbug.com/881040: Media controls now contain an overflow menu.
- crbug.com/667551: A bunch of tests are listed under this bug, and just
require rebaselining to fix non-material single-pixel
antialiasing failures.
- crbug.com/891427: These either start failing, or become flaky, when the
--enable-display-compositor-pixel-dump flag is enabled.
They need to be debugged prior to re-enabling.
- crbug.com/895556: These tests double their background size when the flag
is enabled. They need to be fixed or rebaselined.
Bug: 667551, 891427, 881040, 887140, 894613, 895556
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel
Change-Id: I46946e6377f659c9dedc0dfaa20e7658e8cc519d
Reviewed-on: https://chromium-review.googlesource.com/c/1213864
Commit-Queue: Mason Freed <masonfreed@chromium.org>
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Reviewed-by: Peter Beverloo <peter@chromium.org>
Reviewed-by: Stephen Chenney <schenney@chromium.org>
Cr-Commit-Position: refs/heads/master@{#603127}
Now for tests that don't call any dumpAsXXX methods, they by default
generate pixel results only, and run_web_tests.py just check their
pixel results against the corresponding pixel baselines only.
Provide testRunner.dumpAsLayout() and dumpAsLayoutWithPixelResults()
for tests that do need to dump the layout tree.
We still dump layout tree when running such tests in content_shell
--run-web-tests command line (instead of running from run_web_tests.py
in protocol mode).
Changes to layout tests and their expectations:
- Use testRunner.dumpAsLayoutWithPixelResults() or dumpAsLayout()
in some tests that still requires the layout dump:
- editing/ tests need layout dump to show the caret position.
- two printing/ tests need layout dump to show how the bug is
fixed.
In the future, we may just get rid of layout dumps if it's
feasible.
- Remove testRunner.dumpAsTextWithPixelResults() from some tests
which used the function just to suppress layout output. They didn't
actually want text output but we hadn't had a way to dump image only
until we had the ImageFirst feature.
- Previous platform-specific -expected.txt for tests under previous
ImageFirst directories are removed and the common baselines are
created because the baselines are no longer platform-specific without
the layout dump.
Extra -expected.txt baselines are not deleted in this CL. Will
follow-up in https://chromium-review.googlesource.com/c/chromium/src/+/1286894.
Bug: 703899
Cq-Include-Trybots: luci.chromium.try:linux-blink-gen-property-trees;luci.chromium.try:linux_layout_tests_slimming_paint_v2;luci.chromium.try:linux_layout_tests_layout_ng
Change-Id: Ifcf7fea90684e133b4222909ee508469d0c985e0
Reviewed-on: https://chromium-review.googlesource.com/c/1277654
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Reviewed-by: Mike West <mkwst@chromium.org>
Cr-Commit-Position: refs/heads/master@{#602161}
Previous to this CL, these autoscroll/animation tests passed only because there
is a default surface deadline (typically 4 frames) in place. Surface activation
does not happen automatically, because in layout test mode, rasterization is not
performed until the very end of the test, for image capture. For the tests in
question, that deadline was previously always being hit, causing the surface to
be activated, triggering the monitored scroll events and passing the test.
When the --enable-display-compositor-pixel-dump flag is enabled, however, the
default surface activation deadline is changed to be infinite (no deadline). As
a result, all such autoscroll/animation tests fail.
With this CL, those tests that looked at animation behavior have been modified
to call a new continuouslyRunAnimations() javascript method, which sets a flag
that causes full rasterization to be included in the lifecycle update that was
already taking place. This triggers surface activation, and runs the
animations, allowing these tests to pass even with an infinite deadline.
Bug: 883547, 874162
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_layout_tests_slimming_paint_v2;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: I0d9e1ae7d10fab186207cdfbc8e2f85f60d99f35
Reviewed-on: https://chromium-review.googlesource.com/c/1229354
Commit-Queue: Mason Freed <masonfreed@chromium.org>
Reviewed-by: Kent Tamura <tkent@chromium.org>
Reviewed-by: Philip Rogers <pdr@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Reviewed-by: Timothy Dresser <tdresser@chromium.org>
Cr-Commit-Position: refs/heads/master@{#595898}
The test runner is `run_web_tests.py` (and not `run_webkit_tests.py`).
This patch also makes smaller tweaks and improvements to the doc.
Change-Id: I559cf54e98135f1fe90986e1cf06d140ca911918
Reviewed-on: https://chromium-review.googlesource.com/1203991
Commit-Queue: Mathias Bynens <mathias@chromium.org>
Reviewed-by: Kent Tamura <tkent@chromium.org>
Reviewed-by: Victor Costan <pwnall@chromium.org>
Cr-Commit-Position: refs/heads/master@{#588805}
This CL adds the `is_regression` per-test field (though it won't
yet be set by anything), and adds a bunch of text clarifying the
differences between failures, unexpected results, and regressions.
It also does some general editorial cleanup, fixing typos and
re-sorting some of the enumerated values.
BUG=822078, 837047
Change-Id: If78f6ef66c01d7c2924bfd98589832fe9299968f
Reviewed-on: https://chromium-review.googlesource.com/1096486
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Chan Li <chanli@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#566615}
This doesn't include those in 'if' statements.
Bug: 703768
TBR: already agreed in blink-dev; too many files.
Cq-Include-Trybots: master.tryserver.chromium.linux:linux_layout_tests_layout_ng
Change-Id: I499b91695e177d2a498d96cf20537ee146e22d06
Reviewed-on: https://chromium-review.googlesource.com/1057947
Reviewed-by: Xianzhu Wang <wangxianzhu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#558467}
All code in third_party/WebKit/Tools/Scripts was moved to
third_party/blink/tools.
This CL also fixes references to old names of scripts in Tools/Scripts.
* third_party/WebKit/PRESUBMIT.py:
Remove dead code. This PRESUBMIT.py doesn't use
|audit_non_blink_usage|.
Bug: 829697
Change-Id: I6c482a795708fa207124cb2030a4fdaa8ceac343
Reviewed-on: https://chromium-review.googlesource.com/1038863
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Kent Tamura <tkent@chromium.org>
Cr-Commit-Position: refs/heads/master@{#555366}
which is the new name of run-webkit-tests.
This CL also replaces references to "run-webkit-tests" with
"run_web_tests.py".
* //blink/tools/run_layout_tests.py
Show a message, and run run_web_tests.py.
* //BUILD.gn
"webkit_layout_tests" doesn't need //third_party/WebKit/Tools/Scripts
any longer.
* //testing/buildbot/gn_isolate_map.pyl
Replace run-webkit-tests with run_web_tests.py.
* //third_party/blink/tools/run_web_tests.{bat,py}
Added.
* //third_party/WebKit/Tools/Scripts/run-webkit-tests
Show a message.
* Other files: Really trivial changes. In many files, this CL
replaces "run-webkit-tests" with "run_web_tests.py" in comments or
strings.
Note that we decided to rename LayoutTests to web_tests.
https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/KKNbuzj-3HY/H8FWgtKrBgAJ
Bug: 829697
Change-Id: I899457cb3bcdb306aa47843a4c767c5980d9edc0
Reviewed-on: https://chromium-review.googlesource.com/1025543
Commit-Queue: Kent Tamura <tkent@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553452}
It will supersedes //third_party/WebKit/Tools/Scripts/lint-test-expectations.
lint-test-expectaions isn't removed yet due to dependency from
build/scripts/slave/chromium/.
Bug: 829697
Change-Id: I213b413304240e2f8672a1d2209a2800331b1ade
Reviewed-on: https://chromium-review.googlesource.com/1025437
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Kent Tamura <tkent@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553357}
The suffix part of the new name is 'tool' instead of 'patch' because
this command doesn't handle patches so much.
Bug: 829697
Change-Id: I90f34b6260032a7bbaf562d9b58ffc1a55c3922f
Reviewed-on: https://chromium-review.googlesource.com/1018700
Commit-Queue: Kent Tamura <tkent@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#552382}
