Device Setup > Emulators is largely a pointer to
//docs/android_emulator.md, which has more setup instructions.
However, it also references the instructions for how to run,
but those don't appear until later in the document, and thus
is unclear when read in order. Add a link to the later
section, to make it clearer that it's discussed later in the
document.
Bug: none
Change-Id: I2b3fee4d973ab500bba942a016bcb52a299aab19
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1930140
Reviewed-by: danakj <danakj@chromium.org>
Commit-Queue: danakj <danakj@chromium.org>
Cr-Commit-Position: refs/heads/master@{#718231}
It specifies the name of the flag-specific expectation file
under web_tests/FlagExpectations and the baseline directory under
web_tests/flag-specific, in the following format:
{
"name": "short-name",
"args": ["--arg1", "--arg2"]
}
When at least --additional-driver-flag=--arg1 and
--additional-driver-flag=--arg2 are in run_web_tests.py command line,
or --flag-specific=short-name is in the command line,
we will find web_tests/FlagExpectations/short-name for the additional
expectation file and web_tests/flag-specific/short-name for the
additional baseline directory.
Bug: 1019501
Change-Id: Idf0621abc89efc18cd17f9a9612074aa9d7de298
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1894069
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Steve Kobes <skobes@chromium.org>
Cr-Commit-Position: refs/heads/master@{#712013}
Now group previous multiple virtual test suites with the same
prefix and args together.
For example, previous configuration
{
"prefix": "composite-after-paint",
"base": "compositing",
"args": ["--enable-blink-features=CompositeAfterPaint"]
}
{
"prefix": "composite-after-paint",
"base": "paint",
"args": ["--enable-blink-features=CompositeAfterPaint"]
}
now becomes
{
"prefix": "composite-after-paint",
"bases": ["compositing",
"paint"],
"args": ["--enable-blink-features=CompositeAfterPaint"]
}
This shortens VirtualTestSuites by nearly half, avoids accidental
inconsistent args (We should create different virtual test suite
for different args), and allow "pure physical" virtual test suite
which is an alternative of physical test suite.
A "pure physical" virtual test suite has empty "bases", and the
"virtual/<prefix>" test path contains real tests only.
Normal vitual test suites can still contain real tests.
Bug: 1014162
Change-Id: I8d24be0e62b44ec8987aef65cb6f9ff0423b2d2b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1873504
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#710210}
The Monorail component for code coverage was renamed from:
Tools>CodeCoverage to Infra>Test>CodeCoverage, and this CL updates the
references.
Change-Id: I886422c5b8f839570ade415354fe2c122ee35ff7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1726371
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Yuke Liao <liaoyuke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#682373}
- add support for resolving shared library deps on android
- add work around for missing local build id on android
- use devil.py to access the android device
- hook script up to swarming on android
Bug: 688737, 849904
Change-Id: I9ff2e3ae1fed279fef83a44b07c693c338b8ee13
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1583715
Commit-Queue: Jochen Eisinger <jochen@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#657138}
This field has been moved from the top level to the metadata field
in the JSON results. The field has also been removed from the JSON
results documentation.
Bug: None
Change-Id: I7da08bcaf71c48d52b49f9cc7595cf551ab94f3c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1565371
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Cr-Commit-Position: refs/heads/master@{#650159}
This CL had to be reverted because GPU bots were failing because the
results_merger.py script could not handle the test_name_prefix field
that was still getting populated in the JSON results. This CL will
add the metadata field along side the test_name_prefix field in the
Optional fields list in the results_merger.py script. It also removes
the checking of test_name_prefix in the gpu unit tests because the
test_name_prefix field will be removed from the top level of the
results in a future CL.
Bug: chromium:698902
Change-Id: I4867e60ca5a97d426d5fd1a4de08e26c14d326ae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1559492
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#649665}
This reverts commit 7c7d23256a.
Reason for revert: We reverted the removal of the test_name_prefix
top level field in crrev.com/c/1558856. So for the JSON results to be properly
merged, we need to re-add the test_name_prefix field to the
results_merger.py script.
Original change's description:
> Adds the metadata optional field to the standard JSON results
>
> A test runner may have several pieces of metadata. Instead of putting all
> of them in the top level, they can be put in the metadata field. I
> also removed the test name prefix, tags and expectations files fields
> from the top level. These fields will be put in the metadata field.
>
> Bug: chromium:698902
> Change-Id: I97973e4aed4cb2fe3761e047e48ce455422d300e
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1552490
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Reviewed-by: Stephen Martinis <martiniss@chromium.org>
> Commit-Queue: Rakib Hasan <rmhasan@google.com>
> Cr-Commit-Position: refs/heads/master@{#648900}
TBR=kbr@chromium.org,martiniss@chromium.org,rmhasan@google.com
Change-Id: Ic1ccc87851fb10540ac505789fd2d2a0be3ecb24
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: chromium:698902
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1559246
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#648943}
A test runner may have several pieces of metadata. Instead of putting all
of them in the top level, they can be put in the metadata field. I
also removed the test name prefix, tags and expectations files fields
from the top level. These fields will be put in the metadata field.
Bug: chromium:698902
Change-Id: I97973e4aed4cb2fe3761e047e48ce455422d300e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1552490
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Cr-Commit-Position: refs/heads/master@{#648900}
This is a split of a previous review: https://chromium-review.googlesource.com/c/chromium/src/+/1500784
There we discussed both the non-headless content_shell, as well as fuzzy pixel
diffs. However that review also had testing infrastructure changes which we are
blocked on landing.
This change pulls out the portion required for running SkiaRenderer and Vulkan
blink_web_tests. So that the team can run them locally.
content_shell defaults to a headless state when running the blink_web_tests.
Which forces Chrome to perform software rendering. There is currently no Swift
Shader equivalent for Vulkan. So we cannot use this for testing.
This change introduces a new Chrome switch "--disable-headless-mode" which
prevents the content_shell from triggering the headless state.
Actual GPU output introduces a variance from the baseline expectation files. So
this change also updates image_diff to have a new flag "--fuzzy-diff" which
allows it to have a tolerance in pixel differences. A corresponding
"--fuzzy-diff" flag has been added for blink_web_tests to be able to
trigger this.
I've also updated documentation of blink_web_tests based on findings
while writting this.
A follow up change will land tests on the Chromium GPU.FYI bots.
Bug: 925161
Change-Id: Ifd3837ec48a32bbdc776433e8c543138dde7418d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1553280
Commit-Queue: Jonathan Ross <jonross@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#648767}
These new fields can be useful for debugging regressions due to
invalid test expectations. It also updates the JSON results documentation
to include the new fields as well as the optional test_name_prefix field
which was added in a previous CL.
Bug: chromium:698902
Change-Id: Icc0edd59a2a3b585fb793a70355ee585cac28fad
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1546257
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Cr-Commit-Position: refs/heads/master@{#646000}
- The protocol mode
- Running HTTP web tests in Content Shell
- Running WPT tests in Content Shell
- Debugging with --single-process
- Debugging web tests
- Some corrections
Change-Id: I1d23d0f8a75b9d597edab153e4ec0ea9cdb7cb67
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1541863
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#645002}
Before crrev.com/c/1277654 when we created text results for ref tests
most of which don't need text baselines, we let reftest's text baseline
be optional, so that most reftests can ignore the text results, while
a few of them for which the text result is significant can still check
the text results.
After crrev.com/1277654, reftests no longer produce text results thus
don't need text baseline, unless a test explicitly produces text result.
In this case, we can just let the test determine if the text baseline
is needed, instead of run_web_tests.py checking existance of the
text baselines.
About existing reftests producing text results but without text
baselines:
1. for tests that don't actually need text results, modify them so that
they no longer produce text results. This includes some tests
previously calling testRunner.dumpAsTextWithPixelResults() just
to dump pixel results.
2. for tests that produce meaningful text results, create text
baselines.
3. for tests that don't produce text results intentionally, but the
text results are created because of some console messages, etc.
also create text baselines. There are several cases:
a) the message shows the purpose of the test, e.g. intentionally
testing with some invalid values. New text baselines are created
in this CL.
b) the message shows some problem of the test that doesn't need to
fix for now, e.g. the test is using some deprecating API. New
baselines are created in this CL.
c) the message indicates some fatal bug in the test, e.g. using
undefined variable, invalid JavaScript grammar, etc. The bugs are
fixed in this CL.
Bug: 703899
Change-Id: I82a6b25c61d1fb794840fdaa0da248909a690592
Reviewed-on: https://chromium-review.googlesource.com/c/1298120
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#604116}
SHERIFFS: PLEASE DO NOT REVERT THIS CL BECAUSE OF A SMALL AMOUNT OF LAYOUT
TEST FLAKINESS. If a few layout tests begin to show flakiness after this CL
lands, please add them to TestExpectations and email me (masonfreed@) to
investigate. Though I tried to identify all potentially flaky tests, a few
may have slipped through.
With this CL, the --enable-display-compositor-pixel-dump flag becomes the default
for content_shell. With this flag in place, layout test pixel dumps are performed
from the browser side, instead of from the renderer side. Note that to avoid a
significant amount of layout test flakiness, another change was also made to
not add the --run-all-compositor-stages-before-draw flag by default. There is
a bug (crbug.com/894613) tracking that problem separately.
With the flip of this switch, several modifications had to be made to the
TestExpectations file. First, there are a number of tests that change their
appearance slightly when being captured from the browser, and these tests need
to be rebaselined. These are summarized below, and will be rebaselined as a
separate CL, once this one lands and has had time to stabilize.
These bugs track the items added to TestExpectations:
- crbug.com/887140: HDR support
- crbug.com/881040: Media controls now contain an overflow menu.
- crbug.com/667551: A bunch of tests are listed under this bug, and just
require rebaselining to fix non-material single-pixel
antialiasing failures.
- crbug.com/891427: These either start failing, or become flaky, when the
--enable-display-compositor-pixel-dump flag is enabled.
They need to be debugged prior to re-enabling.
- crbug.com/895556: These tests double their background size when the flag
is enabled. They need to be fixed or rebaselined.
Bug: 667551, 891427, 881040, 887140, 894613, 895556
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel
Change-Id: I46946e6377f659c9dedc0dfaa20e7658e8cc519d
Reviewed-on: https://chromium-review.googlesource.com/c/1213864
Commit-Queue: Mason Freed <masonfreed@chromium.org>
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Reviewed-by: Peter Beverloo <peter@chromium.org>
Reviewed-by: Stephen Chenney <schenney@chromium.org>
Cr-Commit-Position: refs/heads/master@{#603127}
