This adds an optional --isolated=outdir=PATH flag to the
test executable API; this gives the test runners a stable path
to the output directory from a swarming task, so that we can key
other files off of it without needing to pass them all explicitly
via the command line.
This also adds the arg to the isolated_script wrapper, so that we
those types of tests can handle the flag (by ignoring it for now).
We may need to work through the other cases (like gtest tests)
individually, which will be done in subsequent CLs.
Bug: 816629
Change-Id: Ibff6a47544bf94af5d7693a8446dfb669351cc07
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2358604
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Erik Staab <estaab@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#799378}
This reverts commit b679821 (change #784739) and
re-lands commit 956c4dc (change #784653) (without the addition
of //docs/testing/test_executable_api.md, which
landed separately already). Other changes landed that made
the re-land possible.
Original change's description:
> Add a first pass of the desktop test wrapper scripts.
>
> As described in the linked bug and documented in
> https://docs.google.com/document/d/19qpDeR2z-8j-WQJC-5waSd134kz94UXgYU_e_N-VplU/edit?usp=sharing
> we want to change how tests are launched under swarming so that
> they all have a consistent API and usage.
>
> We need to do at least some of this work now in order to unblock
> work going on in Swarming to change how a task's command line and
> list of files are specified, but doing all of the work in the linked
> doc and bug will significantly restructure how we configure and launch
> tests, and should bring a lot more consistency to our continuous
> integration infrastructure.
>
> This CL is a first step: a proof of concept that converts
> url_unittests on desktop to the new API in a backwards-compatible
> way (i.e., this will work with the existing chromium recipes).
TBR=tikuta@chromium.org, liaoyuke@chromium.org
Bug: 816629
Change-Id: Ia4d20501783f844fdbc1728ee986724c5301d638
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2280322
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Cr-Commit-Position: refs/heads/master@{#788424}
Contrary to crrev.com/c/2217275 it is possible to specify the directory
which the script uses to serve generated content.
Point users towards --enable-blink-features which is required for some
tests.
Change-Id: I4b9cb454dce3fabdc62c778c810342047f18b539
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2241199
Auto-Submit: Johann Koenig <johannkoenig@google.com>
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#777844}
Updates:
- Guides added for adding experimental attribute to a story and when to
do so.
- How to run perf tests locally is added to the document
Bug: none
Change-Id: Ie2c3f34906d185cd3992e4a614c49f43e4c3ad4f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2210632
Reviewed-by: Sadrul Chowdhury <sadrul@chromium.org>
Commit-Queue: Behdad Bakhshinategh <behdadb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#771156}
This followups crrev.com/c/2173656 which let virtual tests inherit
base tests' expectations except for [ Skip ]. This CL also allows
inheritance of [ Skip ].
In less common cases, we intentionally skip some base tests which is
only meaningful to run with specific flags in virtual tests. In these
cases we need to override the [ Skip ] with [ Pass ], like:
(In NeverFixTests)
# This test run under virtual suites only.
crbug.com/981970 external/wpt/fetch/http-cache/split-cache.tentative.html [ Skip ]
virtual/not-split-http-cache/external/wpt/fetch/http-cache/split-cache.tentative.html [ Pass ]
virtual/split-http-cache/external/wpt/fetch/http-cache/split-cache.tentative.html [ Pass ]
Bug: 1072015
Change-Id: Ie6c5b5472203bd81120dd2c3589a89d9df6fa45b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2178173
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Rakib Hasan <rmhasan@google.com>
Reviewed-by: Robert Ma <robertma@chromium.org>
Reviewed-by: Philip Rogers <pdr@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#766598}
This can avoid duplicated entries for virtual tests. This allows
a virtual test without specific expectations inherit the base test
expectations. If a virtual test has specific expectations, they will
override the base test expectations, except that [ Slow ] will be
merged.
This requires flag-specific expectations to use the same override rule.
Previously they always inherit the base test expectations, causing us
unable to override the expectations to e.g. [ Pass ].
Requires https://chromium-review.googlesource.com/c/catapult/+/2173206.
Bug: 1072015
Change-Id: Ie3d1911a96f70799a996b6abbd78f9a5107d7eff
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2173656
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Rakib Hasan <rmhasan@google.com>
Reviewed-by: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#765359}
This reverts commit 0b8c0e6a09.
Reason for revert: Fixed runtime deps
Original change's description:
> Revert "Make java_deobfuscate not require being built first"
>
> This reverts commit 962e70305e.
>
> Reason for revert: Failing on swarming
> https://ci.chromium.org/p/chromium/builders/ci/android-pie-x86-fyi-rel/747
>
> Original change's description:
> > Make java_deobfuscate not require being built first
> >
> > Makes the tool easier to use, and easier to document how to use by not
> > requiring it to be built first. We have only a single non-prebuilt .java
> > file, so it's easy enough to just check in the .class file for it.
Bug: 995290
Change-Id: Ib1f396e07d6b0737133f03f1bded1a2d312ebfc8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2106972
Commit-Queue: Andrew Grieve <agrieve@chromium.org>
Reviewed-by: Sam Maier <smaier@chromium.org>
Cr-Commit-Position: refs/heads/master@{#751322}
Makes the tool easier to use, and easier to document how to use by not
requiring it to be built first. We have only a single non-prebuilt .java
file, so it's easy enough to just check in the .class file for it.
Bug: 995290
Change-Id: I7b934b3337eccb044a470dd92270ce7338b1fd20
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2106257
Commit-Queue: Andrew Grieve <agrieve@chromium.org>
Reviewed-by: Sam Maier <smaier@chromium.org>
Cr-Commit-Position: refs/heads/master@{#750855}
This involves
* Moving non-MD-related code out of Initialize into LinearAnimation
* Moving other Initialize() code into the constructor
* Removing all Initialize() calls
* Replacing other static methods with instance methods
* Adding GetInstance() calls to callers to get the instance
* Converting the TestAPI class to a scoping object
* Other assorted #include removal etc. in the process
Bug: 940773
Change-Id: Ieb8d44eabae63b014a90ff452010eb4c067e5e90
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2090877
Commit-Queue: Peter Kasting <pkasting@chromium.org>
Reviewed-by: Scott Violet <sky@chromium.org>
Cr-Commit-Position: refs/heads/master@{#748895}
The guides for investigating the representative perf tests are added:
- Common failures listed
- accessing the traces and benchmark logs are documented
Bug: none
Change-Id: Ic087fb97ee362e04d839414f1396165f111b9934
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2087220
Commit-Queue: Behdad Bakhshinategh <behdadb@chromium.org>
Reviewed-by: Sadrul Chowdhury <sadrul@chromium.org>
Cr-Commit-Position: refs/heads/master@{#747732}
Device Setup > Emulators is largely a pointer to
//docs/android_emulator.md, which has more setup instructions.
However, it also references the instructions for how to run,
but those don't appear until later in the document, and thus
is unclear when read in order. Add a link to the later
section, to make it clearer that it's discussed later in the
document.
Bug: none
Change-Id: I2b3fee4d973ab500bba942a016bcb52a299aab19
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1930140
Reviewed-by: danakj <danakj@chromium.org>
Commit-Queue: danakj <danakj@chromium.org>
Cr-Commit-Position: refs/heads/master@{#718231}
It specifies the name of the flag-specific expectation file
under web_tests/FlagExpectations and the baseline directory under
web_tests/flag-specific, in the following format:
{
"name": "short-name",
"args": ["--arg1", "--arg2"]
}
When at least --additional-driver-flag=--arg1 and
--additional-driver-flag=--arg2 are in run_web_tests.py command line,
or --flag-specific=short-name is in the command line,
we will find web_tests/FlagExpectations/short-name for the additional
expectation file and web_tests/flag-specific/short-name for the
additional baseline directory.
Bug: 1019501
Change-Id: Idf0621abc89efc18cd17f9a9612074aa9d7de298
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1894069
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Steve Kobes <skobes@chromium.org>
Cr-Commit-Position: refs/heads/master@{#712013}
Now group previous multiple virtual test suites with the same
prefix and args together.
For example, previous configuration
{
"prefix": "composite-after-paint",
"base": "compositing",
"args": ["--enable-blink-features=CompositeAfterPaint"]
}
{
"prefix": "composite-after-paint",
"base": "paint",
"args": ["--enable-blink-features=CompositeAfterPaint"]
}
now becomes
{
"prefix": "composite-after-paint",
"bases": ["compositing",
"paint"],
"args": ["--enable-blink-features=CompositeAfterPaint"]
}
This shortens VirtualTestSuites by nearly half, avoids accidental
inconsistent args (We should create different virtual test suite
for different args), and allow "pure physical" virtual test suite
which is an alternative of physical test suite.
A "pure physical" virtual test suite has empty "bases", and the
"virtual/<prefix>" test path contains real tests only.
Normal vitual test suites can still contain real tests.
Bug: 1014162
Change-Id: I8d24be0e62b44ec8987aef65cb6f9ff0423b2d2b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1873504
Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#710210}
