046f987e02
See this doc: go/chromium-coil-change for more info
BUG=1210385
Change-Id: I6cd8548301e360bae8e71384dbd8ef11aa6c985f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2902904
Reviewed-by: My Nguyen <myy@chromium.org>
Commit-Queue: My Nguyen <myy@chromium.org>
Auto-Submit: John Palmer <jopalmer@chromium.org>
Cr-Commit-Position: refs/heads/master@{#885851}
90 lines
4.2 KiB
Markdown
90 lines
4.2 KiB
Markdown
# Parsing Test Results
|
|
|
|
Chromium runs over 500,000 tests for each CL. There are many layers of UI for
|
|
parsing and interpreting these test results. This doc provides a brief guide
|
|
for navigating these UI layers.
|
|
|
|
## Polygerrit UI
|
|
|
|
Tests are segmented by build and test configurations. The segments are usually
|
|
referred to as *builds*. In the example below, each green and red rectangle
|
|
refers to a *build*.
|
|
|
|
|
|
|
|
The name of each build usually contains enough information to get a rough idea
|
|
of the configuration. Some examples:
|
|
|
|
* *android_compile_dbg* is a compile-only [no tests] build of Chromium for
|
|
Android, using the *debug* configuration.
|
|
* *android-lollipop-arm-rel* builds and runs tests for Chromium for Android,
|
|
using the *release* configuration on a lollipop device.
|
|
* *win7-rel* builds and runs tests for Chromium on Windows, using
|
|
the release configuration on a Windows 7 device. *ng* stands for next
|
|
generation, but this has no meaning as the previous generation was already
|
|
phased out.
|
|
|
|
Green boxes refer to builds that passed. Red boxes refer to builds that failed.
|
|
Some failed builds get automatically retried by the CQ. In this example,
|
|
*linux-rel* and *mac-rel* were automatically retried [hence the two **X**s], but
|
|
*win7-rel* was not. The **X** on the left is the first build, and the **X** on the right
|
|
is the second build.
|
|
|
|
Each of these boxes is a link that provides more information about the build
|
|
failure.
|
|
|
|
## Build Results UI
|
|
|
|
Selecting any of the build results from the previous section will navigate to
|
|
the build results UI. Each build is implemented by a [recipe] --
|
|
effectively a Python script. Each recipe is divided into *steps*. Each *step*
|
|
represents a well-defined action, such as updating the repository to point to
|
|
tip of tree, or compiling the necessary build artifacts.
|
|
|
|
[recipe]: https://chromium.googlesource.com/external/github.com/luci/recipes-py/+/main/doc/user_guide.md
|
|
|
|
|
|
|
|
Under the **Steps and Logfiles** heading is a list of numbered *steps*. Each
|
|
*step* has a color (red, green or purple) which indicates whether the *step*
|
|
failed, succeeded, or encountered an unexpected condition. Failing steps are
|
|
also grouped into the **Results** section at the very top for convenience.
|
|
|
|
## Build Results UI -- Overview
|
|
|
|
Most builds follow a similar pattern. The key *steps* are listed here.
|
|
|
|
* **bot_update** Update the repository to point to tip of tree. Apply the CL
|
|
as a patch.
|
|
* **analyze** Analyze dependencies of test suites to determine which test
|
|
suites are affected by the patch.
|
|
* **compile (with patch)** Builds test suites and associated artifacts.
|
|
* **isolate tests** Archives test suite binaries and artifacts.
|
|
* **test_pre_run.[trigger] blink_web_tests (with patch)** Triggers a test
|
|
suite on swarming [remote execution framework] -- in this case,
|
|
blink_web_tests.
|
|
* **blink_web_tests (with patch)** Collects the results from swarming for a
|
|
test suite.
|
|
|
|
If all test suites pass, then the *build* is marked as a success and no further
|
|
steps are run. If at least one test suite has failures, then the failing tests
|
|
are rerun with the patch deapplied. This allows the recipe to determine if the
|
|
test failure is due to the CL or due to a problem with tip of tree.
|
|
|
|
* **bot_update [without patch]** Deapplies the CL patch.
|
|
* **compile [without patch]** Compiles test suites.
|
|
* **isolate tests (2)** Archives test suite binaries and artifacts.
|
|
* **test_pre_run.[trigger] blink_web_tests (without patch)** Triggers test
|
|
suite on swarming. Only failing tests are rerun.
|
|
* **blink_web_tests (without patch)** Collects results from swarming.
|
|
|
|
**Important safety notice**. When test suites are run with the patch applied,
|
|
each test is run up to N times -- any success will mark the test as a success.
|
|
When test suites are run without the patch, each failing test is run exactly N
|
|
times. Any failure will mark the test as a failure.
|
|
|
|
If there are tests that failed with the patch applied, but not with the patch
|
|
deapplied, then that implies that it's likely that the CL broke a test. Just to
|
|
confirm, the first suite of steps is run again, this time with the suffix
|
|
**(retry with patch)**.
|