[dPWA Testing] Moved docs and added test addition how-to.
This change updates a lot of documentation, adds a how-to guide for
adding new integration tests, and starts the documentation move to
docs/webapps.
Bug: 1264155
Change-Id: I44fbd3e79f4153c91b977170671d37d503d868fe
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3256208
Commit-Queue: Daniel Murphy <dmurph@chromium.org>
Reviewed-by: Evan Stade <estade@chromium.org>
Reviewed-by: Phillis Tang <phillis@chromium.org>
Cr-Commit-Position: refs/heads/main@{#937979}
This commit is contained in:
Daniel Murphy
committed by
Chromium LUCI CQ
Chromium LUCI CQ
parent
30f053594e
commit
27a6bd024d
chrome
docs/webapps
@ -1,129 +1,3 @@
|
||||
# dPWA Integration Tests
|
||||
|
||||
The dPWA integration tests use a special framework. Each test is defined by a
|
||||
series of "testing actions", and test cases programmatically generated by a
|
||||
python script. The script will only yield tests for which every "action" is
|
||||
currently supported by the testing framework.
|
||||
|
||||
## Background
|
||||
|
||||
See this [design doc](https://docs.google.com/document/d/e/2PACX-1vTFI0sXhZMvvg1B3sctYVUe64WbLVNzuXFUa6f3XyYTzKs2JnuFR8qKNyXYZsxE-rPPvsq__4ZCyrcS/pub) for background information, and the testing script [README](../../../../test/webapps/README.md) for how integration tests are generated.
|
||||
|
||||
## Future work
|
||||
|
||||
* Fix how sync tests await web app quiescense. Currently, there is no waiting,
|
||||
and tests will fail.
|
||||
|
||||
## Disabling a Test
|
||||
|
||||
Tests can be disabled in the same manner that other integration/browser tests
|
||||
are disabled, using macros. See [On disabling
|
||||
tests](https://chromium.googlesource.com/chromium/src/+/main/docs/testing/on_disabling_tests.md)
|
||||
for more information.
|
||||
|
||||
## How to Contribute
|
||||
|
||||
### Adding a Testing Action
|
||||
|
||||
Adding support for a new testing action in the framework is as simple as adding
|
||||
a new method in `WebAppIntegrationBrowserTestBase`. In some cases, you may want
|
||||
to implement actions related to profile sync, and need to call into SyncTest
|
||||
helper methods. In this case, the meat of the action can be implemented in
|
||||
`TwoClientWebAppsIntegrationSyncTest`. From there, you implement a method of the
|
||||
same name in the base class, and have it call your sync action on the
|
||||
`TestDelegate` member (ie `delegate_->SwitchProfileClients()`).
|
||||
|
||||
The list of testing actions are maintained in
|
||||
`chrome/test/webapps/data/framework_supported_actions.csv`. To add a new testing
|
||||
action, add the action into that csv. Use the emojis to indicate which platforms
|
||||
the action is supported on (at the time of your CL landing, rather than in the
|
||||
future). Use the following emojis to specify coverage for an action on a given
|
||||
platform:
|
||||
|
||||
* 🌕 - Full coverage - This means that the testing framework implements this
|
||||
action in a way that completely matches (or almost matches) the code paths
|
||||
that are used when the user triggers this action.
|
||||
* 🌓 - Partial coverage - This means that the testing framework implements this
|
||||
action in a way that accomplishes the state change or check, but does not
|
||||
fully match the code path that is used when the user triggers this action.
|
||||
* 🌑 - No coverage - This means the action is not supported.
|
||||
|
||||
|
||||
### Managing State
|
||||
|
||||
After every state-change action, a state snapshot is constructed and stored as a
|
||||
member of WebAppIntegrationBrowserTestBase, `after_state_change_action_state_`.
|
||||
This is done in `ConstructStateSnapshot()`, called from
|
||||
`AfterStateChangeAction()`. At the start of every state-change action, this
|
||||
state snapshot gets moved into the `before_state_change_action_state_` member,
|
||||
giving us the ability to compare the pre- and post-conditions of every
|
||||
state-change action.
|
||||
|
||||
When adding actions, it may be useful to bolt onto this state snapshot in order
|
||||
to verify the results of state-change actions within state-check actions. To do
|
||||
this, simply add onto the relevant state objects, and update the objects `==`
|
||||
operator. Then, you need to make sure that your new state fields are updated in
|
||||
`ConstructStateSnapshot()`.
|
||||
|
||||
## How It Works
|
||||
|
||||
This testing framework uses a [script](../../../../test/webapps/README.md) to
|
||||
generate a minimal set of test cases that produce the maximum amount of code
|
||||
coverage. The script will output in stdout tests cases to add and remove, and
|
||||
the developer then performs the instructed changes to make the testing
|
||||
implementation up-to-date.
|
||||
|
||||
|
||||
This suite of tests has two main parts:
|
||||
1. A script that analyzes dPWA code ([See design
|
||||
doc](https://docs.google.com/document/d/1YmeNZCpIwUbeV3K3HGUdXzJjZDKIDyKrGfyjnYaLR5k).
|
||||
2. The test implementations ([See design
|
||||
doc](https://docs.google.com/document/d/1Gd14fjwA4VKoRzL2TAvi9paXwyh36ehlS4gbpUmUeeI).
|
||||
|
||||
### Script
|
||||
See the [README](../../../../test/webapps/README.md) and [design doc](https://docs.google.com/document/d/e/2PACX-1vTFI0sXhZMvvg1B3sctYVUe64WbLVNzuXFUa6f3XyYTzKs2JnuFR8qKNyXYZsxE-rPPvsq__4ZCyrcS/pub).
|
||||
|
||||
### Test Structure
|
||||
The high level flow of execution is as follows:
|
||||
* Each tests lives as a regular browsertest with a specific name so the script
|
||||
can determine if it exists.
|
||||
* The script outputs a test where actions are method calls, and before & after
|
||||
each test there is Pre and Post action method to call allow the framework to
|
||||
record state, clean up, or wait as necessary.
|
||||
|
||||
## Components
|
||||
[Design
|
||||
doc](https://docs.google.com/document/d/139ktCajbmbFKh4T-vEhipTxilyYrXf_rlCBHIvrdeSg).
|
||||
|
||||
### WebAppIntegrationBrowserTestBase
|
||||
A helper class containing most of the test implementation, meant to be used
|
||||
as a private member on the test-driving classes. Contains most of the test
|
||||
framework implementation:
|
||||
* Most action implementation methods.
|
||||
* Capturing state snapshots.
|
||||
|
||||
### WebAppIntegrationBrowserTestBase::TestDelegate
|
||||
An abstract class that’s an application of the delegate interface pattern.
|
||||
`WebAppIntegrationBrowserTestBase` stores an instance of this class as a
|
||||
private member, `delegate_`, allowing the base class to call into protected
|
||||
members of `InProcessBrowserTest` or `SyncTest`, such as
|
||||
`InProcessBrowserTest::browser()` or `SyncTest::GetAllProfiles()`. This also
|
||||
has pure virtual methods for sync functionality that needs to be implemented
|
||||
in `TwoClientWebAppsSyncTest`, but called from the base class.
|
||||
|
||||
### WebAppIntegrationBrowserTest
|
||||
Class that drives the tests. Subclass of both `InProcessBrowserTest` and
|
||||
`WebAppIntegrationBrowserTestBase::TestDelegate`. Responsible
|
||||
for telling the base class where the test input files live, handling test setup,
|
||||
and implementing `TestDelegate` methods to expose protected members of
|
||||
`InProcessBrowserTest` to the base class. This class owns the base class, and
|
||||
stores it as a private member, `helper_`, passing it an instance of itself (as
|
||||
the TestDelegate) on construction.
|
||||
|
||||
|
||||
### TwoClientWebAppsSyncTest
|
||||
Similar to `WebAppIntegrationBrowserTest`, but inheriting from `SyncTest`
|
||||
instead of from `InProcessBrowserTest`. In addition, some testing actions
|
||||
related to profile sync are implemented in this class, and are exposed via
|
||||
`TestDelegate` pure virtual method overrides.
|
||||
# Desktop Web App Integration Tests
|
||||
|
||||
Documentation has moved [here](/docs/webapps/integration-testing-framework.md).
|
||||
|
||||
@ -1,193 +1,3 @@
|
||||
# Desktop Web App Integration Testing Framework
|
||||
|
||||
## Background
|
||||
|
||||
The WebAppProvider system has very wide action space and testing state interactions between all of the subsystems is very difficult. The goal of this piece is to accept:
|
||||
* A list of action-based tests which fully test the WebAppProvider system (required-coverage tests), and
|
||||
* A list of actions supported by the integration test framework (per-platform),
|
||||
and output
|
||||
* a minimal number of tests (per-platform) for the framework to execute that has the most coverage possible of the original list of tests, and
|
||||
* the resulting coverage of the system (with required-coverage tests as 100%).
|
||||
|
||||
This is done by downloading [data](data/) from a [google sheet](https://docs.google.com/spreadsheets/d/e/2PACX-1vSbO6VsnWsq_9MN6JEXlL8asMqATHc2-pz9ed_Jlf5zHJGg2KAtegsorHqkQ5kydU6VCqebv_1gUCD5/pubhtml), processing it, and saving the results in the [output/](output/) folder.
|
||||
|
||||
See the [design doc](https://docs.google.com/document/d/e/2PACX-1vTFI0sXhZMvvg1B3sctYVUe64WbLVNzuXFUa6f3XyYTzKs2JnuFR8qKNyXYZsxE-rPPvsq__4ZCyrcS/pub) for more information and links.
|
||||
|
||||
This README covers how the spreadsheet & tests are read to compute coverage information and tell the script runner how to modify the tests to get the respective coverage. For information about how to implement actions in the framework (or how the framework implementation works), see the [testing framework implementation README.md](../../browser/ui/views/web_apps/README.md).
|
||||
|
||||
Related:
|
||||
* [WebAppProvider README.md](../../browser/web_applications/README.md)
|
||||
|
||||
## Future Work
|
||||
|
||||
* Integration manual test support.
|
||||
|
||||
## Terminology
|
||||
|
||||
### Action
|
||||
A primitive test operation, or test building block, that can be used to create coverage tests. The integration test framework may or may not support an action on a given platform. Actions can fall into one of three types:
|
||||
* State-change action
|
||||
* State-check action
|
||||
* parameterized action
|
||||
|
||||
Actions also can be fully, partially, or not supported on a given platform. This information is used when generating the tests to run & coverage report for a given platform. To make parsing easier, actions are always snake_case.
|
||||
|
||||
#### State-change Action
|
||||
A state-changing action is expected to change the state chrome or the web app provider system.
|
||||
|
||||
Examples: `navigate_browser(SiteA)`, `switch_incognito_profile`, `sync_turn_off`, `set_app_badge`
|
||||
|
||||
#### State-check Action
|
||||
Some actions are classified as "state check" actions, which means they do not change any state and only inspect the state of the system. In graph representations, state check actions are not given nodes, and instead live under the last non-state-check action.
|
||||
|
||||
All actions that start with `check_` are considered state-check actions.
|
||||
|
||||
Examples: `check_app_list_empty`, `check_install_icon_shown`, `check_platform_shortcut_exists(SiteA)`, `check_tab_created`
|
||||
|
||||
#### Action Mode
|
||||
When creating tests, there emerged a common scenario where a given action could be applied to multiple different sites. For example, the “navigate the browser to an installable site” action was useful if “site” could be customized.
|
||||
|
||||
The simplest possible mode system to solve this:
|
||||
* Each action can have at most one mode.
|
||||
* Modes are static / pre-specified per action.
|
||||
* A default mode can be specified to support the case where an action has modes but none were specified in the test.
|
||||
|
||||
To allow for future de-parsing of modes (when generating C++ tests), modes will always be CapitalCase.
|
||||
|
||||
#### Parameterized Action
|
||||
To help with testing scenarios like outlined above, an action can be defined that references or 'turns into' a set of non-parameterized actions. For example, an action `install_windowed` can be created and reference the set of actions `install_omnibox_icon`, `install_menu_option`, `install_create_shortcut_windowed`, `add_policy_app_windowed_shortcuts`, and `add_policy_app_windowed_no_shortcuts`. When a test case includes this action, it will generate multiple tests in which the parameterized action is replaced with the non-parameterized action.
|
||||
|
||||
### Tests
|
||||
A sequence of actions used to test the WebAppProvider system. A test that can be run by the test framework must not have any "parameterized" actions, as these are supposed to be used to generate multiple tests.
|
||||
|
||||
#### Unprocessed Required-coverage tests
|
||||
This is the set of tests that, if all executed, should provide full test coverage for the WebAppProvider system. They currently live in this sheet as "unprocessed".
|
||||
|
||||
#### Required-coverage tests (processed)
|
||||
Processed tests go through the following steps from the unprocessed version in the sheet:
|
||||
* Tests with one or more "parameterized" actions have been processed to produce the resulting tests without parameterized actions.
|
||||
* Actions in tests that have modes but do not specify them have the default mode added to them.
|
||||
|
||||
#### Platform-specific tests
|
||||
Some tests are going to be platform-specific. For example, all tests that involve "locally installing" an app are only applicable on Windows/Mac/Linux, as ChromeOS automatically locally installs all apps from sync. Because of this, tests must be able to specify which platforms they should be run on. This is done by specifying the platforms each test applies to in a column on the spreadsheet.
|
||||
|
||||
## Understanding and Implementing Test Cases
|
||||
|
||||
Actions are the basic building blocks of integration tests. A test is a sequence of actions. Each action has a name that must be a valid C++ identifier.
|
||||
|
||||
Actions are defined (and can be modified) in [this](https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vSbO6VsnWsq_9MN6JEXlL8asMqATHc2-pz9ed_Jlf5zHJGg2KAtegsorHqkQ5kydU6VCqebv_1gUCD5/pubhtml?gid=1864725389) sheet. Tests are defined (and can be modified) in [this](https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vSbO6VsnWsq_9MN6JEXlL8asMqATHc2-pz9ed_Jlf5zHJGg2KAtegsorHqkQ5kydU6VCqebv_1gUCD5/pubhtml?gid=2008870403) sheet.
|
||||
|
||||
### Action Creation & Specification
|
||||
|
||||
[Actions](https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vSbO6VsnWsq_9MN6JEXlL8asMqATHc2-pz9ed_Jlf5zHJGg2KAtegsorHqkQ5kydU6VCqebv_1gUCD5/pubhtml?gid=1864725389) are the building blocks of tests.
|
||||
|
||||
#### Templates
|
||||
To help making test writing less repetitive, actions are described as templates in the [actions](https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vSbO6VsnWsq_9MN6JEXlL8asMqATHc2-pz9ed_Jlf5zHJGg2KAtegsorHqkQ5kydU6VCqebv_1gUCD5/pubhtml?gid=1864725389) spreadsheet. Action templates specify actions while avoiding rote repetition. Each action template has a name (the **action base name**). Each action template supports a mode, which takes values from a predefined list associated with the template. Parameter values must also be valid C++ identifiers.
|
||||
|
||||
An action template without modes specifies one action whose name matches the template. For example, the `check_tab_created` template generates the `check_tab_created` action.
|
||||
|
||||
An action template with a mode that can take N values specifies N actions, whose names are the concatenations of the template name and the corresponding value name, separated by an underscore (_). For example, the `clear_app_badge` template generates the `clear_app_badge_SiteA` and `clear_app_badge_SiteB` actions.
|
||||
|
||||
The templates also support [parameterizing](#parameterized-action) an action, which causes any test that uses the action to be expanded into multiple tests, one per specified output action. Modes will carry over into the output action, and if an output action doesn't support a given mode then that parameterization is simply excluded during test generation.
|
||||
|
||||
#### Default Values
|
||||
|
||||
All templates with modes can mark one of their mode values as the default value. This value is used to magically convert template names to action names.
|
||||
|
||||
#### Specifying a Mode
|
||||
Human-friendly action names are a slight variation of the canonical names above.
|
||||
|
||||
Actions generated by mode-less templates have the same human-friendly name as their (canonical?) name.
|
||||
|
||||
Actions generated by parametrized templates use parenthesis to separate the template name from the value name. For example, the actions generated by the `clear_app_badge` template have the human-friendly names `clear_app_badge(SiteA)` and `clear_app_badge(SiteB)`.
|
||||
|
||||
The template name can be used as the human-friendly name of the action generated by the template with the default value. For example, `clear_app_badge` is a shorter human-friendly name equivalent to `clear_app_badge(SiteA)`.
|
||||
|
||||
### Test Creation & Specification
|
||||
|
||||
[Tests](https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vSbO6VsnWsq_9MN6JEXlL8asMqATHc2-pz9ed_Jlf5zHJGg2KAtegsorHqkQ5kydU6VCqebv_1gUCD5/pubhtml?gid=2008870403) are created specifying actions.
|
||||
|
||||
#### Mindset
|
||||
The mindset for test creation and organization is to really exhaustively check every possible string of user actions. The framework will automatically combine tests that are the same except for state check actions. They are currently organized by:
|
||||
1. **Setup actions** - The state change actions needed to enter the system state that is being tested.
|
||||
2. **Primary state-change action/s** - Generally one action that will change the system state after the setup.
|
||||
3. **State check action** - One state check action, checking the state of the system after the previous actions have executed.
|
||||
|
||||
Each test can have at most one [state check](#state-check-action) action as the last action.
|
||||
|
||||
One way to enumerate tests is to think about **affected-by** action edges. These are a pair of two actions, where the first action affects the second. For example, the action `set_app_badge` will affect the action `check_app_badge_has_value`. Or, `uninstall_from_app_list` will affect `check_platform_shortcut_exists`. There is often then different setup states that would effect these actions. Once these edges are identified, tests can be created around them.
|
||||
|
||||
#### Creating a test
|
||||
|
||||
A test should be created that does the bare minimum necessary to set up the test before testing the primary state change action and then checking the state.
|
||||
|
||||
The framework is designed to be able to collapse tests that contain common non-'state-check' actions, so adding a new test does not always mean that a whole new test will be run by the framework. Sometimes it only adds a few extra state-check actions in an existing test.
|
||||
|
||||
#### Adding an action
|
||||
|
||||
If a new test needs a new action implemented, it will only be used in the generated tests if it is added to the [actions](https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vSbO6VsnWsq_9MN6JEXlL8asMqATHc2-pz9ed_Jlf5zHJGg2KAtegsorHqkQ5kydU6VCqebv_1gUCD5/pubhtml?gid=1864725389) sheet, and it is specifically marked as supported or partially supported. Then the script will print out the relevant browsertest to add to the relevant file.
|
||||
|
||||
The action also needs to be implemented by the testing framework. See the [Testing Framework Implementation README.md](../../browser/ui/views/web_apps/README.md) for more info about how to do that.
|
||||
|
||||
#### Adding 'support' for an action
|
||||
|
||||
To tell the script that an action is supported by the testing framework (on a given platform), modify the [`framework_supported_actions.csv`](./data/framework_supported_actions.csv) file, and use the following emojis to specify coverage for an action on a given platform:
|
||||
|
||||
* 🌕 - Full coverage
|
||||
* 🌓 - Partial coverage
|
||||
* 🌑 - No coverage
|
||||
|
||||
The script reads this file to determine what tests to generate.
|
||||
|
||||
## Script Usage
|
||||
|
||||
### Downloading test data
|
||||
|
||||
The test data is hosted in this [spreadsheet](https://docs.google.com/spreadsheets/d/1d3iAOAnojp4_WrPky9exz1-mjkeulOJVUav5QYG99MQ/edit). To download the latest copy of the data, run the included script:
|
||||
```sh
|
||||
./chrome/test/webapps/download_data_from_sheet.py
|
||||
```
|
||||
|
||||
This will download the data from the sheet into csv files in the [data/](data/) directory:
|
||||
|
||||
* `actions.csv` This describes all actions that can be used in the required coverage tests (processed or unprocessed).
|
||||
* `coverage_required.csv` This is the full list of all tests needed to fully cover the Web App system. The first column specifies the platforms for testing, and the test starts on the fifth column.
|
||||
|
||||
### Generating test descriptions & coverage
|
||||
|
||||
Required test changes are printed and coverage files are written by running:
|
||||
```sh
|
||||
chrome/test/webapps/generate_framework_tests_and_coverage.py
|
||||
```
|
||||
This uses the files in `chrome/test/webapps/data` and existing browsertests on the system (see `custom_partitions` and `default_partitions` in [generate_framework_tests_and_coverage.py](generate_framework_tests_and_coverage.py)) to:
|
||||
|
||||
#### 1) Print to `stdout` all detected changes needed to browsertests.
|
||||
The script is not smart enough to automatically add/remove/move tests to keep complexity to a minimum. Instead, it prints out the tests that need to be added or removed to have the tests match what it expects. It assumes:
|
||||
* Browsertests are correctly described by the `TestPartitionDescription`s in `generate_framework_tests_and_coverage.py`.
|
||||
* Browsertests with the per-platform suffixes (e.g. `_mac`, `_win`, etc) are only run on those platforms
|
||||
|
||||
This process doesn't modify the browsertest files so any test disabling done by sheriffs can remain. The script runner is thus expected to make the requested changes manually. In the rare case that a test is moving between files (if we are enabling a test on a new platform, for example), then the script runner should be careful to copy any sheriff changes to the browsertest as well.
|
||||
|
||||
#### 2) Generate per-platform processed required coverage `tsv` files in `chrome/test/webapps/coverage`
|
||||
These are the processed required coverage tests with markers per action to allow a conditional formatter (like the one [here](https://docs.google.com/spreadsheets/d/1d3iAOAnojp4_WrPky9exz1-mjkeulOJVUav5QYG99MQ/edit#gid=884228058)) to highlight what was and was not covered by the testing framework.
|
||||
* These files also contain a coverage % at the top of the file. Full coverage is the percent of the actions of the processed required coverage test that were executed and fully covered by the framework. Partial coverage also includes actions that are partially covered by the framework.
|
||||
* This includes loss of coverage from any disabled tests. Cool!
|
||||
|
||||
### Exploring the tested and coverage graphs
|
||||
|
||||
To view the directed graphs that are generated to process the test and coverage data, the `--graphs` switch can be specified:
|
||||
```sh
|
||||
chrome/test/webapps/generate_framework_tests_and_coverage.py --graphs
|
||||
```
|
||||
|
||||
This will generate:
|
||||
* `coverage_required_graph.dot` - The graph of all of the required test coverage. Green nodes are actions explicitly listed in the coverage list, and orange nodes specify partial coverage paths.
|
||||
* `framework_test_graph_<platform>.dot` - The graph that is now tested by the generated framework tests for the given platform, including partial coverage.
|
||||
|
||||
The [graphviz](https://graphviz.org/) library can be used to view these graphs. An install-free online version is [here](https://dreampuf.github.io/GraphvizOnline/).
|
||||
|
||||
### Debugging further
|
||||
|
||||
To help debug or explore further, please see the [`graph_cli_tool.py`](graph_cli_tool.py) script which includes a number of command line utilities to process the various files.
|
||||
|
||||
Both this file and the [`generate_framework_tests_and_coverage.py`](generate_framework_tests_and_coverage.py) file support the `-v` option to print out informational logging.
|
||||
Documentation has moved [here](/docs/webapps/integration-testing-framework.md).
|
||||
|
||||
140
docs/webapps/how-to-create-webapp-integration-tests.md
Normal file
140
docs/webapps/how-to-create-webapp-integration-tests.md
Normal file
@ -0,0 +1,140 @@
|
||||
# How to create WebApp Integration Tests
|
||||
|
||||
Please see the the [Integration Testing Framework document][integration-testing-framework] for more information about specifics.
|
||||
|
||||
## 1. Familiarize yourself with the test generation, building, and running process.
|
||||
|
||||
- Run `chrome/test/webapps/generate_framework_tests_and_coverage.py` and verify nothing is outputted to the console.
|
||||
- Build the tests by building the targets `browser_tests` and `sync_integration_tests`
|
||||
- (e.g. `autoninja -C out/Release browser_tests sync_integration_tests`)
|
||||
- Run the generated tests, using the filter `--gtest_filter=*WebAppIntegration_*`:
|
||||
- `testing/run_with_dummy_home.py testing/xvfb.py out/Release/browser_tests --gtest_filter=*WebAppIntegration_*`
|
||||
- `testing/run_with_dummy_home.py testing/xvfb.py out/Release/sync_integration_tests --gtest_filter=*WebAppIntegration_*`
|
||||
- These will take a long time! No need to run them all, this is just so you know how to run them if you need to.
|
||||
|
||||
## 2. Determine what actions are needed for new critical user journeys
|
||||
|
||||
At the end of this process, all critical user journeys be decomposed into existing and new user actions and enumerated in the [Critical User Journey Spreadsheet][cuj-spreadsheet]. However, for this step the goal is to create any new actions that those journeys might need. See all existing actions in the [Actions sub-sheet][cuj-actions-sheet], which should give a good idea of what actions are currently supported in the framework.
|
||||
|
||||
Given the existing actions:
|
||||
1. Draft what our critical user journeys will be.
|
||||
2. Figure out if any new actions (or sites) will need to be implemented for these new journeys
|
||||
|
||||
Please [contact the team](#contact-the-team) if you have any questions or trouble here.
|
||||
|
||||
### Example - File Handlers
|
||||
The file handlers feature:
|
||||
- Allows a web app to register itself as a file handler on the operating system.
|
||||
- Launches the web app when a registered file is launched by the system.
|
||||
- Protects the user by presenting them with a confirmation dialog on launch, allowing the user to confirm or deny the launch.
|
||||
|
||||
What critical user journeys will be needed? Generally:
|
||||
- Verify that installing a site with file handlers registers that site as a file handler on the system.
|
||||
- Verify that launching a registered file will trigger the user confirmation dialog
|
||||
- Verify that each user choice causes the correct behavior to happen.
|
||||
- "allow without remember" will launch the app. If the file is launched again, the user should be presented with the dialog again.
|
||||
- "allow with remember" will launch the app. If the file is launched again, the user should NOT be presented with the dialog.
|
||||
- "deny without remember" will NOT launch the app. If the file is launched again, the user should be presented with the dialog again.
|
||||
- "deny with remember" will NOT launch the app and unregister the app as a file handler. The operating system should no longer have the file registered with the web app.
|
||||
|
||||
The existing [actions][cuj-actions-sheet] already have a lot of support for installing, launching, checking if a window was created, etc. The following changes will have to happen:
|
||||
- Modify an existing site (maybe Site B?), or create a new site (Site D? File Handler?), which handles a test file type in it's manifest.
|
||||
- Because multiple browsertests can be running at the same time on a trybot, this type will probably have to be uniquely generated per test to avoid conflicts.
|
||||
- Action 1: detect if a file type is registered on an operating system.
|
||||
- This will be unique per operating system.
|
||||
- Action 2: Launch the associated file.
|
||||
- Due to operating system complexities, it might be hard to have this be a true 'launch from the operating system'. It may have to be faked in our code, but we want it to be as close as possible to what would happen if the operating system actually launched the file.
|
||||
- Action 3 & 4: User action to allow or deny the dialog, with a "mode" of "Remember" "NotRemember".
|
||||
|
||||
## 3. Create action implementation with 'manual' tests
|
||||
|
||||
The goal of this step is to implement the actions (or other changes) that were determined by the last step. The action should be tested to make sure there are no bugs and it works on all applicable platforms.
|
||||
|
||||
For details about how to implement actions, see [Creating Actions in the `WebAppIntegrationTestDriver`][creating-actions].
|
||||
|
||||
Implementing or changing actions is usually done in [`WebAppIntegrationTestDriver`](https://source.chromium.org/search?q=WebAppIntegrationTestDriver&ss=chromium). If the action only works with the sync system, then it may have to be implemented in the `TestDelegate` interface and then in the [`TwoClientWebAppsIntegrationTestBase`](https://source.chromium.org/search?q=TwoClientWebAppsIntegrationTestBase&sq=&ss=chromium). See [test partitioning][test-partitioning] for more information.
|
||||
|
||||
Adding or modifying sites would happen in the [test data directory](https://source.chromium.org/chromium/chromium/src/+/main:chrome/test/data/web_apps/).
|
||||
|
||||
After the action and/or other changes are implemented, one or more "manual" tests should be implemented to ensure that they are working properly. See [test partitioning][test-partitioning] for more information about these various files, but basically:
|
||||
- Non-sync related tests are put in the [regular browsertests][regular-browsertests] (windows-mac-linux only example [here][regular-browsertests-wml]).
|
||||
- Sync-related tests are put in the [sync partition][sync-browsertests-wml].
|
||||
|
||||
### Example - `UninstallFromList`
|
||||
|
||||
[This](https://chromium-review.googlesource.com/c/chromium/src/+/3252543) is an example CL of implementing an action. It:
|
||||
1. Adds the action implementation to the [`WebAppIntegrationTestDriver`](https://source.chromium.org/search?q=WebAppIntegrationTestDriver&ss=chromium).
|
||||
2. Creates a simple "manual" (not generated) test of the action which runs on all platforms.
|
||||
|
||||
## 4. Add the new Critical User Journeys to the [spreadsheet][cuj-spreadsheet], generate, and submit them.
|
||||
|
||||
Finally, now that the changes are implemented and tested, they can be used in generated critical user journey tests. Please work with someone on the team (see [contact the team](#contact-the-team)) to review the new tests being added here and to help make sure nothing is missed. Working together you will add those new tests (and possibly actions) to the [spreadsheet][cuj-spreadsheet] so that you can execute the following steps:
|
||||
|
||||
### 4.1. Download the new actions and user journeys.
|
||||
|
||||
This command will download the data from the [spreadsheet][cuj-spreadsheet] into the local checkout.
|
||||
|
||||
```bash
|
||||
chrome/test/webapps/download_data_from_sheet.py
|
||||
```
|
||||
|
||||
### 4.2 Mark the action as supported.
|
||||
|
||||
To have the script actually generate tests using the new actions, they must be marked as supported in the [supported actions file][supported-actions]. The support is specified by a symbol per platform:
|
||||
|
||||
- 🌕 - Full coverage - This means that the driver implements this action in a way that completely matches (or almost matches) the code paths that are used when the user triggers this action.
|
||||
- 🌓 - Partial coverage - This means that the testing framework implements this action in a way that accomplishes the state change or check, but does not fully match the code path that is used when the user triggers this action.
|
||||
- 🌑 - No coverage - This means the action is not supported.
|
||||
|
||||
After recording an action as supported here, the next step should generate new tests!
|
||||
|
||||
### 4.2. Generate test changes.
|
||||
|
||||
This command will output all changes that need to happen to the critical user journeys.
|
||||
|
||||
```bash
|
||||
chrome/test/webapps/generate_framework_tests_and_coverage.py
|
||||
```
|
||||
|
||||
The output should:
|
||||
1. Generate a coverage report for the change in the [data directory][script-data-dir].
|
||||
1. Print new tests that need to be added to the integration browsertest files.
|
||||
2. Print out test ids that need to be removed.
|
||||
|
||||
After you make changes to the integration browsertests, please re-run the above command to verify that all of the changes were performed and no mistakes were made.
|
||||
|
||||
Possible issues / Things to know:
|
||||
1. Tests being removed are often replaced by a test that has all the same actions, plus some new ones. Finding these & replacing these tests inline helps the diff look more reasonable.
|
||||
2. Sometimes a test that is removed is currently disabled. If this is the case, please find the new test that is being added that has all of the same actions as the old test, and also disable it (please add the same comments, etc). Because the old test was failing, likely the new test will be failing too.
|
||||
|
||||
After all tests are added, `git cl format` is often required. It's a good idea to test all of the new tests locally if you can, and then after local verification a patch can be uploaded, the the trybots can be run, and a review can be requested from the team.
|
||||
|
||||
### 4.3 (optional) Disable failing tests
|
||||
|
||||
If the "manual" browsertest didn't catch a bug that is now failing for the generated tests and there is no obvious fix for the bug, it is OK to submit the new tests as disabled. To do this:
|
||||
1. Try to figure out generally why the generated tests are failing, or what the problem is, and create a bug.
|
||||
2. Mark the affected tests as disabled and add a comment referencing the bug.
|
||||
3. Make sure to call `chrome/test/webapps/generate_framework_tests_and_coverage.py` again to update the coverage percentage.
|
||||
4. Submit the patch with tests disabled.
|
||||
5. Create a follow up patch to fix the action and re-enable the tests.
|
||||
|
||||
Why is this OK? Adding the generated tests can be a big pain, especially if others are modifying the tests as well. It is often better to get them compiling and submitted quickly with a few tests disabled instead of waiting until everything works.
|
||||
|
||||
### Example - `UninstallFromList`
|
||||
|
||||
[Here](https://chromium-review.googlesource.com/c/chromium/src/+/3252305) is an example CL of adding generated tests for the `UninstallFromList` action addition. During the development of this action, it was discovered that some of the critical user journeys were incorrect and needed updating - you can see this in the downloaded file changes.
|
||||
|
||||
## Contact the team
|
||||
|
||||
To contact the team for help, send an email to pwa-dev@chromium.org and/or post on #pwas on Chromium Slack.
|
||||
|
||||
[cuj-spreadsheet]: https://docs.google.com/spreadsheets/d/1d3iAOAnojp4_WrPky9exz1-mjkeulOJVUav5QYG99MQ/edit#gid=2008870403
|
||||
[cuj-actions-sheet]: https://docs.google.com/spreadsheets/d/1d3iAOAnojp4_WrPky9exz1-mjkeulOJVUav5QYG99MQ/edit#gid=1864725389
|
||||
[regular-browsertests]: https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/ui/views/web_apps/web_app_integration_browsertest.cc
|
||||
[regular-browsertests-wml]: https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/ui/views/web_apps/web_app_integration_browsertest_mac_win_linux.cc
|
||||
[sync-browsertests-wml]: https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/sync/test/integration/two_client_web_apps_integration_test_mac_win_linux.cc
|
||||
[test-partitioning]: todo
|
||||
[supported-actions]: ../../chrome/test/webapps/data/framework_supported_actions.csv
|
||||
[script-data-dir]: ../../chrome/test/webapps/data/
|
||||
[integration-testing-framework]: integration-testing-framework.md
|
||||
[creating-actions]: integration-testing-framework.md#creating-actions-in-the-webappintegrationtestdriver
|
||||
259
docs/webapps/integration-testing-framework.md
Normal file
259
docs/webapps/integration-testing-framework.md
Normal file
@ -0,0 +1,259 @@
|
||||
# Desktop Web App Integration Testing Framework
|
||||
|
||||
[TOC]
|
||||
|
||||
## Background
|
||||
|
||||
The WebAppProvider system has very wide action space and testing state interactions between all of the subsystems is very difficult. The integration testing framework is intended to help testing critical user journeys for installable web apps in a scalable fashion.
|
||||
|
||||
The framework and process is broken down into the following pieces:
|
||||
1. A list of [critical user journeys][cuj-spreadsheet] and the [actions][cuj-actions-sheet] used to make those journeys.
|
||||
2. A [script][generate-script] that can process these (along with information about which action is supported by which platform).
|
||||
3. [Tests][default-tests] generated by the script that use the [`WebAppIntegrationTestDriver`][test-driver] to execute actions.
|
||||
4. [Coverage][coverage-win] information about what percentage of the critical user journeys are covered.
|
||||
|
||||
How-tos:
|
||||
* [How to create WebApp Integration Tests][how-to-create]
|
||||
* [Disabling a test](#disabling-a-test)
|
||||
|
||||
Related:
|
||||
* [WebAppProvider README.md](/chrome/browser/web_applications/README.md)
|
||||
|
||||
## Future Work
|
||||
|
||||
* Allowing parameterized state-check actions.
|
||||
* Making test generation friendlier for the developer.
|
||||
* Detecting incorrect tests.
|
||||
* Modifying tests automatically in common scenarios.
|
||||
|
||||
## Terminology
|
||||
|
||||
### Action
|
||||
A primitive test operation, or test building block, that can be used to create coverage tests. The integration test framework may or may not support an action on a given platform. Actions can fall into one of three types:
|
||||
* State-change action
|
||||
* State-check action
|
||||
* parameterized action
|
||||
|
||||
Actions also can be fully, partially, or not supported on a given platform. This information is used when generating the tests to run & coverage report for a given platform. To make parsing easier, actions are always snake_case.
|
||||
|
||||
#### State-change Action
|
||||
A state-changing action is expected to change the state chrome or the web app provider system.
|
||||
|
||||
Examples: `navigate_browser(SiteA)`, `switch_incognito_profile`, `sync_turn_off`, `set_app_badge`
|
||||
|
||||
#### State-check Action
|
||||
Some actions are classified as "state check" actions, which means they do not change any state and only inspect the state of the system. In graph representations, state check actions are not given nodes, and instead live under the last non-state-check action.
|
||||
|
||||
All actions that start with `check_` are considered state-check actions.
|
||||
|
||||
Examples: `check_app_list_empty`, `check_install_icon_shown`, `check_platform_shortcut_exists(SiteA)`, `check_tab_created`
|
||||
|
||||
#### Action Mode
|
||||
When creating tests, there emerged a common scenario where a given action could be applied to multiple different sites. For example, the “navigate the browser to an installable site” action was useful if “site” could be customized.
|
||||
|
||||
The simplest possible mode system to solve this:
|
||||
* Each action can have at most one mode.
|
||||
* Modes are static / pre-specified per action.
|
||||
* A default mode can be specified to support the case where an action has modes but none were specified in the test.
|
||||
|
||||
To allow for future de-parsing of modes (when generating C++ tests), modes will always be CapitalCase.
|
||||
|
||||
#### Parameterized Action
|
||||
To help with testing scenarios like outlined above, an action can be defined that references or 'turns into' a set of non-parameterized actions. For example, an action `install_windowed` can be created and reference the set of actions `install_omnibox_icon`, `install_menu_option`, `install_create_shortcut_windowed`, `add_policy_app_windowed_shortcuts`, and `add_policy_app_windowed_no_shortcuts`. When a test case includes this action, it will generate multiple tests in which the parameterized action is replaced with the non-parameterized action.
|
||||
|
||||
### Tests
|
||||
A sequence of actions used to test the WebAppProvider system. A test that can be run by the test framework must not have any "parameterized" actions, as these are supposed to be used to generate multiple tests.
|
||||
|
||||
#### Unprocessed Required-coverage tests
|
||||
This is the set of tests that, if all executed, should provide full test coverage for the WebAppProvider system. They currently live in this sheet as "unprocessed".
|
||||
|
||||
#### Required-coverage tests (processed)
|
||||
Processed tests go through the following steps from the unprocessed version in the sheet:
|
||||
* Tests with one or more "parameterized" actions have been processed to produce the resulting tests without parameterized actions.
|
||||
* Actions in tests that have modes but do not specify them have the default mode added to them.
|
||||
|
||||
#### Platform-specific tests
|
||||
Some tests are going to be platform-specific. For example, all tests that involve "locally installing" an app are only applicable on Windows/Mac/Linux, as ChromeOS automatically locally installs all apps from sync. Because of this, tests must be able to specify which platforms they should be run on. This is done by specifying the platforms each test applies to in a column on the spreadsheet.
|
||||
|
||||
### Sync Partition and Default Partition
|
||||
Due to some browsertest support limitations, certain actions are only supported in the sync testing framework. Because of this, the script supports a separate "partition" of tests for any test that uses sync actions. This means that at test output time, a test will either go in the "sync" partition or the "default" partition.
|
||||
|
||||
See the [sync tests design doc][sync-tests-dd] for more information.
|
||||
|
||||
## Script Design & Usage
|
||||
The [script][generate-script] takes the following information:
|
||||
* A list of [action][cuj-actions-sheet]-based [tests][cuj-spreadsheet] which fully test the WebAppProvider system (a.k.a. required-coverage tests).
|
||||
* A list of actions [supported][framework-supported-actions] by the integration test framework (per-platform).
|
||||
|
||||
The results of running the script is:
|
||||
* Console output (to stdout) of the minimal number of tests (per-platform) to run to achieve the maximum coverage of the critical user journeys.
|
||||
* If tests already exist, then these are taken into account and not printed.
|
||||
* If any existing tests are unnecessary, then the script will inform the developer that they can be removed.
|
||||
* The resulting [coverage][coverage-win] of the system (with required-coverage tests as 100%).
|
||||
|
||||
See the [design doc][design-doc] for more information and links.
|
||||
|
||||
### Downloading test data
|
||||
|
||||
The test data is hosted in this [spreadsheet][cuj-spreadsheet]. To download the latest copy of the data, run the included script:
|
||||
```sh
|
||||
./chrome/test/webapps/download_data_from_sheet.py
|
||||
```
|
||||
|
||||
This will download the data from the sheet into csv files in the [data/][script-data-dir] directory:
|
||||
|
||||
* `actions.csv` This describes all actions that can be used in the required coverage tests (processed or unprocessed).
|
||||
* `coverage_required.csv` This is the full list of all tests needed to fully cover the Web App system. The first column specifies the platforms for testing, and the test starts on the fifth column.
|
||||
|
||||
### Generating test descriptions & coverage
|
||||
|
||||
Required test changes are printed and coverage files are written by running:
|
||||
```sh
|
||||
chrome/test/webapps/generate_framework_tests_and_coverage.py
|
||||
```
|
||||
This uses the files in `chrome/test/webapps/data` and existing browsertests on the system (see `custom_partitions` and `default_partitions` in [`generate_framework_tests_and_coverage.py`][generate-script]) to:
|
||||
|
||||
#### 1) Print to `stdout` all detected changes needed to browsertests.
|
||||
The script is not smart enough to automatically add/remove/move tests to keep complexity to a minimum. Instead, it prints out the tests that need to be added or removed to have the tests match what it expects. It assumes:
|
||||
* Browsertests are correctly described by the `TestPartitionDescription`s in [`generate_framework_tests_and_coverage.py`][generate-script].
|
||||
* Browsertests with the per-platform suffixes (e.g. `_mac`, `_win`, etc) are only run on those platforms
|
||||
|
||||
This process doesn't modify the browsertest files so any test disabling done by sheriffs can remain. The script runner is thus expected to make the requested changes manually. In the rare case that a test is moving between files (if we are enabling a test on a new platform, for example), then the script runner should be careful to copy any sheriff changes to the browsertest as well.
|
||||
|
||||
#### 2) Generate per-platform processed required coverage `tsv` files in `chrome/test/webapps/coverage`
|
||||
These are the processed required coverage tests with markers per action to allow a conditional formatter (like the one [here][cuj-coverage-sheet]) to highlight what was and was not covered by the testing framework.
|
||||
* These files also contain a coverage % at the top of the file. Full coverage is the percent of the actions of the processed required coverage test that were executed and fully covered by the framework. Partial coverage also includes actions that are partially covered by the framework.
|
||||
* This includes loss of coverage from any disabled tests. Cool!
|
||||
|
||||
### Exploring the tested and coverage graphs
|
||||
|
||||
To view the directed graphs that are generated to process the test and coverage data, the `--graphs` switch can be specified:
|
||||
```sh
|
||||
chrome/test/webapps/generate_framework_tests_and_coverage.py --graphs
|
||||
```
|
||||
|
||||
This will generate:
|
||||
* `coverage_required_graph.dot` - The graph of all of the required test coverage. Green nodes are actions explicitly listed in the coverage list, and orange nodes specify partial coverage paths.
|
||||
* `framework_test_graph_<platform>.dot` - The graph that is now tested by the generated framework tests for the given platform, including partial coverage.
|
||||
|
||||
The [graphviz](https://graphviz.org/) library can be used to view these graphs. An install-free online version is [here](https://dreampuf.github.io/GraphvizOnline/).
|
||||
|
||||
### Debugging Further
|
||||
|
||||
To help debug or explore further, please see the [`graph_cli_tool.py`](graph_cli_tool.py) script which includes a number of command line utilities to process the various files.
|
||||
|
||||
Both this file and the [`generate_framework_tests_and_coverage.py`](generate_framework_tests_and_coverage.py) file support the `-v` option to print out informational logging.
|
||||
|
||||
## [`WebAppIntegrationTestDriver`][test-driver] and Browsertest Implementation
|
||||
|
||||
After the script has output the tests that are needed, they still need to be compiled and run by something. The [`WebAppIntegrationTestDriver`][test-driver] is what runs the actions, and the browsertests themselves are put into specific files based on which partition they will be run in (default or sync), and which platforms will be running them.
|
||||
|
||||
These are all of the files that make up the browsertest and browsertest support of the dPWA integration test framework:
|
||||
|
||||
* [`WebAppIntegrationTestDriver`][test-driver] - This class implements most actions that are used by the generated tests.
|
||||
* [`web_app_integration_browsertest.cc`][default-tests] - These are the cross-platform tests in the default partition.
|
||||
* [`web_app_integration_browsertest_mac_win_linux.cc`][default-tests-mwl] - These are the default partition tests that only run on mac, windows, and linux.
|
||||
* [`web_app_integration_browsertest_cros.cc`][default-tests-cros].
|
||||
* `two_client_web_apps_integration_test_*` - These are the tests in the sync partition.
|
||||
* [`two_client_web_apps_integration_test_mac_win_linux.cc`][sync-tests-mwl]
|
||||
* [`two_client_web_apps_integration_test_cros.cc`][sync-tests-cros]
|
||||
|
||||
### Creating Action Implementations
|
||||
|
||||
The driver implements all actions for the generated tests. For tests in the sync partition (which require the functionality of the [`SyncTest`][sync-test-base] base class), some actions are delegated to the [`two_client_web_apps_integration_test_base.h`][sync-tests-base]).
|
||||
|
||||
Testing actions must:
|
||||
* Call the appropriate `BeforeState*Action()` and `AfterState*Action()` functions inside of their function body.
|
||||
* Wait until the action is fully completed before returning.
|
||||
* Try to exercise code as close to the user-level action as reasonably possible.
|
||||
* Accommodate multiple browsertests running at the same time on the trybot (be careful modifying global information on an operating system).
|
||||
* Ensure that, at the end of the test, all side effects are cleaned up in `TearDownOnMainThread()`.
|
||||
|
||||
To help with state-check actions, the state of the system is recorded before and after every state-change action. This allows for actions to detect any changes that happened in the last state-change action. This is stored in `before_state_change_action_state_` and `after_state_change_action_state_`, and generated by calling `ConstructStateSnapshot()`.
|
||||
|
||||
When adding actions, it may be useful to add information into this state snapshot in order to verify the results of state-change actions within state-check actions. To do this:
|
||||
* Add a field onto the relevant state object.
|
||||
* Update the `==` and `<<` operators.
|
||||
* Populate the field in the `ConstructStateSnapshot()` method.
|
||||
|
||||
Then, this field can be accessed in the `before_state_change_action_state_` and `after_state_change_action_state_` members appropriately.
|
||||
|
||||
### Disabling a Test
|
||||
|
||||
Tests can be disabled in the same manner that other integration/browser tests are disabled, using macros. See [on disabling tests](/docs/testing/on_disabling_tests.md) for more information.
|
||||
|
||||
## Understanding and Implementing Test Cases
|
||||
|
||||
Actions are the basic building blocks of integration tests. A test is a sequence of actions. Each action has a name that must be a valid C++ identifier.
|
||||
|
||||
Actions are defined (and can be modified) in [this][cuj-actions-sheet] sheet. Tests are defined (and can be modified) in [this][cuj-spreadsheet] sheet.
|
||||
|
||||
### Action Creation & Specification
|
||||
|
||||
[Actions][cuj-actions-sheet] are the building blocks of tests.
|
||||
|
||||
#### Templates
|
||||
To help making test writing less repetitive, actions are described as templates in the [actions][cuj-actions-sheet] spreadsheet. Action templates specify actions while avoiding rote repetition. Each action template has a name (the **action base name**). Each action template supports a mode, which takes values from a predefined list associated with the template. Parameter values must also be valid C++ identifiers.
|
||||
|
||||
An action template without modes specifies one action whose name matches the template. For example, the `check_tab_created` template generates the `check_tab_created` action.
|
||||
|
||||
An action template with a mode that can take N values specifies N actions, whose names are the concatenations of the template name and the corresponding value name, separated by an underscore (`_`). For example, the `clear_app_badge` template generates the `clear_app_badge_SiteA` and `clear_app_badge_SiteB` actions.
|
||||
|
||||
The templates also support [parameterizing](#parameterized-action) an action, which causes any test that uses the action to be expanded into multiple tests, one per specified output action. Modes will carry over into the output action, and if an output action doesn't support a given mode then that parameterization is simply excluded during test generation.
|
||||
|
||||
#### Default Values
|
||||
|
||||
All templates with modes can mark one of their mode values as the default value. This value is used to magically convert template names to action names.
|
||||
|
||||
#### Specifying a Mode
|
||||
Human-friendly action names are a slight variation of the canonical names above.
|
||||
|
||||
Actions generated by mode-less templates have the same human-friendly name as their (canonical?) name.
|
||||
|
||||
Actions generated by parametrized templates use parenthesis to separate the template name from the value name. For example, the actions generated by the `clear_app_badge` template have the human-friendly names `clear_app_badge(SiteA)` and `clear_app_badge(SiteB)`.
|
||||
|
||||
The template name can be used as the human-friendly name of the action generated by the template with the default value. For example, `clear_app_badge` is a shorter human-friendly name equivalent to `clear_app_badge(SiteA)`.
|
||||
|
||||
### Test Creation & Specification
|
||||
|
||||
[Tests][cuj-spreadsheet] are created specifying actions.
|
||||
|
||||
For a step-by-step guide for creating a new integration test, see [this guide][how-to-create].
|
||||
|
||||
#### Mindset
|
||||
The mindset for test creation and organization is to really exhaustively check every possible string of user actions. The framework will automatically combine tests that are the same except for state check actions. They are currently organized by:
|
||||
1. **Setup actions** - The state change actions needed to enter the system state that is being tested.
|
||||
2. **Primary state-change action/s** - Generally one action that will change the system state after the setup.
|
||||
3. **State check action** - One state check action, checking the state of the system after the previous actions have executed.
|
||||
|
||||
Each test can have at most one [state check](#state-check-action) action as the last action.
|
||||
|
||||
One way to enumerate tests is to think about **affected-by** action edges. These are a pair of two actions, where the first action affects the second. For example, the action `set_app_badge` will affect the action `check_app_badge_has_value`. Or, `uninstall_from_app_list` will affect `check_platform_shortcut_exists`. There is often then different setup states that would effect these actions. Once these edges are identified, tests can be created around them.
|
||||
|
||||
#### Creating a test
|
||||
|
||||
A test should be created that does the bare minimum necessary to set up the test before testing the primary state change action and then checking the state.
|
||||
|
||||
The framework is designed to be able to collapse tests that contain common non-'state-check' actions, so adding a new test does not always mean that a whole new test will be run by the framework. Sometimes it only adds a few extra state-check actions in an existing test.
|
||||
|
||||
If new actions are required for a test, see [How to create WebApp Integration Tests][how-to-create] for more information about how to add a new action.
|
||||
|
||||
[design-doc]: https://docs.google.com/document/d/e/2PACX-1vTFI0sXhZMvvg1B3sctYVUe64WbLVNzuXFUa6f3XyYTzKs2JnuFR8qKNyXYZsxE-rPPvsq__4ZCyrcS/pub
|
||||
[cuj-spreadsheet]: https://docs.google.com/spreadsheets/d/e/2PACX-1vSbO6VsnWsq_9MN6JEXlL8asMqATHc2-pz9ed_Jlf5zHJGg2KAtegsorHqkQ5kydU6VCqebv_1gUCD5/pubhtml
|
||||
[cuj-actions-sheet]: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vSbO6VsnWsq_9MN6JEXlL8asMqATHc2-pz9ed_Jlf5zHJGg2KAtegsorHqkQ5kydU6VCqebv_1gUCD5/pubhtml?gid=1864725389
|
||||
[cuj-coverage-sheet]: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vSbO6VsnWsq_9MN6JEXlL8asMqATHc2-pz9ed_Jlf5zHJGg2KAtegsorHqkQ5kydU6VCqebv_1gUCD5/pubhtml?gid=884228058
|
||||
[how-to-create]: how-to-create-webapp-integration-tests.md
|
||||
[script-data-dir]: /chrome/test/webapps/data/
|
||||
[script-output-dir]: /chrome/test/webapps/output/
|
||||
[test-driver]: /chrome/browser/ui/views/web_apps/web_app_integration_test_driver.h
|
||||
[default-tests]: /chrome/browser/ui/views/web_apps/web_app_integration_browsertest.cc
|
||||
[default-tests-mwl]: /chrome/browser/ui/views/web_apps/web_app_integration_browsertest_mac_win_linux.cc
|
||||
[default-tests-cros]: /chrome/browser/ui/views/web_apps/web_app_integration_browsertest_cros.cc
|
||||
[sync-tests-mwl]: /chrome/browser/sync/test/integration/two_client_web_apps_integration_test_mac_win_linux.cc
|
||||
[sync-tests-cros]: /chrome/browser/sync/test/integration/two_client_web_apps_integration_test_cros.cc
|
||||
[sync-tests-base]: /chrome/browser/sync/test/integration/two_client_web_apps_integration_test_base.h
|
||||
[sync-tests-dd]: https://docs.google.com/document/d/139ktCajbmbFKh4T-vEhipTxilyYrXf_rlCBHIvrdeSg/edit
|
||||
[generate-script]: /chrome/test/webapps/generate_framework_tests_and_coverage.py
|
||||
[coverage-win]: /chrome/test/webapps/coverage/coverage_win.tsv
|
||||
[framework-supported-actions]: /chrome/test/webapps/data/framework_supported_actions.csv
|
||||
[sync-test-base]: chrome/browser/sync/test/integration/sync_test.h
|
||||
Reference in New Issue
Block a user