3a40f9f2a4
Change-Id: I86c0e79a6b22dbb2d98d5a5d8dd9cacf8baca79a Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3935170 Reviewed-by: Avi Drissman <avi@chromium.org> Reviewed-by: Nicolás Peña <npm@chromium.org> Commit-Queue: Mustaq Ahmed <mustaq@chromium.org> Cr-Commit-Position: refs/heads/main@{#1057589}
120 lines
5.2 KiB
Markdown
120 lines
5.2 KiB
Markdown
# Web Tests with Manual Fallback
|
|
|
|
Some Blink features cannot be automatically tested using the Web Platform. Prime
|
|
examples are the APIs that require
|
|
[user activation](https://html.spec.whatwg.org/multipage/interaction.html#tracking-user-activation)
|
|
(also known as _a user gesture_), such as [Fullscreen](https://developer.mozilla.org/en-US/docs/Web/API/Fullscreen_API).
|
|
Automated tests for these Blink features must rely on special APIs, which are
|
|
only exposed in testing environments, and are therefore not available in a
|
|
normal browser session.
|
|
|
|
A popular pattern used in these tests is to rely on the user to perform some
|
|
manual steps in order to run the test case in a normal browser session. These
|
|
tests are effectively
|
|
[manual tests](https://web-platform-tests.org/writing-tests/manual.html), with
|
|
additional JavaScript code that automatically performs the desired manual steps,
|
|
when loaded in an environment that exposes the needed testing APIs.
|
|
|
|
## Motivation
|
|
|
|
Web tests that degrade to manual tests in the absence of testing APIs have
|
|
the following benefits.
|
|
|
|
* The manual test component can be debugged in a normal browser session, using
|
|
the rich [developer tools](https://developer.chrome.com/devtools). Tests
|
|
without a manual fallback can only be debugged in the test runner.
|
|
* The manual tests can run in other browsers, making it easy to check whether
|
|
our behavior matches other browsers.
|
|
* The web tests can form the basis for manual tests that are contributed to
|
|
[web-platform-tests](./web_platform_tests.md).
|
|
|
|
Therefore, the desirability of adding a manual fallback to a test heavily
|
|
depends on whether the feature under test is a Web Platform feature or a
|
|
Blink-only feature, and on the developer's working style. The benefits above
|
|
should be weighed against the added design effort needed to build a manual test,
|
|
and the size and complexity introduced by the manual fallback.
|
|
|
|
## Development Tips
|
|
|
|
A natural workflow for writing a web test that gracefully degrades to a
|
|
manual test is to first develop the manual test in a browser, and then add code
|
|
that feature-checks for testing APIs, and uses them to automate the test's
|
|
manual steps.
|
|
|
|
Manual tests should minimize the chance of user error. This implies keeping the
|
|
manual steps to a minimum, and having simple and clear instructions that
|
|
describe all the configuration changes and user gestures that match the effect
|
|
of the Blink-specific APIs used by the test.
|
|
|
|
## Example
|
|
|
|
Below is an example of a fairly minimal test that uses a Blink-Specific API
|
|
(`window.eventSender`), and gracefully degrades to a manual test.
|
|
|
|
```html
|
|
<!doctype html>
|
|
<meta charset="utf-8">
|
|
<title>DOM: Event.isTrusted for UI events</title>
|
|
<link rel="help" href="https://dom.spec.whatwg.org/#dom-event-istrusted">
|
|
<link rel="help" href="https://dom.spec.whatwg.org/#constructing-events">
|
|
<meta name="assert"
|
|
content="Event.isTrusted is true for events generated by user interaction">
|
|
<script src="../../resources/testharness.js"></script>
|
|
<script src="../../resources/testharnessreport.js"></script>
|
|
|
|
<p>Please click on the button below.</p>
|
|
<button>Click Me!</button>
|
|
|
|
<script>
|
|
'use strict';
|
|
|
|
setup({ explicit_timeout: true });
|
|
|
|
promise_test(() => {
|
|
const button = document.querySelector('button');
|
|
return new Promise((resolve, reject) => {
|
|
const button = document.querySelector('button');
|
|
button.addEventListener('click', (event) => {
|
|
resolve(event);
|
|
});
|
|
|
|
if (window.eventSender) {
|
|
eventSender.mouseMoveTo(button.offsetLeft, button.offsetTop);
|
|
eventSender.mouseDown();
|
|
eventSender.mouseUp();
|
|
}
|
|
}).then((clickEvent) => {
|
|
assert_true(clickEvent.isTrusted);
|
|
});
|
|
|
|
}, 'Click generated by user interaction');
|
|
|
|
</script>
|
|
```
|
|
|
|
The test exhibits the following desirable features:
|
|
|
|
* It has a second specification URL (`<link rel="help">`), because the paragraph
|
|
that documents the tested feature (referenced by the primary URL) is not very
|
|
informative on its own.
|
|
* It links to the
|
|
[WHATWG Living Standard](https://wiki.whatwg.org/wiki/FAQ#What_does_.22Living_Standard.22_mean.3F),
|
|
rather than to a frozen version of the specification.
|
|
* It contains clear instructions for manually triggering the test conditions.
|
|
The test starts with a paragraph (`<p>`) that tells the tester exactly what to
|
|
do, and the `<button>` that needs to be clicked is clearly labeled.
|
|
* It disables the timeout mechanism built into `testharness.js` by calling
|
|
`setup({ explicit_timeout: true });`
|
|
* It checks for the presence of the Blink-specific testing APIs
|
|
(`window.eventSender`) before invoking them. The test does not automatically
|
|
fail when the APIs are not present.
|
|
* It uses [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise)
|
|
to separate the test setup from the assertions. This is particularly helpful
|
|
for manual tests that depend on a sequence of events to occur, as Promises
|
|
offer a composable way to express waiting for asynchronous events that avoids
|
|
[callback hell](http://stackabuse.com/avoiding-callback-hell-in-node-js/).
|
|
|
|
Notice that the test is pretty heavy compared to a minimal JavaScript test that
|
|
does not rely on testing APIs. Only use testing APIs when the desired testing
|
|
conditions cannot be set up using Web Platform APIs.
|