chromium/src
0
Fork 0
Code Activity

Update "LayoutTests" / "layout test" in docs/*.md.

Browse Source 
This CL renames the following md files:
 - layout_tests_linux -> web_tests_linux.md
 - how_to_extend_layout_test_framework.md ->
   how_to_extend_web_test_framework.md
 - testing/layout_test_baseline_fallback.md ->
   testing/web_test_baseline_fallback.md
 - testing/layout_test_expectations.md -> testing/web_test_expectations.md
 - testing/layout_tests.md -> testing/web_tests.md
 - testing/layout_tests_in_content_shell.md ->
   testing/web_tests_in_content_shell.md
 - testing/layout_tests_tips.md -> testing/web_tests_tips.md
 - testing/layout_tests_with_manual_fallback.md ->
   testing/web_tests_with_manual_fallback.md
 - testing/writing_layout_tests -> testing/writing_web_tests.md

No-Try: true
Change-Id: Ied34815c92e449888b20818540a5ed0a76107bd1
Bug: 843412
Reviewed-on: https://chromium-review.googlesource.com/c/1350559
Commit-Queue: Kent Tamura <tkent@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#611024}
This commit is contained in:
Kent Tamura
2018-11-27 05:30:56 +00:00
committed by Commit Bot
parent 762831a3f4
commit 59ffb020bd
41 changed files with 281 additions and 289 deletions

27
docs/README.md

@ -145,20 +145,19 @@ used when committed.
work.
### Testing
* [Running and Debugging Layout Tests](testing/layout_tests.md)
* [Writing Layout Tests](testing/writing_layout_tests.md) - Layout Tests using
* [Running and Debugging Web Tests](testing/web_tests.md)
* [Writing Web Tests](testing/writing_web_tests.md) - Web Tests using
`content_shell`
* [Layout Test Expectations and Baselines](testing/layout_test_expectations.md) -
Setting expected results of layout tests.
* [Layout Tests Tips](testing/layout_tests_tips.md) - Best practices for Layout
Tests
* [Layout Tests with Manual Fallback](testing/layout_tests_with_manual_fallback.md) -
* [Web Test Expectations and Baselines](testing/web_test_expectations.md) -
Setting expected results of web tests.
* [Web Tests Tips](testing/web_tests_tips.md) - Best practices for web tests
* [Web Tests with Manual Fallback](testing/webt_tests_with_manual_fallback.md) -
Writing tests that simulate manual interventions
* [Extending the Layout Test Framework](how_to_extend_layout_test_framework.md)
* [Fixing Layout Test Flakiness](testing/identifying_tests_that_depend_on_order.md) -
Diagnosing and fixing layout test flakiness due to ordering dependencies.
* [Running Layout Tests using `content_shell`](testing/layout_tests_in_content_shell.md) -
Running layout tests by hand.
* [Extending the Web Test Framework](how_to_extend_web_test_framework.md)
* [Fixing Web Test Flakiness](testing/identifying_tests_that_depend_on_order.md) -
Diagnosing and fixing web test flakiness due to ordering dependencies.
* [Running Web Tests using `content_shell`](testing/web_tests_in_content_shell.md) -
Running web tests by hand.
* [Web Platform Tests](testing/web_platform_tests.md) - Shared tests across
browser vendors
* [Using Breakpad with `content_shell`](testing/using_breakpad_with_content_shell.md) -
@ -217,8 +216,8 @@ used when committed.
isolate browser components with different privileges.
* [Zygote Process](linux_zygote.md) - How the Linux Zygote process, used to
spawn new processes, works.
* [Running Layout Tests on Linux](layout_tests_linux.md) - Linux-specific
instructions for running layout tests.
* [Running Web Tests on Linux](web_tests_linux.md) - Linux-specific
instructions for running web tests.
* [Linux Sysroot Images](linux_sysroot.md) - How builds use libraries on Linux
* [Linux Hardware Video Decoding](linux_hw_video_decode.md) - Enabling
hardware video decode codepaths on Linux

6
docs/accessibility/tests.md

@ -22,7 +22,7 @@ a new one.
Test files:
```
third_party/WebKit/LayoutTests/accessibility
third_party/blink/web_tests/accessibility
```
Source code to AccessibilityController and WebAXObjectProxy:
@ -30,7 +30,7 @@ Source code to AccessibilityController and WebAXObjectProxy:
content/shell/test_runner
```
To run all accessibility LayoutTests:
To run all accessibility web tests:
```
autoninja -C out/release blink_tests
third_party/blink/tools/run_web_tests.py --build-directory=out --target=release accessibility/
@ -39,7 +39,7 @@ third_party/blink/tools/run_web_tests.py --build-directory=out --target=release
To run just one test by itself without the script:
```
autoninja -C out/release blink_tests
out/release/content_shell --run-web-tests third_party/WebKit/LayoutTests/accessibility/name-calc-inputs.html
out/release/content_shell --run-web-tests third_party/blink/web_tests/accessibility/name-calc-inputs.html
```
## DumpAccessibilityTree tests

4
docs/android_test_instructions.md

@ -284,9 +284,9 @@ out/Release/bin/java_deobfuscate out/Release/apks/ChromePublicTest.apk.mapping <
Any stacks produced by test runner output will already be deobfuscated.
## Running Blink Layout Tests
## Running Blink Web Tests
See [Layout Tests](testing/layout_tests.md).
See [Web Tests](testing/web_tests.md).
## Running GPU tests

2
docs/atom.md

@ -54,7 +54,7 @@ $ ./build.py --clang-completer --system-libclang
## JavaScript lint
Install JavaScript linter for Blink layout tests.
Install JavaScript linter for Blink web tests.
```
$ npm install -g eslint eslint-config-google

2
docs/how_to_add_your_feature_flag.md

@ -28,7 +28,7 @@ to see
4. how to use it in blink
[[1](https://chromium-review.googlesource.com/c/554510/8/third_party/blnk/renderere/core/workers/worker_thread.cc)]
Also, this patch added a virtual test for running layout tests with the flag.
Also, this patch added a virtual test for running web tests with the flag.
When you add a flag, you can consider to use that.
## Step 2: Adding the feature flag to the chrome://flags UI.

29
docs/how_to_extend_layout_test_framework.md → docs/how_to_extend_web_test_framework.md

@ -1,6 +1,6 @@
# How to Extend the Layout Test Framework
# How to Extend the Web Test Framework
The Layout Test Framework that Blink uses is a regression testing tool that is
The Web Test Framework that Blink uses is a regression testing tool that is
multi-platform and it has a large amount of tools that help test varying types
of regression, such as pixel diffs, text diffs, etc. The framework is mainly
used by Blink, however it was made to be extensible so that other projects can
@ -12,18 +12,17 @@ to help people who want to actually the framework to test whatever they want.
## Background
Before you can start actually extending the framework, you should be familiar
with how to use it. See the
[layout tests documentation](testing/layout_tests.md).
with how to use it. See the [web tests documentation](testing/web_tests.md).
## How to Extend the Framework
There are two parts to actually extending framework to test a piece of software.
The first part is extending certain files in:
[/third_party/blink/tools/blinkpy/web_tests/](/third_party/blink/tools/blinkpy/web_tests/)
The code in `blinkpy/web_tests` is the layout test framework itself
The code in `blinkpy/web_tests` is the web test framework itself
The second part is creating a driver (program) to actually communicate the
layout test framework. This part is significantly more tricky and dependent on
web test framework. This part is significantly more tricky and dependent on
what exactly exactly is being tested.
### Part 1
@ -31,10 +30,10 @@ what exactly exactly is being tested.
This part isn’t too difficult. There are basically two classes that need to be
extended (ideally, just inherited from). These classes are:
* `Driver`. Located in `layout_tests/port/driver.py`. Each instance of this is
* `Driver`. Located in `web_tests/port/driver.py`. Each instance of this is
the class that will actually an instance of the program that produces the
test data (program in Part 2).
* `Port`. Located in `layout_tests/port/base.py`. This class is responsible
* `Port`. Located in `web_tests/port/base.py`. This class is responsible
creating drivers with the correct settings, giving access to certain OS
functionality to access expected files, etc.
@ -129,7 +128,7 @@ Here are some of the functions that most likely need to be overridden.
* `layout_tests_dir`
* This tells the port where to look for all the and everything associated
with them such as resources files.
* By default it returns the absolute path to the layout tests directory.
* By default it returns the absolute path to the web tests directory.
* If you are planning on running something in the chromium src/ directory,
there are helper functions to allow you to return a path relative to the
base of the chromium src directory.
@ -137,7 +136,7 @@ Here are some of the functions that most likely need to be overridden.
The rest of the functions can definitely be overridden for your projects
specific needs, however these are the bare minimum needed to get it running.
There are also functions you can override to make certain actions that aren’t on
by default always take place. For example, the layout test framework always
by default always take place. For example, the web test framework always
checks for system dependencies unless you pass in a switch. If you want them
disabled for your project, just override `check_sys_deps` to always return OK.
This way you don’t need to pass in so many switches.
@ -173,7 +172,7 @@ and has stdout, stdin, stderr.
#### Goals
Your goal for this part of the project is to create a program (or extend a
program) to interface with the layout test framework. The layout test framework
program) to interface with the web test framework. The layout test framework
will communicate with this program to tell it what to do and it will accept data
from this program to perform the regression testing or create new base line
files.
@ -189,7 +188,7 @@ This is how your code should be laid out.
the `run_test()` in the driver.
1. Infinite Loop (!)
* After initialization, your program needs to actually wait for input,
then process that input to carry out the test. In the context of layout
then process that input to carry out the test. In the context of web
testing, the `content_shell` needs to wait for an html file to navigate
to, render it, then convert that rendering to a PNG. It does this
constantly, until a signal/message is sent to indicate that no more
@ -228,12 +227,12 @@ That’s basically what the skeleton of your program should be.
### Details
This is information about how to do some specific things, such as sending data
to the layout test framework.
to the web test framework.
* Content Blocks
* The layout test framework accepts output from your program in blocks of
* The web test framework accepts output from your program in blocks of
data through stdout. Therefore, printing to stdout is really sending
data to the layout test framework.
data to the web test framework.
* Structure of block
* “Header: Data\n”
* Header indicates what type of data will be sent through. A list

22
docs/infra/cq_builders.md

@ -116,18 +116,18 @@ builder.
* [linux-blink-gen-property-trees](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-blink-gen-property-trees) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux-blink-gen-property-trees)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux-blink-gen-property-trees))
Path regular expressions:
* [`//third_party/WebKit/LayoutTests/FlagExpectations/enable-blink-gen-property-trees`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/WebKit/LayoutTests/FlagExpectations/enable-blink-gen-property-trees)
* [`//third_party/WebKit/LayoutTests/flag-specific/enable-blink-gen-property-trees/.+`](https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/flag-specific/enable-blink-gen-property-trees/)
* [`//third_party/blink/web_tests/FlagExpectations/enable-blink-gen-property-trees`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/FlagExpectations/enable-blink-gen-property-trees)
* [`//third_party/blink/web_tests/flag-specific/enable-blink-gen-property-trees/.+`](https://cs.chromium.org/chromium/src/third_party/blink/web_tests/flag-specific/enable-blink-gen-property-trees/)
* [linux-blink-rel](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-blink-rel) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux-blink-rel)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux-blink-rel))
Path regular expressions:
* [`//cc/.+`](https://cs.chromium.org/chromium/src/cc/)
* [`//third_party/WebKit/LayoutTests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees)`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/WebKit/LayoutTests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees))
* [`//third_party/WebKit/LayoutTests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/WebKit/LayoutTests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/)
* [`//third_party/blink/renderer/core/layout/compositing/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/layout/compositing/)
* [`//third_party/blink/renderer/core/(svg|paint)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/renderer/core/(svg|paint)/)
* [`//third_party/blink/renderer/core/layout/compositing/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/layout/compositing/)
* [`//third_party/blink/renderer/platform/graphics/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/platform/graphics/)
* [`//third_party/blink/web_tests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees)`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees))
* [`//third_party/blink/web_tests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/)
* [linux_chromium_dbg_ng](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_dbg_ng) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_chromium_dbg_ng)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_chromium_dbg_ng))
@ -137,27 +137,27 @@ builder.
* [linux_layout_tests_layout_ng](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_layout_tests_layout_ng) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_layout_tests_layout_ng)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_layout_tests_layout_ng))
Path regular expressions:
* [`//third_party/Webkit/LayoutTests/FlagExpectations/enable-blink-features=LayoutNG`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/Webkit/LayoutTests/FlagExpectations/enable-blink-features=LayoutNG)
* [`//third_party/Webkit/LayoutTests/flag-specific/enable-blink-features=LayoutNG/.+`](https://cs.chromium.org/chromium/src/third_party/Webkit/LayoutTests/flag-specific/enable-blink-features=LayoutNG/)
* [`//third_party/blink/renderer/core/editing/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/editing/)
* [`//third_party/blink/renderer/core/(layout|paint)/ng/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/renderer/core/(layout|paint)/ng/)
* [`//third_party/blink/renderer/core/editing/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/editing/)
* [`//third_party/blink/renderer/platform/fonts/shaping/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/platform/fonts/shaping/)
* [`//third_party/blink/web_tests/FlagExpectations/enable-blink-features=LayoutNG`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/FlagExpectations/enable-blink-features=LayoutNG)
* [`//third_party/blink/web_tests/flag-specific/enable-blink-features=LayoutNG/.+`](https://cs.chromium.org/chromium/src/third_party/blink/web_tests/flag-specific/enable-blink-features=LayoutNG/)
* [linux_layout_tests_slimming_paint_v2](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_layout_tests_slimming_paint_v2) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_layout_tests_slimming_paint_v2)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_layout_tests_slimming_paint_v2))
Path regular expressions:
* [`//third_party/WebKit/LayoutTests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees)`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/WebKit/LayoutTests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees))
* [`//third_party/WebKit/LayoutTests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/WebKit/LayoutTests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/)
* [`//third_party/blink/renderer/core/layout/compositing/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/layout/compositing/)
* [`//third_party/blink/renderer/core/(svg|paint)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/renderer/core/(svg|paint)/)
* [`//third_party/blink/renderer/platform/graphics/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/platform/graphics/)
* [`//third_party/blink/web_tests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees)`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees))
* [`//third_party/blink/web_tests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/)
* [linux_mojo](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_mojo) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_mojo)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_mojo))
Path regular expressions:
* [`//services/network/.+`](https://cs.chromium.org/chromium/src/services/network/)
* [`//testing/buildbot/filters/mojo\\.fyi\\.network_.*`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:testing/buildbot/filters/mojo\\.fyi\\.network_.*)
* [`//third_party/WebKit/LayoutTests/FlagExpectations/enable-features=NetworkService`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/WebKit/LayoutTests/FlagExpectations/enable-features=NetworkService)
* [`//third_party/blink/web_tests/FlagExpectations/enable-features=NetworkService`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/FlagExpectations/enable-features=NetworkService)
* [linux_optional_gpu_tests_rel](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_optional_gpu_tests_rel) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_optional_gpu_tests_rel)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_optional_gpu_tests_rel))

2
docs/linux_build_instructions.md

@ -413,7 +413,7 @@ pulseaudio-libs-devel zlib.i686 httpd mod_ssl php php-cli python-psutil wdiff \
xorg-x11-server-Xvfb'
```
The fonts needed by Blink's LayoutTests can be obtained by following [these
The fonts needed by Blink's web tests can be obtained by following [these
instructions](https://gist.github.com/pwnall/32a3b11c2b10f6ae5c6a6de66c1e12ae).
For the optional packages:

12
docs/linux_debugging.md

@ -306,7 +306,7 @@ See [linux_minidump_to_core.md](linux_minidump_to_core.md)
Many of our tests bring up windows on screen. This can be annoying (they steal
your focus) and hard to debug (they receive extra events as you mouse over them).
Instead, use `Xvfb` or `Xephyr` to run a nested X session to debug them, as
outlined on [layout_tests_linux.md](layout_tests_linux.md).
outlined on [web_tests_linux.md](web_tests_linux.md).
### Browser tests
@ -323,10 +323,10 @@ browser process share the outermost process.
To debug a renderer process in this case, use the tips above about renderers.
### Layout tests
### Web tests
See [layout_tests_linux.md](layout_tests_linux.md) for some tips. In particular,
note that it's possible to debug a layout test via `ssh`ing to a Linux box; you
See [web_tests_linux.md](web_tests_linux.md) for some tips. In particular,
note that it's possible to debug a web test via `ssh`ing to a Linux box; you
don't need anything on screen if you use `Xvfb`.
### UI tests
@ -464,7 +464,7 @@ See the last section of [Linux Crash Dumping](linux_crash_dumping.md).
If you break in a debugger during a drag, Chrome will have grabbed your mouse
and keyboard so you won't be able to interact with the debugger! To work around
this, run via `Xephyr`. Instructions for how to use `Xephyr` are on the
[Running layout tests on Linux](layout_tests_linux.md) page.
[Running web tests on Linux](web_tests_linux.md) page.
## Tracking Down Bugs
@ -520,7 +520,7 @@ Some strategies are:
To test on various window managers, you can use a nested X server like `Xephyr`.
Instructions for how to use `Xephyr` are on the
[Running layout tests on Linux](layout_tests_linux.md) page.
[Running web tests on Linux](web_tests_linux.md) page.
If you need to test something with hardware accelerated compositing
(e.g., compiz), you can use `Xgl` (`sudo apt-get install xserver-xgl`). E.g.:

4
docs/linux_eclipse_dev.md

@ -143,8 +143,8 @@ In the Project Explorer on the left side:
1. Add another filter:
* Exclude all
* Folders
* Name matches `out_.*|\.git|LayoutTests` regular expression
* If you aren't working on WebKit, adding `|WebKit` will remove more
* Name matches `out_.*|\.git|web_tests` regular expression
* If you aren't working on WebKit, adding `|blink` will remove more
files
1. Click "Apply and Close"

8
docs/origin_trials_integration.md

@ -109,7 +109,7 @@ tools/origin_trials/generate_token.py http://localhost:8000 MyFeature
```
The file `tools/origin_trials/eftest.key` is used by default as the private key
for the test keypair used by Origin Trials unit tests and layout tests (i.e. in
for the test keypair used by Origin Trials unit tests and web tests (i.e. in
content shell). Tokens generated with this key will **not** work in the browser
by default (see the [Developer Guide] for instructions on creating real tokens).
To use a test token with the browser, run Chrome with the command-line flag:
@ -123,8 +123,8 @@ it doesn't work, see [trial_token_unittest.cc]. If you cannot set command-line
switches (e.g., on Chrome OS), you can also directly modify
[chrome_origin_trial_policy.cc].
### Layout Tests
When using the \[OriginTrialEnabled\] IDL attribute, you should add layout tests
### Web Tests
When using the \[OriginTrialEnabled\] IDL attribute, you should add web tests
to verify that the V8 bindings code is working as expected. Depending on how
your feature is exposed, you'll want tests for the exposed interfaces, as well
as tests for script-added tokens. For examples, refer to the existing tests in
@ -135,6 +135,6 @@ as tests for script-added tokens. For examples, refer to the existing tests in
[generate_token.py]: /tools/origin_trials/generate_token.py
[Developer Guide]: https://github.com/jpchase/OriginTrials/blob/gh-pages/developer-guide.md
[OriginTrialEnabled]: /third_party/blink/renderer/bindings/IDLExtendedAttributes.md#_OriginTrialEnabled_i_m_a_c_
[origin_trials/webexposed]: /third_party/WebKit/LayoutTests/http/tests/origin_trials/webexposed/
[origin_trials/webexposed]: /third_party/blink/web_tests/http/tests/origin_trials/webexposed/
[runtime\_enabled\_features.json5]: /third_party/blink/renderer/platform/runtime_enabled_features.json5
[trial_token_unittest.cc]: /third_party/blink/common/origin_trials/trial_token_unittest.cc

4
docs/servicification.md

@ -195,7 +195,7 @@ Below, we go through strategies for some common challenges encountered when
servicifying features that have Blink as a client.
#### Mocking Interface Impls in JS
It is a common pattern in Blink's layout tests to mock a remote Mojo interface
It is a common pattern in Blink's web tests to mock a remote Mojo interface
in JS. [This CL](https://codereview.chromium.org/2643713002) illustrates the
basic pattern for porting such mocking of an interface hosted by
//content/browser to an interface hosted by an arbitrary service (see the
@ -308,7 +308,7 @@ To answer this question, there are several different strategies. These
strategies are not mutually-exclusive; they can and should be combined to
preserve the full breadth of coverage.
- Blink client-side behavior can be tested via [layout tests](https://codereview.chromium.org/2731953003)
- Blink client-side behavior can be tested via [web tests](https://codereview.chromium.org/2731953003)
- To test service impl behavior, create [service tests](https://codereview.chromium.org/2774783003).
- To preserve tests of end-to-end behavior (e.g., that when Blink makes a
request via a Web API in JS, the relevant feature impl receives a connection

2
docs/speed/benchmark/harnesses/blink_perf.md

@ -5,7 +5,7 @@
## Overview
Blink perf tests are used for micro benchmarking the surface of Blink that
is exposed to the Web. They are the counterpart of [LayoutTests/](../../../docs/testing/layout_tests.md)
is exposed to the Web. They are the counterpart of [web_tests/](../../../docs/testing/layout_tests.md)
but for performance coverage.
## Writing Tests

2
docs/testing/identifying_tests_that_depend_on_order.md

@ -1,5 +1,5 @@
# Fixing layout test flakiness
# Fixing web test flakiness
We'd like to stamp out all the tests that have ordering dependencies. This helps
make the tests more reliable and, eventually, will make it so we can run tests

46
docs/testing/json_test_results_format.md

@ -14,7 +14,7 @@ output of the test run, and hence the test can be tailored for human readability
as a result). We also upload the test results to dashboards like the
[Flakiness Dashboard](http://test-results.appspot.com).
The test format originated with the Blink layout tests, but has since been
The test format originated with the Blink web tests, but has since been
adopted by GTest-based tests and Python unittest-based tests, so we've
standardized on it for anything related to tracking test flakiness.
@ -78,7 +78,7 @@ object may appear in any order.
|------------|-----------|-------------|
| `interrupted` | boolean | **Required.** Whether the test run was interrupted and terminated early (either via the runner bailing out or the user hitting ctrl-C, etc.) If true, this indicates that not all of the tests in the suite were run and the results are at best incomplete and possibly totally invalid. |
| `num_failures_by_type` | dict | **Required.** A summary of the totals of each result type. If a test was run more than once, only the first invocation's result is included in the totals. Each key is one of the result types listed below. A missing result type is the same as being present and set to zero (0). |
| `path_delimiter` | string | **Optional, will be mandatory.** The separator string to use in between components of a tests name; normally "." for GTest- and Python-based tests and "/" for layout tests; if not present, you should default to "/" for backwards-compatibility. |
| `path_delimiter` | string | **Optional, will be mandatory.** The separator string to use in between components of a tests name; normally "." for GTest- and Python-based tests and "/" for web tests; if not present, you should default to "/" for backwards-compatibility. |
| `seconds_since_epoch` | float | **Required.** The start time of the test run expressed as a floating-point offset in seconds from the UNIX epoch. |
| `tests` | dict | **Required.** The actual trie of test results. Each directory or module component in the test name is a node in the trie, and the leaf contains the dict of per-test fields as described below. |
| `version` | integer | **Required.** Version of the file format. Current version is 3. |
@ -87,10 +87,10 @@ object may appear in any order.
| `build_number` | string | **Optional.** If this test run was produced on a bot, this should be the build number of the run, e.g., "1234". |
| `builder_name` | string | **Optional.** If this test run was produced on a bot, this should be the builder name of the bot, e.g., "Linux Tests". |
| `chromium_revision` | string | **Optional.** The revision of the current Chromium checkout, if relevant, e.g. "356123". |
| `has_pretty_patch` | bool | **Optional, layout test specific, deprecated.** Whether the layout tests' output contains PrettyDiff-formatted diffs for test failures. |
| `has_wdiff` | bool | **Optional, layout test specific, deprecated.** Whether the layout tests' output contains wdiff-formatted diffs for test failures. |
| `layout_tests_dir` | string | **Optional, layout test specific.** Path to the LayoutTests directory for the test run (used so that we can link to the tests used in the run). |
| `pixel_tests_enabled` | bool | **Optional, layout test specific.** Whether the layout tests' were run with the --pixel-tests flag. |
| `has_pretty_patch` | bool | **Optional, layout test specific, deprecated.** Whether the web tests' output contains PrettyDiff-formatted diffs for test failures. |
| `has_wdiff` | bool | **Optional, layout test specific, deprecated.** Whether the web tests' output contains wdiff-formatted diffs for test failures. |
| `layout_tests_dir` | string | **Optional, layout test specific.** Path to the web_tests directory for the test run (used so that we can link to the tests used in the run). |
| `pixel_tests_enabled` | bool | **Optional, layout test specific.** Whether the web tests' were run with the --pixel-tests flag. |
| `fixable` | integer | **Optional, deprecated.** The number of tests that were run but were expected to fail. |
| `num_flaky` | integer | **Optional, deprecated.** The number of tests that were run more than once and produced different results each time. |
| `num_passes` | integer | **Optional, deprecated.** The number of successful tests; equivalent to `num_failures_by_type["Pass"]` |
@ -117,17 +117,17 @@ then is retried and passes, both `is_regression` and `is_unexpected` will be fal
| `is_unexpected` | bool | **Optional.** If present and true, the test result was unexpected. This might include an unexpected pass, i.e., it is not necessarily a regression. If false (or if the key is not present at all), the test produced the expected result. |
| `time` | float | **Optional.** If present, the time it took in seconds to execute the first invocation of the test. |
| `times` | array of floats | **Optional.** If present, the times in seconds of each invocation of the test. |
| `has_repaint_overlay` | bool | **Optional, layout test specific.** If present and true, indicates that the test output contains the data needed to draw repaint overlays to help explain the results (only used in layout tests). |
| `is_missing_audio` | bool | **Optional, layout test specific.** If present and true, the test was supposed to have an audio baseline to compare against, and we didn't find one. |
| `is_missing_text` | bool | **Optional, layout test specific.** If present and true, the test was supposed to have a text baseline to compare against, and we didn't find one. |
| `is_missing_video` | bool | **Optional, layout test specific.** If present and true, the test was supposed to have an image baseline to compare against and we didn't find one. |
| `is_testharness_test` | bool | **Optional, layout test specific.** If present, indicates that the layout test was written using the w3c's test harness and we don't necessarily have any baselines to compare against. |
| `reftest_type` | string | **Optional, layout test specific.** If present, one of `==` or `!=` to indicate that the test is a "reference test" and the results were expected to match the reference or not match the reference, respectively (only used in layout tests). |
| `has_repaint_overlay` | bool | **Optional, web test specific.** If present and true, indicates that the test output contains the data needed to draw repaint overlays to help explain the results (only used in layout tests). |
| `is_missing_audio` | bool | **Optional, we test specific.** If present and true, the test was supposed to have an audio baseline to compare against, and we didn't find one. |
| `is_missing_text` | bool | **Optional, web test specific.** If present and true, the test was supposed to have a text baseline to compare against, and we didn't find one. |
| `is_missing_video` | bool | **Optional, web test specific.** If present and true, the test was supposed to have an image baseline to compare against and we didn't find one. |
| `is_testharness_test` | bool | **Optional, web test specific.** If present, indicates that the layout test was written using the w3c's test harness and we don't necessarily have any baselines to compare against. |
| `reftest_type` | string | **Optional, web test specific.** If present, one of `==` or `!=` to indicate that the test is a "reference test" and the results were expected to match the reference or not match the reference, respectively (only used in layout tests). |
## Test result types
Any test may fail in one of several different ways. There are a few generic
types of failures, and the layout tests contain a few additional specialized
types of failures, and the web tests contain a few additional specialized
failure types.
| Result type | Description |
@ -137,15 +137,15 @@ failure types.
| `PASS` | The test ran as expected. |
| `SKIP` | The test was not run. |
| `TIMEOUT` | The test hung (did not complete) and was aborted. |
| `AUDIO` | **Layout test specific, deprecated.** The test is expected to produce audio output that doesn't match the expected result. Normally you will see `FAIL` instead. |
| `IMAGE` | **Layout test specific, deprecated.** The test produces image (and possibly text output). The image output doesn't match what we'd expect, but the text output, if present, does. Normally you will see `FAIL` instead. |
| `IMAGE+TEXT` | **Layout test specific, deprecated.** The test produces image and text output, both of which fail to match what we expect. Normally you will see `FAIL` instead. |
| `LEAK` | **Layout test specific, deprecated.** Memory leaks were detected during the test execution. |
| `MISSING` | **Layout test specific, deprecated.** The test completed but we could not find an expected baseline to compare against. |
| `NEEDSREBASELINE` | **Layout test specific, deprecated.** The expected test result is out of date and will be ignored (as above); the auto-rebaseline-bot will look for tests of this type and automatically update them. This should never show up as an `actual` result. |
| `REBASELINE` | **Layout test specific, deprecated.** The expected test result is out of date and will be ignored (any result other than a crash or timeout will be considered as passing). This test result should only ever show up on local test runs, not on bots (it is forbidden to check in a TestExpectations file with this expectation). This should never show up as an "actual" result. |
| `SLOW` | **Layout test specific, deprecated.** The test is expected to take longer than normal to run. This should never appear as an `actual` result, but may (incorrectly) appear in the expected fields. |
| `TEXT` | **Layout test specific, deprecated.** The test is expected to produce a text-only failure (the image, if present, will match). Normally you will see `FAIL` instead. |
| `AUDIO` | **Web test specific, deprecated.** The test is expected to produce audio output that doesn't match the expected result. Normally you will see `FAIL` instead. |
| `IMAGE` | **Web test specific, deprecated.** The test produces image (and possibly text output). The image output doesn't match what we'd expect, but the text output, if present, does. Normally you will see `FAIL` instead. |
| `IMAGE+TEXT` | **Web test specific, deprecated.** The test produces image and text output, both of which fail to match what we expect. Normally you will see `FAIL` instead. |
| `LEAK` | **Web test specific, deprecated.** Memory leaks were detected during the test execution. |
| `MISSING` | **Web test specific, deprecated.** The test completed but we could not find an expected baseline to compare against. |
| `NEEDSREBASELINE` | **Web test specific, deprecated.** The expected test result is out of date and will be ignored (as above); the auto-rebaseline-bot will look for tests of this type and automatically update them. This should never show up as an `actual` result. |
| `REBASELINE` | **Web test specific, deprecated.** The expected test result is out of date and will be ignored (any result other than a crash or timeout will be considered as passing). This test result should only ever show up on local test runs, not on bots (it is forbidden to check in a TestExpectations file with this expectation). This should never show up as an "actual" result. |
| `SLOW` | **Web test specific, deprecated.** The test is expected to take longer than normal to run. This should never appear as an `actual` result, but may (incorrectly) appear in the expected fields. |
| `TEXT` | **Web test specific, deprecated.** The test is expected to produce a text-only failure (the image, if present, will match). Normally you will see `FAIL` instead. |
Unexpected results, failures, and regressions are different things.
@ -161,7 +161,7 @@ A regression is a result that is both unexpected and a failure.
## `full_results.json` and `failing_results.json`
The layout tests produce two different variants of the above file. The
The web tests produce two different variants of the above file. The
`full_results.json` file matches the above definition and contains every test
executed in the run. The `failing_results.json` file contains just the tests
that produced unexpected results, so it is a subset of the `full_results.json`

2
docs/testing/using_breakpad_with_content_shell.md

@ -1,6 +1,6 @@
# Using breakpad with content shell
When running layout tests, it is possible to use
When running web tests, it is possible to use
[breakpad](../../third_party/breakpad/) to capture stack traces on crashes while
running without a debugger attached and with the sandbox enabled.

34
docs/testing/web_platform_tests.md

@ -10,7 +10,7 @@ test suite where all browser engines are collaborating.
Chromium has a 2-way import/export process with the upstream web-platform-tests
repository, where tests are imported into
[LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
[web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt)
and any changes to the imported tests are also exported to web-platform-tests.
See https://web-platform-tests.org/ for general documentation on
@ -22,7 +22,7 @@ web-platform-tests, including tips for writing and reviewing tests.
Chromium has a [mirror](https://chromium.googlesource.com/external/w3c/web-platform-tests/)
of the GitHub repo and periodically imports a subset of the tests to
run as part of the regular Blink layout test testing process.
run as part of the regular Blink web test testing process.
The goals of this process are to be able to run web-platform-tests unmodified
locally just as easily as we can run the Blink tests, and ensure that we are
@ -59,14 +59,14 @@ there are several possible causes, including:
Because these tests are imported from the Web Platform tests, it is better
to have them in the repository (and marked failing) than not, so prefer to
[add test expectations](layout_test_expectations.md) rather than reverting.
[add test expectations](web_test_expectations.md) rather than reverting.
However, if a huge number of tests are failing, please revert the CL so we
can fix it manually.
### Automatic export process
If you upload a CL with any changes in
[third_party/WebKit/LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt),
[third_party/blink/web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt),
once you add reviewers the exporter will create a provisional pull request with
those changes in the [upstream WPT GitHub repository](https://github.com/web-platform-tests/wpt/).
@ -101,7 +101,7 @@ For maintainers:
### Skipped tests
We control which tests are imported via a file called
[W3CImportExpectations](../../third_party/WebKit/LayoutTests/W3CImportExpectations),
[W3CImportExpectations](../../third_party/blink/web_tests/W3CImportExpectations),
which has a list of directories to skip while importing.
In addition to the directories and tests explicitly skipped there, tests may
@ -159,21 +159,21 @@ the script.
## Writing tests
To contribute changes to web-platform-tests, just commit your changes directly
to [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
to [web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt)
and the changes will be automatically upstreamed within 24 hours.
Changes involving adding, removing or modifying tests can all be upstreamed.
Any changes outside of
[external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be
[external/wpt](../../third_party/blink/web_tests/external/wpt) will not be
upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`,
will also not be upstreamed.
Running the layout tests will automatically regenerate MANIFEST.json to pick up
Running the web tests will automatically regenerate MANIFEST.json to pick up
any local modifications.
Most tests are written using testharness.js, see
[Writing Layout Tests](./writing_layout_tests.md) and
[Layout Tests Tips](./layout_tests_tips.md) for general guidelines.
[Writing Web Tests](./writing_web_tests.md) and
[Web Tests Tips](./web_tests_tips.md) for general guidelines.
### Write tests against specifications
@ -193,22 +193,22 @@ and what needs to be resolved to make it non-tentative.
[testdriver.js](https://web-platform-tests.org/writing-tests/testdriver.html)
provides a means to automate tests that cannot be written purely using web
platform APIs, similar to `internals.*` and `eventSender.*` in regular Blink
layout tests.
web tests.
If no testdriver.js API exists, check if it's a
[known issue](https://github.com/web-platform-tests/wpt/labels/testdriver.js)
and otherwise consider filing a new issue.
An alternative is to write manual tests that are automated with scripts from
[wpt_automation](../../third_party/WebKit/LayoutTests/external/wpt_automation).
[wpt_automation](../../third_party/blink/web_tests/external/wpt_automation).
Injection of JS in manual tests is determined by `loadAutomationScript` in
[testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js).
[testharnessreport.js](../../third_party/blink/web_tests/resources/testharnessreport.js).
Such tests still require case-by-case automation to run for other browser
engines, but are more valuable than purely manual tests.
Manual tests that have no automation are still imported, but skipped in
[NeverFixTests](../../third_party/WebKit/LayoutTests/NeverFixTests); see
[NeverFixTests](../../third_party/blink/web_tests/NeverFixTests); see
[issue 738489](https://crbug.com/738489).
### Adding new top-level directories
@ -244,15 +244,15 @@ https://web-platform-tests.org/appendix/github-intro.html.
## Running tests
Same as Blink layout tests, you can use
[`run_web_tests.py`](layout_tests.md#running-the-tests) to run any WPT test.
Same as Blink web tests, you can use
[`run_web_tests.py`](web_tests.md#running-the-tests) to run any WPT test.
One thing to note is that glob patterns for WPT tests are not yet supported.
## Reviewing tests
Anyone who can review code and tests in Chromium can also review changes in
[external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
[external/wpt](../../third_party/blink/web_tests/external/wpt)
that will be automatically upstreamed. There will be no additional review in
web-platform-tests as part of the export process.

12
docs/testing/layout_test_baseline_fallback.md → docs/testing/web_test_baseline_fallback.md

@ -1,8 +1,8 @@
# Layout Test Baseline Fallback
# Web Test Baseline Fallback
*** promo
Read [Layout Test Expectations and Baselines](layout_test_expectations.md) first
Read [Web Test Expectations and Baselines](web_test_expectations.md) first
if you have not.
***
@ -18,8 +18,8 @@ optimization and rebaselining.
## Terminology
* **Root directory**:
[`//src/third_party/WebKit/LayoutTests`](../../third_party/WebKit/LayoutTests)
is the root directory (of all the layout tests and baselines). All relative
[`//src/third_party/blink/web_tests`](../../third_party/blink/web_tests)
is the root directory (of all the web tests and baselines). All relative
paths in this document start from this directory.
* **Test name**: the name of a test is its relative path from the root
directory (e.g. `html/dom/foo/bar.html`).
@ -29,10 +29,10 @@ optimization and rebaselining.
`virtual/gpu/html/dom/foo/bar.html` is the virtual variant of
`html/dom/foo/bar.html` in the `gpu` suite. Only the latter file exists on
disk, and is called the base of the virtual test. See
[Layout Tests#Testing Runtime Flags](layout_tests.md#testing-runtime-flags)
[Web Tests#Testing Runtime Flags](web_tests.md#testing-runtime-flags)
for more details.
* **Platform directory**: each directory under
[`platform/`](../../third_party/WebKit/LayoutTests/platform) is a platform
[`platform/`](../../third_party/blink/web_tests/platform) is a platform
directory that contains baselines (no tests) for that platform. Directory
names are in the form of `PLATFORM-VERSION` (e.g. `mac-mac10.12`), except
for the latest version of a platform which is just `PLATFORM` (e.g. `mac`).

40
docs/testing/layout_test_expectations.md → docs/testing/web_test_expectations.md

@ -1,14 +1,14 @@
# Layout Test Expectations and Baselines
# Web Test Expectations and Baselines
The primary function of the LayoutTests is as a regression test suite; this
The primary function of the web tests is as a regression test suite; this
means that, while we care about whether a page is being rendered correctly, we
care more about whether the page is being rendered the way we expect it to. In
other words, we look more for changes in behavior than we do for correctness.
[TOC]
All layout tests have "expected results", or "baselines", which may be one of
All web tests have "expected results", or "baselines", which may be one of
several forms. The test may produce one or more of:
* A text file containing JavaScript log messages.
@ -16,12 +16,12 @@ several forms. The test may produce one or more of:
* A screen capture of the rendered page as a PNG file.
* WAV files of the audio output, for WebAudio tests.
For any of these types of tests, baselines are checked into the LayoutTests
For any of these types of tests, baselines are checked into the web_tests
directory. The filename of a baseline is the same as that of the corresponding
test, but the extension is replaced with `-expected.{txt,png,wav}` (depending on
the type of test output). Baselines usually live alongside tests, with the
exception when baselines vary by platforms; read
[Layout Test Baseline Fallback](layout_test_baseline_fallback.md) for more
[Web Test Baseline Fallback](web_test_baseline_fallback.md) for more
details.
Lastly, we also support the concept of "reference tests", which check that two
@ -49,10 +49,10 @@ lines to TestExpectations.
Bugs at [crbug.com](https://crbug.com) should track fixing incorrect behavior,
not lines in
[TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations). If a
[TestExpectations](../../third_party/blink/web_tests/TestExpectations). If a
test is never supposed to pass (e.g. it's testing Windows-specific behavior, so
can't ever pass on Linux/Mac), move it to the
[NeverFixTests](../../third_party/WebKit/LayoutTests/NeverFixTests) file. That
[NeverFixTests](../../third_party/blink/web_tests/NeverFixTests) file. That
gets it out of the way of the rest of the project.
There are some cases where you can't rebaseline and, unfortunately, we don't
@ -112,7 +112,7 @@ results from try jobs, by using the command-tool
This way, the new baselines can be reviewed along with the changes, which helps
the reviewer verify that the new baselines are correct. It also means that there
is no period of time when the layout test results are ignored.
is no period of time when the web test results are ignored.
#### Options
@ -133,36 +133,36 @@ depends on its arguments.
## Kinds of expectations files
* [TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations): The
* [TestExpectations](../../third_party/blink/web_tests/TestExpectations): The
main test failure suppression file. In theory, this should be used for
temporarily marking tests as flaky.
* [ASANExpectations](../../third_party/WebKit/LayoutTests/ASANExpectations):
* [ASANExpectations](../../third_party/blink/web_tests/ASANExpectations):
Tests that fail under ASAN.
* [LeakExpectations](../../third_party/WebKit/LayoutTests/LeakExpectations):
* [LeakExpectations](../../third_party/blink/web_tests/LeakExpectations):
Tests that have memory leaks under the leak checker.
* [MSANExpectations](../../third_party/WebKit/LayoutTests/MSANExpectations):
* [MSANExpectations](../../third_party/blink/web_tests/MSANExpectations):
Tests that fail under MSAN.
* [NeverFixTests](../../third_party/WebKit/LayoutTests/NeverFixTests): Tests
* [NeverFixTests](../../third_party/blink/web_tests/NeverFixTests): Tests
that we never intend to fix (e.g. a test for Windows-specific behavior will
never be fixed on Linux/Mac). Tests that will never pass on any platform
should just be deleted, though.
* [SlowTests](../../third_party/WebKit/LayoutTests/SlowTests): Tests that take
* [SlowTests](../../third_party/blink/web_tests/SlowTests): Tests that take
longer than the usual timeout to run. Slow tests are given 5x the usual
timeout.
* [SmokeTests](../../third_party/WebKit/LayoutTests/SmokeTests): A small subset
* [SmokeTests](../../third_party/blink/web_tests/SmokeTests): A small subset
of tests that we run on the Android bot.
* [StaleTestExpectations](../../third_party/WebKit/LayoutTests/StaleTestExpectations):
* [StaleTestExpectations](../../third_party/blink/web_tests/StaleTestExpectations):
Platform-specific lines that have been in TestExpectations for many months.
They're moved here to get them out of the way of people doing rebaselines
since they're clearly not getting fixed anytime soon.
* [W3CImportExpectations](../../third_party/WebKit/LayoutTests/W3CImportExpectations):
* [W3CImportExpectations](../../third_party/blink/web_tests/W3CImportExpectations):
A record of which W3C tests should be imported or skipped.
### Flag-specific expectations files
It is possible to handle tests that only fail when run with a particular flag
being passed to `content_shell`. See
[LayoutTests/FlagExpectations/README.txt](../../third_party/WebKit/LayoutTests/FlagExpectations/README.txt)
[web_tests/FlagExpectations/README.txt](../../third_party/blink/web_tests/FlagExpectations/README.txt)
for more.
## Updating the expectations files
@ -233,14 +233,14 @@ Bug(darin) [ Mac10.9 Debug ] fast/html/keygen.html [ Skip ]
* `WontFix` implies `Skip` and also indicates that we don't have any plans to
make the test pass.
* `WontFix` lines always go in the
[NeverFixTests file](../../third_party/WebKit/LayoutTests/NeverFixTests) as
[NeverFixTests file](../../third_party/blink/web_tests/NeverFixTests) as
we never intend to fix them. These are just for tests that only apply to some
subset of the platforms we support.
* `WontFix` and `Skip` must be used by themselves and cannot be specified
alongside `Crash` or another expectation keyword.
* `Slow` causes the test runner to give the test 5x the usual time limit to run.
`Slow` lines go in the
[SlowTests file ](../../third_party/WebKit/LayoutTests/SlowTests). A given
[SlowTests file ](../../third_party/blink/web_tests/SlowTests). A given
line cannot have both Slow and Timeout.
Also, when parsing the file, we use two rules to figure out if an expectation

104
docs/testing/layout_tests.md → docs/testing/web_tests.md

@ -1,32 +1,32 @@
# Layout Tests
# Web Tests (formly known as "Layout Tests" "LayoutTests")
Layout tests are used by Blink to test many components, including but not
limited to layout and rendering. In general, layout tests involve loading pages
Web tests are used by Blink to test many components, including but not
limited to layout and rendering. In general, web tests involve loading pages
in a test renderer (`content_shell`) and comparing the rendered output or
JavaScript output against an expected output file.
This document covers running and debugging existing layout tests. See the
[Writing Layout Tests documentation](./writing_layout_tests.md) if you find
yourself writing layout tests.
This document covers running and debugging existing web tests. See the
[Writing Web Tests documentation](./writing_web_tests.md) if you find
yourself writing web tests.
Note that we're in process of changing the term "layout tests" to "web tests".
Note that we changed the term "layout tests" to "web tests".
Please assume these terms mean the identical stuff. We also call it as
"WebKit tests" and "WebKit layout tests".
[TOC]
## Running Layout Tests
## Running Web Tests
### Initial Setup
Before you can run the layout tests, you need to build the `blink_tests` target
Before you can run the web tests, you need to build the `blink_tests` target
to get `content_shell` and all of the other needed binaries.
```bash
autoninja -C out/Default blink_tests
```
On **Android** (layout test support
On **Android** (web test support
[currently limited to KitKat and earlier](https://crbug.com/567947)) you need to
build and install `content_shell_apk` instead. See also:
[Android Build Instructions](../android_build_instructions.md).
@ -67,17 +67,17 @@ python third_party/blink/tools/run_web_tests.py -t android --android
```
Tests marked as `[ Skip ]` in
[TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations)
[TestExpectations](../../third_party/blink/web_tests/TestExpectations)
won't be run at all, generally because they cause some intractable tool error.
To force one of them to be run, either rename that file or specify the skipped
test as the only one on the command line (see below). Read the
[Layout Test Expectations documentation](./layout_test_expectations.md) to learn
[Web Test Expectations documentation](./web_test_expectations.md) to learn
more about TestExpectations and related files.
*** promo
Currently only the tests listed in
[SmokeTests](../../third_party/WebKit/LayoutTests/SmokeTests)
are run on the Android bots, since running all layout tests takes too long on
[SmokeTests](../../third_party/blink/web_tests/SmokeTests)
are run on the Android bots, since running all web tests takes too long on
Android (and may still have some infrastructure issues). Most developers focus
their Blink testing on Linux. We rely on the fact that the Linux and Android
behavior is nearly identical for scenarios outside those covered by the smoke
@ -85,8 +85,8 @@ tests.
***
To run only some of the tests, specify their directories or filenames as
arguments to `run_web_tests.py` relative to the layout test directory
(`src/third_party/WebKit/LayoutTests`). For example, to run the fast form tests,
arguments to `run_web_tests.py` relative to the web test directory
(`src/third_party/blink/web_tests`). For example, to run the fast form tests,
use:
```bash
@ -100,7 +100,7 @@ python third_party/blink/tools/run_web_tests.py fast/fo\*
```
*** promo
Example: To run the layout tests with a debug build of `content_shell`, but only
Example: To run the web tests with a debug build of `content_shell`, but only
test the SVG tests and run pixel tests, you would run:
```bash
@ -119,7 +119,7 @@ as in:
```bash
out/Default/content_shell.exe --run-web-tests --no-sandbox \
c:/chrome/src/third_party/WebKit/LayoutTests/fast/forms/001.html
c:/chrome/src/third_party/blink/web_tests/fast/forms/001.html
```
but this requires a manual diff against expected results, because the shell
@ -135,8 +135,8 @@ python run_web_tests.py --help
**Linux Note:** We try to match the Windows render tree output exactly by
matching font metrics and widget metrics. If there's a difference in the render
tree output, we should see if we can avoid rebaselining by improving our font
metrics. For additional information on Linux Layout Tests, please see
[docs/layout_tests_linux.md](../layout_tests_linux.md).
metrics. For additional information on Linux web tests, please see
[docs/web_tests_linux.md](../web_tests_linux.md).
***
*** note
@ -174,7 +174,7 @@ a diff of the actual and expected results, into
summary and link to the results/diffs.
The expected results for tests are in the
`src/third_party/WebKit/LayoutTests/platform` or alongside their respective
`src/third_party/blink/web_tests/platform` or alongside their respective
tests.
*** note
@ -190,14 +190,14 @@ A row of dots in the script's output indicates one or more tests that passed.
## Test expectations
The
[TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations) file (and related
files) contains the list of all known layout test failures. See the
[Layout Test Expectations documentation](./layout_test_expectations.md) for more
[TestExpectations](../../third_party/blink/web_tests/TestExpectations) file (and related
files) contains the list of all known web test failures. See the
[Web Test Expectations documentation](./web_test_expectations.md) for more
on this.
## Testing Runtime Flags
There are two ways to run layout tests with additional command-line arguments:
There are two ways to run web tests with additional command-line arguments:
* Using `--additional-driver-flag`:
@ -209,12 +209,12 @@ There are two ways to run layout tests with additional command-line arguments:
content_shell binary.
It will also look for flag-specific expectations in
`LayoutTests/FlagExpectations/blocking-repaint`, if this file exists. The
`web_tests/FlagExpectations/blocking-repaint`, if this file exists. The
suppressions in this file override the main TestExpectations file.
* Using a *virtual test suite* defined in
[LayoutTests/VirtualTestSuites](../../third_party/WebKit/LayoutTests/VirtualTestSuites).
A virtual test suite runs a subset of layout tests under a specific path with
[web_tests/VirtualTestSuites](../../third_party/blink/web_tests/VirtualTestSuites).
A virtual test suite runs a subset of web tests under a specific path with
additional flags. For example, you could test a (hypothetical) new mode for
repainting using the following virtual test suite:
@ -228,7 +228,7 @@ There are two ways to run layout tests with additional command-line arguments:
This will create new "virtual" tests of the form
`virtual/blocking_repaint/fast/repaint/...` which correspond to the files
under `LayoutTests/fast/repaint` and pass `--blocking-repaint` to
under `web_tests/fast/repaint` and pass `--blocking-repaint` to
content_shell when they are run.
These virtual tests exist in addition to the original `fast/repaint/...`
@ -269,7 +269,7 @@ Consider the following when choosing between them:
## Tracking Test Failures
All bugs, associated with layout test failures must have the
All bugs, associated with web test failures must have the
[Test-Layout](https://crbug.com/?q=label:Test-Layout) label. Depending on how
much you know about the bug, assign the status accordingly:
@ -279,7 +279,7 @@ much you know about the bug, assign the status accordingly:
* **Available** -- You know the root cause of the issue.
* **Assigned** or **Started** -- You will fix this issue.
When creating a new layout test bug, please set the following properties:
When creating a new web test bug, please set the following properties:
* Components: a sub-component of Blink
* OS: **All** (or whichever OS the failure is on)
@ -290,9 +290,9 @@ When creating a new layout test bug, please set the following properties:
You can also use the _Layout Test Failure_ template, which pre-sets these
labels for you.
## Debugging Layout Tests
## Debugging Web Tests
After the layout tests run, you should get a summary of tests that pass or
After the web tests run, you should get a summary of tests that pass or
fail. If something fails unexpectedly (a new regression), you will get a
`content_shell` window with a summary of the unexpected failures. Or you might
have a failing test in mind to investigate. In any case, here are some steps and
@ -310,7 +310,7 @@ tips for finding the problem.
ask.
* Some tests only work properly in content_shell, not Chrome, because they
rely on extra APIs exposed there.
* Some tests only work properly when they're run in the layout-test
* Some tests only work properly when they're run in the web-test
framework, not when they're loaded into content_shell directly. The test
should mention that in its visible text, but not all do. So try that too.
See "Running the tests", above.
@ -342,15 +342,15 @@ tips for finding the problem.
tracking down the issue.
* Otherwise, you're running in a standard message loop just like in Chrome.
If you have no other information, set a breakpoint on page load.
* If your test only works in full layout-test mode, or if you find it simpler to
* If your test only works in full web-test mode, or if you find it simpler to
debug without all the overhead of an interactive session, start the
content_shell with the command-line flag `--run-web-tests`, followed by the
URL (`file:` or `http:`) to your test. More information about running layout tests
in content_shell can be found [here](./layout_tests_in_content_shell.md).
URL (`file:` or `http:`) to your test. More information about running web tests
in content_shell can be found [here](./web_tests_in_content_shell.md).
* In VS, you can do this in the Debugging section of the content_shell
project Properties.
* Now you're running with exactly the same API, theme, and other setup that
the layout tests use.
the web tests use.
* Again, if your test contains a JS call, DOM manipulation, or other
distinctive piece of code that you think is failing, search for that in
the Chrome solution. That's a good place to put a starting breakpoint to
@ -358,7 +358,7 @@ tips for finding the problem.
* If you can't find any better place to set a breakpoint, start at the
`TestShell::RunFileTest()` call in `content_shell_main.cc`, or at
`shell->LoadURL() within RunFileTest()` in `content_shell_win.cc`.
* Debug as usual. Once you've gotten this far, the failing layout test is just a
* Debug as usual. Once you've gotten this far, the failing web test is just a
(hopefully) reduced test case that exposes a problem.
### Debugging HTTP Tests
@ -370,9 +370,9 @@ cd src/third_party/blink/tools
python run_blink_httpd.py
```
The layout tests are served from `http://127.0.0.1:8000/`. For example, to
The web tests are served from `http://127.0.0.1:8000/`. For example, to
run the test
`LayoutTest/http/tests/serviceworker/chromium/service-worker-allowed.html`,
`web_tests/http/tests/serviceworker/chromium/service-worker-allowed.html`,
navigate to
`http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
tests behave differently if you go to `127.0.0.1` vs. `localhost`, so use
@ -382,7 +382,7 @@ To kill the server, hit any key on the terminal where `run_blink_httpd.py` is
running, use `taskkill` or the Task Manager on Windows, or `killall` or
Activity Monitor on macOS.
The test server sets up an alias to the `LayoutTests/resources` directory. For
The test server sets up an alias to the `web_tests/resources` directory. For
example, in HTTP tests, you can access the testing framework using
`src="/js-test-resources/js-test.js"`.
@ -441,7 +441,7 @@ NOTE: If the test is an html file, this means it's a legacy test so you need to
## Bisecting Regressions
You can use [`git bisect`](https://git-scm.com/docs/git-bisect) to find which
commit broke (or fixed!) a layout test in a fully automated way. Unlike
commit broke (or fixed!) a web test in a fully automated way. Unlike
[bisect-builds.py](http://dev.chromium.org/developers/bisect-builds-py), which
downloads pre-built Chromium binaries, `git bisect` operates on your local
checkout, so it can run tests with `content_shell`.
@ -449,7 +449,7 @@ checkout, so it can run tests with `content_shell`.
Bisecting can take several hours, but since it is fully automated you can leave
it running overnight and view the results the next day.
To set up an automated bisect of a layout test regression, create a script like
To set up an automated bisect of a web test regression, create a script like
this:
```bash
@ -461,7 +461,7 @@ autoninja -C out/Debug -j100 blink_tests || exit 125
third_party/blink/tools/run_web_tests.py -t Debug \
--no-show-results --no-retry-failures \
path/to/layout/test.html
path/to/web/test.html
```
Modify the `out` directory, ninja args, and test name as appropriate, and save
@ -474,11 +474,11 @@ git bisect run ~/checkrev.sh
git bisect reset # quit the bisect session
```
## Rebaselining Layout Tests
## Rebaselining Web Tests
*** promo
To automatically re-baseline tests across all Chromium platforms, using the
buildbot results, see [How to rebaseline](./layout_test_expectations.md#How-to-rebaseline).
buildbot results, see [How to rebaseline](./web_test_expectations.md#How-to-rebaseline).
Alternatively, to manually run and test and rebaseline it on your workstation,
read on.
***
@ -488,7 +488,7 @@ cd src/third_party/blink
python tools/run_web_tests.py --reset-results foo/bar/test.html
```
If there are current expectation files for `LayoutTests/foo/bar/test.html`,
If there are current expectation files for `web_tests/foo/bar/test.html`,
the above command will overwrite the current baselines at their original
locations with the actual results. The current baseline means the `-expected.*`
file used to compare the actual result when the test is run locally, i.e. the
@ -497,7 +497,7 @@ first file found in the [baseline search path]
If there are no current baselines, the above command will create new baselines
in the platform-independent directory, e.g.
`LayoutTests/foo/bar/test-expected.{txt,png}`.
`web_tests/foo/bar/test-expected.{txt,png}`.
When you rebaseline a test, make sure your commit description explains why the
test is being re-baselined.
@ -513,7 +513,7 @@ python tools/run_web_tests.py --additional-driver-flag=--enable-flag --reset-res
```
New baselines will be created in the flag-specific baselines directory, e.g.
`LayoutTests/flag-specific/enable-flag/foo/bar/test-expected.{txt,png}`.
`web_tests/flag-specific/enable-flag/foo/bar/test-expected.{txt,png}`.
Then you can commit the new baselines and upload the patch for review.
@ -539,14 +539,14 @@ files. You can follow the steps below for easier review.
## web-platform-tests
In addition to layout tests developed and run just by the Blink team, there is
In addition to web tests developed and run just by the Blink team, there is
also a shared test suite, see [web-platform-tests](./web_platform_tests.md).
## Known Issues
See
[bugs with the component Blink>Infra](https://bugs.chromium.org/p/chromium/issues/list?can=2&q=component%3ABlink%3EInfra)
for issues related to Blink tools, include the layout test runner.
for issues related to Blink tools, include the web test runner.
* If QuickTime is not installed, the plugin tests
`fast/dom/object-embed-plugin-scripting.html` and

10
docs/testing/layout_tests_in_content_shell.md → docs/testing/web_tests_in_content_shell.md

@ -1,8 +1,8 @@
# Running layout tests using the content shell
# Running web tests using the content shell
## Basic usage
Layout tests can be run with `content_shell`. To just dump the render tree, use
Web tests can be run with `content_shell`. To just dump the render tree, use
the `--run-web-tests` flag:
```bash
@ -11,13 +11,13 @@ out/Default/content_shell --run-web-tests foo.html
### Compiling
If you want to run layout tests,
[build the target `blink_tests`](layout_tests.md); this includes all the other
If you want to run web tests,
[build the target `blink_tests`](web_tests.md); this includes all the other
binaries required to run the tests.
### Running
You can run layout tests using `run_web_tests.py` (in
You can run web tests using `run_web_tests.py` (in
`src/third_party/blink/tools`).
```bash

30
docs/testing/layout_tests_tips.md → docs/testing/web_tests_tips.md

@ -1,18 +1,18 @@
# Layout Tests Tips
# Web Tests Tips
The recommendations here are intended to help you write new tests that go
through code review with a minimal number of round trips, remain useful as Blink
evolves, and serve as an asset (rather than a liability) for the team.
While reading existing layout tests, please keep in mind that they represent
While reading existing web tests, please keep in mind that they represent
snapshots taken over many years of an ever-evolving collective opinion of what
good Web pages and solid tests should look like. Thus, it should not come as a
surprise that most existing layout tests are not consistent with these
surprise that most existing web tests are not consistent with these
recommendations, and are not even consistent with each other.
*** note
This document intentionally uses _should_ a lot more than _must_, as defined in
[RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a
[RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing web tests is a
careful act of balancing many concerns, and this humble document cannot possibly
capture the context that rests in the head of an experienced Blink engineer.
***
@ -51,7 +51,7 @@ test results less reliable.
### Fast
Tests should be as **fast** as possible, without compromising on the principles
below. Blink has several thousand layout tests that are run in parallel, and
below. Blink has several thousand web tests that are run in parallel, and
avoiding unnecessary delays is crucial to keeping our Commit Queue in good
shape.
@ -72,7 +72,7 @@ and the Commit Queue.
on a fast system, tests that rely on fixed timeouts can fail when on systems
that are slower than expected.
When adding or significantly modifying a layout test, use the command below to
When adding or significantly modifying a web test, use the command below to
assess its flakiness. While not foolproof, this approach gives you some
confidence, and giving up CPU cycles for mental energy is a pretty good trade.
@ -81,7 +81,7 @@ third_party/blink/tools/run_web_tests.py path/to/test.html --repeat-each=100
```
The
[PSA on writing reliable layout tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit).
[PSA on writing reliable web tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit).
also has good guidelines for writing reliable tests.
### Self-Describing
@ -92,7 +92,7 @@ feature being tested.
`testharness.js` makes a test self-describing when used correctly. Other types
of tests, such as reference tests and
[tests with manual fallback](./layout_tests_with_manual_fallback.md),
[tests with manual fallback](./web_tests_with_manual_fallback.md),
[must be carefully designed](https://web-platform-tests.org/writing-tests/manual.html#requirements-for-a-manual-test)
to be self-describing.
@ -179,9 +179,9 @@ the doctype.
## Coding Style
No coding style is enforced for layout tests. This section highlights coding
style aspects that are not consistent across our layout tests, and suggests some
defaults for unopinionated developers. When writing layout tests for a new part
No coding style is enforced for web tests. This section highlights coding
style aspects that are not consistent across our web tests, and suggests some
defaults for unopinionated developers. When writing web tests for a new part
of the codebase, you can minimize review latency by taking a look at existing
tests, and pay particular attention to these issues. Also beware of per-project
style guides, such as the
@ -192,7 +192,7 @@ style guides, such as the
[Google's JavaScript Style Guide](https://google.github.io/styleguide/jsguide.html)
and
[Google's HTML/CSS Style Guide](https://google.github.io/styleguide/htmlcssguide.xml)
are a reasonable baseline for coding style defaults, with the caveat that layout
are a reasonable baseline for coding style defaults, with the caveat that web
tests do not use Google Closure or JSDoc.
### == vs ===
@ -273,7 +273,7 @@ more familiar to developers coming from other programming languages.
A good default is to prefer classes over other OOP constructs, as they will make
the code easier to read for many of your fellow Chrome developers. At the same
time, most layout tests are simple enough that OOP is not justified.
time, most web tests are simple enough that OOP is not justified.
### Character Encoding
@ -286,7 +286,7 @@ browser-specific heuristics.
The easiest way to not have to think about any of this is to add
`<meta charset="utf-8">` to all your tests. This is easier to remember if you
use a template for your layout tests, rather than writing them from scratch.
use a template for your web tests, rather than writing them from scratch.
## Tests with Manual Feedback
@ -296,6 +296,6 @@ having the tests gracefully degrade to manual tests in the absence of the
testing APIs.
The
[document on layout tests with manual feedback](./layout_tests_with_manual_fallback.md)
[document on web tests with manual feedback](./web_tests_with_manual_fallback.md)
describes the approach in detail and highlights the trade-off between added test
weight and ease of debugging.

8
docs/testing/layout_tests_with_manual_fallback.md → docs/testing/web_tests_with_manual_fallback.md

@ -1,4 +1,4 @@
# Layout Tests with Manual Fallback
# Web Tests with Manual Fallback
Some Blink features cannot be automatically tested using the Web Platform. Prime
examples are the APIs that require
@ -17,7 +17,7 @@ when loaded in an environment that exposes the needed testing APIs.
## Motivation
Layout tests that degrade to manual tests in the absence of testing APIs have
Web tests that degrade to manual tests in the absence of testing APIs have
the following benefits.
* The manual test component can be debugged in a normal browser session, using
@ -25,7 +25,7 @@ the following benefits.
without a manual fallback can only be debugged in the test runner.
* The manual tests can run in other browsers, making it easy to check whether
our behavior matches other browsers.
* The layout tests can form the basis for manual tests that are contributed to
* The web tests can form the basis for manual tests that are contributed to
[web-platform-tests](./web_platform_tests.md).
Therefore, the desirability of adding a manual fallback to a test heavily
@ -36,7 +36,7 @@ and the size and complexity introduced by the manual fallback.
## Development Tips
A natural workflow for writing a layout test that gracefully degrades to a
A natural workflow for writing a web test that gracefully degrades to a
manual test is to first develop the manual test in a browser, and then add code
that feature-checks for testing APIs, and uses them to automate the test's
manual steps.

84
docs/testing/writing_layout_tests.md → docs/testing/writing_web_tests.md

@ -1,16 +1,10 @@
# Writing Layout Tests
_Layout tests_ is a bit of a misnomer. This term is
[a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory/),
and we use it to refer to every test that is written as a Web page (HTML, SVG,
or XHTML) and lives in
[third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests).
# Writing Web Tests
[TOC]
## Overview
Layout tests should be used to accomplish one of the following goals:
Web tests should be used to accomplish one of the following goals:
1. The entire surface of Blink that is exposed to the Web should be covered by
tests that we contribute to [web-platform-tests](./web_platform_tests.md)
@ -22,12 +16,12 @@ Layout tests should be used to accomplish one of the following goals:
2. When a Blink feature cannot be tested using the tools provided by WPT, and
cannot be easily covered by
[C++ unit tests](https://cs.chromium.org/chromium/src/third_party/blink/renderer/web/tests/?q=webframetest&sq=package:chromium&type=cs),
the feature must be covered by layout tests, to avoid unexpected regressions.
the feature must be covered by web tests, to avoid unexpected regressions.
These tests will use Blink-specific testing APIs that are only available in
[content_shell](./layout_tests_in_content_shell.md).
[content_shell](./web_tests_in_content_shell.md).
*** promo
If you know that Blink layout tests are upstreamed to other projects, such as
If you know that Blink web tests are upstreamed to other projects, such as
[test262](https://github.com/tc39/test262), please update this document. Most
importantly, our guidelines should to make it easy for our tests to be
upstreamed. The
@ -38,9 +32,9 @@ repositories.
### Test Types
There are four broad types of layout tests, listed in the order of preference.
There are four broad types of web tests, listed in the order of preference.
* *JavaScript Tests* are the layout test implementation of
* *JavaScript Tests* are the web test implementation of
[xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain
assertions written in JavaScript, and pass if the assertions evaluate to
true.
@ -57,7 +51,7 @@ There are four broad types of layout tests, listed in the order of preference.
system settings. For this reason, it is common for a pixel test to have a
different reference image for each platform that Blink is tested on, and
the reference images are
[quite cumbersome to manage](./layout_test_expectations.md). You
[quite cumbersome to manage](./web_test_expectations.md). You
should only write a pixel test if you cannot use a reference test.
* *Text Tests* output pure text which represents the DOM tree, the DOM inner
text, internal data structure of Blink like layout tree or graphics layer
@ -84,11 +78,11 @@ to the WPT project. To this end, tests should follow the
[WPT guidelines](https://web-platform-tests.org/writing-tests/).
There is no style guide that applies to all layout tests. However, some projects
There is no style guide that applies to all web tests. However, some projects
have adopted style guides, such as the
[ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/testing).
Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most
Our [document on web tests tips](./web_tests_tips.md) summarizes the most
important WPT guidelines and highlights some JavaScript concepts that are worth
paying attention to when trying to infer style rules from existing tests. If
you're unopinionated and looking for a style guide to follow, the document also
@ -110,8 +104,8 @@ are more accessible to browser developers.
See the [API documentation](https://web-platform-tests.org/writing-tests/testharness-api.html)
for a thorough introduction to `testharness.js`.
Layout tests should follow the recommendations of the above documentation.
Furthermore, layout tests should include relevant
Web tests should follow the recommendations of the above documentation.
Furthermore, web tests should include relevant
[metadata](https://web-platform-tests.org/writing-tests/css-metadata.html). The
specification URL (in `<link rel="help">`) is almost always relevant, and is
incredibly helpful to a developer who needs to understand the test quickly.
@ -190,14 +184,14 @@ Some points that are not immediately obvious from the example:
`t.done()` call.
*** promo
Layout tests that load from `file://` origins must currently use relative paths
Web tests that load from `file://` origins must currently use relative paths
to point to
[/resources/testharness.js](../../third_party/WebKit/LayoutTests/resources/testharness.js)
[/resources/testharness.js](../../third_party/blink/web_tests/resources/testharness.js)
and
[/resources/testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js).
[/resources/testharnessreport.js](../../third_party/blink/web_tests/resources/testharnessreport.js).
This is contrary to the WPT guidelines, which call for absolute paths.
This limitation does not apply to the tests in `LayoutTests/http`, which rely on
an HTTP server, or to the tests in `LayoutTests/external/wpt`, which are
This limitation does not apply to the tests in `web_tests/http`, which rely on
an HTTP server, or to the tests in `web_tests/external/wpt`, which are
imported from the [WPT repository](https://github.com/web-platform-tests/wpt).
***
@ -210,7 +204,7 @@ via supplemental testing APIs.
When writing tests that rely on supplemental testing APIs, please consider the
cost and benefits of having the tests
[gracefully degrade to manual tests](./layout_tests_with_manual_fallback.md) in
[gracefully degrade to manual tests](./web_tests_with_manual_fallback.md) in
the absence of the testing APIs.
*** promo
@ -223,7 +217,7 @@ event handling tests can use
Tests that cannot be expressed using the Web Platform APIs or WPT's testing APIs
use Blink-specific testing APIs. These APIs are only available in
[content_shell](./layout_tests_in_content_shell.md), and should only be used as
[content_shell](./web_tests_in_content_shell.md), and should only be used as
a last resort.
A downside of Blink-specific APIs is that they are not as well documented as the
@ -261,7 +255,7 @@ APIs, we should use the qualified name in the `if` statement:
by tests that stick to Web Platform APIs. The `testharnessreport.js` file in
`testharness.js` is specifically designated to hold glue code that connects
`testharness.js` to the testing environment. Our implementation is in
[third_party/WebKit/LayoutTests/resources/testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js),
[third_party/blink/web_tests/resources/testharnessreport.js](../../third_party/blink/web_tests/resources/testharnessreport.js),
and uses the `testRunner` API.
***
@ -288,8 +282,8 @@ In these situations, a test file will be accompanied by a baseline, which is an
`-expected.txt` file that contains the test's expected output.
The baselines are generated automatically when appropriate by
`run_web_tests.py`, which is described [here](./layout_tests.md), and by the
[rebaselining tools](./layout_test_expectations.md).
`run_web_tests.py`, which is described [here](./web_tests.md), and by the
[rebaselining tools](./web_test_expectations.md).
Text baselines for `testharness.js` should be avoided, as having a text baseline
associated with a `testharness.js` indicates the presence of a bug. For this
@ -301,7 +295,7 @@ text expectations.
behavior does not match the specification that is being tested, a text
baseline is necessary. Remember to create an issue tracking the expectation's
removal, and to link the issue in the CL description.
* Layout tests that cannot be upstreamed to WPT should use JavaScript to
* Web tests that cannot be upstreamed to WPT should use JavaScript to
document Blink's current behavior, rather than using JavaScript to document
desired behavior and a text file to document current behavior.
@ -314,7 +308,7 @@ This harness is **deprecated**, and should not be used for new tests.
If you need to understand old tests, the best `js-test` documentation is its
implementation at
[third_party/WebKit/LayoutTests/resources/js-test.js](../../third_party/WebKit/LayoutTests/resources/js-test.js).
[third_party/blink/web_tests/resources/js-test.js](../../third_party/blink/web_tests/resources/js-test.js).
`js-test` tests lean heavily on the Blink-specific `testRunner` testing API.
In a nutshell, the tests call `testRunner.dumpAsText()` to signal that the page
@ -329,7 +323,7 @@ By default, tests are loaded as if via `file:` URLs. Some web platform features
require tests served via HTTP or HTTPS, for example absolute paths (`src=/foo`)
or features restricted to secure protocols.
HTTP tests are those under `LayoutTests/http/tests` (or virtual variants). Use a
HTTP tests are those under `web_tests/http/tests` (or virtual variants). Use a
locally running HTTP server (Apache) to run them. Tests are served off of ports
8000 and 8080 for HTTP, and 8443 for HTTPS. If you run the tests using
`run_web_tests.py`, the server will be started automatically. To run the server
@ -340,7 +334,7 @@ cd src/third_party/blink/tools
./run_blink_httpd.py
```
The layout tests will be served from `http://127.0.0.1:8000`. For example, to
The web tests will be served from `http://127.0.0.1:8000`. For example, to
run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`,
navigate to
`http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
@ -351,12 +345,12 @@ To kill the server, hit any key on the terminal where `run_blink_httpd.py` is
running, or just use `taskkill` or the Task Manager on Windows, and `killall` or
Activity Monitor on MacOS.
The test server sets up an alias to the `LayoutTests/resources` directory. In
The test server sets up an alias to the `web_tests/resources` directory. In
HTTP tests, you can access the testing framework at e.g.
`src="/resources/testharness.js"`.
TODO: Document [wptserve](http://wptserve.readthedocs.io/) when we are in a
position to use it to run layout tests.
position to use it to run web tests.
## Reference Tests (Reftests)
@ -470,7 +464,7 @@ The recommendation of using Ahem in pixel tests is being discussed on
[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
***
The following snippet includes the Ahem font in a layout test.
The following snippet includes the Ahem font in a web test.
```html
<style>
@ -482,31 +476,31 @@ body {
```
*** promo
Tests outside `LayoutTests/http` and `LayoutTests/external/wpt` currently need
Tests outside `web_tests/http` and `web_tests/external/wpt` currently need
to use a relative path to
[/third_party/WebKit/LayoutTests/resources/ahem.js](../../third_party/WebKit/LayoutTests/resources/ahem.js)
[/third_party/blink/web_tests/resources/ahem.js](../../third_party/blink/web_tests/resources/ahem.js)
***
### Tests that need to paint, raster, or draw a frame of intermediate output
A layout test does not actually draw frames of output until the test exits.
A web test does not actually draw frames of output until the test exits.
Tests that need to generate a painted frame can use `runAfterLayoutAndPaint()`
defined in [third_party/WebKit/LayoutTests/resources/run-after-layout-and-paint.js](../../third_party/WebKit/LayoutTests/resources/run-after-layout-and-paint.js)
defined in [third_party/blink/web_tests/resources/run-after-layout-and-paint.js](../../third_party/blink/web_tests/resources/run-after-layout-and-paint.js)
which will run the machinery to put up a frame, then call the passed callback.
There is also a library at
[third_party/WebKit/LayoutTests/paint/invalidation/resources/text-based-repaint.js](../../third_party/WebKit/LayoutTests/paint/invalidation/resources/text-based-repaint.js)
[third_party/blink/web_tests/paint/invalidation/resources/text-based-repaint.js](../../third_party/blink/web_tests/paint/invalidation/resources/text-based-repaint.js)
to help with writing paint invalidation and repaint tests.
### Tests for scrolling animations
Some layout tests need to ensure animations such as middle-click auto-scroll,
Some web tests need to ensure animations such as middle-click auto-scroll,
fling, etc. get performed properly. When testing in display compositor pixel
dump mode (now the standard), the standard behavior for tests is to
synchronously composite without rastering (to save time). However, animations
run upon surface activation, which only happens once rasterization is performed.
Therefore, for these tests, an additional setting needs to be set. Near the
beginning of these tests, call `setAnimationRequiresRaster()` defined in
[third_party/WebKit/LayoutTests/resources/compositor-controls.js](../../third_party/WebKit/LayoutTests/resources/compositor-controls.js)
[third_party/blink/web_tests/resources/compositor-controls.js](../../third_party/blink/web_tests/resources/compositor-controls.js)
which will enable full rasterization during the test.
## Text tests
@ -629,7 +623,7 @@ For a test that is both a pixel/reference test and a text test, both pixel and
text results will be compared to baselines, and the test passes if each result
matches the corresponding baseline.
Many of the [paint invalidation tests](../../third_party/WebKit/LayoutTests/paint/invalidation)
Many of the [paint invalidation tests](../../third_party/blink/web_tests/paint/invalidation)
are of this type. The pixel results (compared against `-expected.png` or
`-expected.html`) ensure correct rendering, and the text results (compared
against `-expected.txt`) ensure correct compositing and raster invalidation
@ -643,7 +637,7 @@ which outputs you really care about.
## Directory Structure
The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently
The [web_tests directory](../../third_party/blink/web_tests) currently
lacks a strict, formal structure. The following directories have special
meaning:
@ -652,7 +646,7 @@ meaning:
as media files, and code that is shared by multiple test files.
*** note
Some layout tests consist of a minimal HTML page that references a JavaScript
Some web tests consist of a minimal HTML page that references a JavaScript
file in `resources/`. Please do not use this pattern for new tests, as it goes
against the minimality principle. JavaScript and CSS files should only live in
`resources/` if they are shared by at least two test files.

10
docs/using_a_linux_chroot.md

@ -1,6 +1,6 @@
# Using a chroot
If you want to run layout tests and you're not running Lucid, you'll get errors
If you want to run web tests and you're not running Lucid, you'll get errors
due to version differences in libfreetype. To work around this, you can use a
chroot.
@ -65,16 +65,16 @@ fi
### Running X apps
I also have `DISPLAY=:0` in my `$debian_chroot` section so I can run test_shell
or layout tests without manually setting my display every time. Your display
or web tests without manually setting my display every time. Your display
number may vary (`echo $DISPLAY` outside the chroot to see what your display
number is).
You can also use `Xvfb` if you only want to
[run tests headless](layout_tests_linux.md#Using-an-embedded-X-server).
[run tests headless](web_tests_linux.md#Using-an-embedded-X-server).
### Having layout test results open in a browser
### Having web test results open in a browser
After running layout tests, you should get a new browser tab or window that
After running web tests, you should get a new browser tab or window that
opens results.html. If you get an error "Failed to open
file:///path/to/results.html, check the
following conditions.

24
docs/layout_tests_linux.md → docs/web_tests_linux.md

@ -1,16 +1,16 @@
# Running layout tests on Linux
# Running web tests on Linux
1. Build `blink_tests` (see [Linux-specific build instructions](https://chromium.googlesource.com/chromium/src/+/master/docs/linux_build_instructions.md))
1. Checkout the layout tests
1. Checkout the web tests
* If you have an entry in your `.gclient` file that includes
"LayoutTests", you may need to comment it out and sync.
"web_tests", you may need to comment it out and sync.
* You can run a subset of the tests by passing in a path relative to
`src/third_party/WebKit/LayoutTests/`. For example,
`src/third_party/blink/web_tests/`. For example,
`third_party/blink/tools/run_web_tests.py fast` will only run the tests under
`src/third_party/WebKit/LayoutTests/fast/`.
`src/third_party/blink/web_tests/fast/`.
1. When the tests finish, any unexpected results should be displayed.
See [Layout Tests](testing/layout_tests.md)
See [Web Tests](testing/web_tests.md)
for full documentation about set up and available options.
## Pixel Tests
@ -61,15 +61,15 @@ If `fast/dom/object-plugin-hides-properties.html` and
set up a Lucid 64 chroot. Learn more about
[using a linux chroot](using_a_linux_chroot.md).
## Getting a layout test into a debugger
## Getting a web test into a debugger
There are two ways:
1. Run `content_shell` directly rather than using `run_web_tests.py`. You
will need to pass some options:
* `--no-timeout` to give you plenty of time to debug
* the fully qualified path of the layout test (rather than relative to
`WebKit/LayoutTests`).
* the fully qualified path of the web test (rather than relative to
`blink/web_tests`).
1. Or, run as normal but with the
`--additional-drt-flag=--renderer-startup-dialog
--additional-drt-flag=--no-timeout --time-out-ms=86400000` flags. The first
@ -86,7 +86,7 @@ create a separate X server for running the tests.
1. Install Xephyr (`sudo apt-get install xserver-xephyr`)
1. Start Xephyr as display 4: `Xephyr :4 -screen 1024x768x24`
1. Run the layout tests in the Xephyr: `DISPLAY=:4 run_web_tests.py`
1. Run the web tests in the Xephyr: `DISPLAY=:4 run_web_tests.py`
Xephyr supports debugging repainting. See the
[Xephyr README](http://cgit.freedesktop.org/xorg/xserver/tree/hw/kdrive/ephyr/README)
@ -99,11 +99,11 @@ If you don't want to see anything at all, you can use Xvfb (should already be
installed).
1. Start Xvfb as display 4: `Xvfb :4 -screen 0 1024x768x24`
1. Run the layout tests in the Xvfb: `DISPLAY=:4 run_web_tests.py`
1. Run the web tests in the Xvfb: `DISPLAY=:4 run_web_tests.py`
## Tiling Window managers
The layout tests want to run with the window at a particular size down to the
The web tests want to run with the window at a particular size down to the
pixel level. This means if your window manager resizes the window it'll cause
test failures. This is another good reason to use an embedded X server.

2
third_party/blink/renderer/core/animation/README.md vendored

@ -341,7 +341,7 @@ pointers for how to get started. If Chrome does not correctly implement the
spec, add a corresponding -expected.txt file with your test listing the expected
failure in Chrome.
[Layout tests](../../../../../docs/testing/writing_layout_tests.md) are located
[Web tests](../../../../../docs/testing/writing_web_tests.md) are located
in [third_party/blink/web_tests][]. These should be written when needing end
to end testing but either when testing chrome specific features (i.e.
non-standardized) such as compositing or when the test requires access to chrome

2
third_party/blink/renderer/devtools/readme.md vendored

@ -91,7 +91,7 @@ npm test -- -f --child-processes=16
npm test -- --time-out-ms=6000000 <test_path>
```
> **Tip**: [Learn about the test harness flags](https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_tests.md#Test-Harness-Options)
> **Tip**: [Learn about the test harness flags](https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_tests.md#Test-Harness-Options)
#### `--fetch-content-shell`
```

2
third_party/blink/tools/blinkpy/web_tests/run_webkit_tests.py vendored

@ -499,7 +499,7 @@ def parse_args(args):
option_parser = optparse.OptionParser(
prog='run_web_tests.py',
usage='%prog [options] [tests]',
description='Runs Blink layout tests as described in docs/testing/layout_tests.md')
description='Runs Blink layout tests as described in docs/testing/web_tests.md')
for group_name, group_options in option_group_definitions:
option_group = optparse.OptionGroup(option_parser, group_name)

2
third_party/blink/web_tests/ASANExpectations vendored

@ -1,7 +1,7 @@
# This file is used for tests that only need to be suppressed on
# the Chromium WebKit ASAN bots.
# Further documentation:
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
# NULL deref in libflashplayer.so, http://crbug.com/145940
# ==12527== ERROR: AddressSanitizer: SEGV on unknown address 0x7f628058b38b

2
third_party/blink/web_tests/LeakExpectations vendored

@ -1,7 +1,7 @@
# This file is used for tests that only need to be suppressed on the Chromium
# WebKit Leak bots.
# Further documentation:
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
###########################################################################
# WARNING: Memory leaks must be fixed asap. Sheriff is expected to revert #

2
third_party/blink/web_tests/MSANExpectations vendored

@ -1,7 +1,7 @@
# This file is used for tests that only need to be suppressed on
# the Chromium Blink MSan bots.
# Further documentation:
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
# Those stress V8's simulator, making them very slow under MSan.
crbug.com/420198 [ Linux ] fast/js/regress [ Skip ]

2
third_party/blink/web_tests/NeverFixTests vendored

@ -8,7 +8,7 @@
# 6. Tests that fail due to Debug rendering variations from Release behavior.
# 7. Tests that are not meant to be run automatically (manual tests).
# 8. Tests that are automatically imported but not intended to be run.
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
# FIXME: Remove the WontFix keyword and have that be implied for tests in this file.

2
third_party/blink/web_tests/SlowTests vendored

@ -6,7 +6,7 @@
# than 2 seconds in Release mode or 6 seconds in Debug mode should be listed here.
#
# Further documentation:
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
crbug.com/24182 storage/indexeddb/objectstore-cursor.html [ Slow ]
crbug.com/24182 storage/indexeddb/mozilla/test_objectStore_openKeyCursor.html [ Slow ]

2
third_party/blink/web_tests/StaleTestExpectations vendored

@ -5,7 +5,7 @@
# Feel free to take ownership of any of these lines and get the test passing.
#
# Further documentation:
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
# We could fix this test for us and upstream it if the test shell user agent
# would let us differentiate test_shell and WebKit DumpTreeNode.

2
third_party/blink/web_tests/TestExpectations vendored

@ -1,6 +1,6 @@
# This is the main failure suppression file for Blink LayoutTests.
# Further documentation:
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
# Intentional failures to test the layout test system.
Bug(intentional) harness-tests/crash.html [ Crash ]

2
third_party/blink/web_tests/W3CImportExpectations vendored

@ -7,7 +7,7 @@
# used to make it easier to work on getting them running and passing.
#
# Further documentation:
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
external/wpt/.codecov.yml [ Skip ]
external/wpt/.gitmodules [ Skip ]

10
third_party/blink/web_tests/accessibility/readme.md vendored

@ -1,13 +1,13 @@
# LayoutTests for Accessibility
# Web Tests for Accessibility
## General Info on LayoutTests: Building and Running the Tests
## General Info on web tests: Building and Running the Tests
See [Layout Tests](/docs/testing/layout_tests.md) for general
info on how to build and run layout tests.
See [Web Tests](/docs/testing/web_tests.md) for general
info on how to build and run web tests.
## Old vs. New
There are two styles of accessibility layout tests:
There are two styles of accessibility web tests:
* Using a ```-expected.txt``` (now deprecated)
* Unit-style tests with assertions

4
third_party/blink/web_tests/fast/harness/results.html vendored

@ -232,7 +232,7 @@ p, h2, h3, h4 {
<h4>2. TestExpectations</a></h4>
<pre><a href="#">crbug.com/bug</a> layout/test/path/<a href="#">test.html</a> [ Status ]</pre>
<p>TestExpectationsshow lines as they'd appear in <a
href="https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md">TestExpectations</a> file.</p>
href="https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md">TestExpectations</a> file.</p>
<p>The interesting part here is [ Status ]. Inside TestExpectations file, [ Status ]
can have multiple values, representing all expected results. For example:</p>
@ -548,7 +548,7 @@ const Report = {
printExpectation: (test, path, traversal) => {
// TestExpectations file format is documented at:
// https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
// https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
let pathParser = new PathParser(path);
// Print directory header if this test's directory is different from the last.

2
third_party/blink/web_tests/http/tests/websocket/throttling.html vendored

@ -14,7 +14,7 @@
// These tests will not function correctly without the --run-web-tests
// option. To debug them interactively you can follow the instructions for a
// "legacy test" in
// https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_tests.md#Debugging-DevTools-Tests.
// https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_tests.md#Debugging-DevTools-Tests.
async_test(t => {
const startTime = Date.now();

2
third_party/blink/web_tests/resources/compositor-controls.js vendored

@ -1,6 +1,6 @@
// To make sure animation events get processed, periodic lifecycle phase runs
// will include rasterization. See
// https://chromium.googlesource.com/chromium/src/+/master/docs/testing/writing_layout_tests.md
// https://chromium.googlesource.com/chromium/src/+/master/docs/testing/writing_web_tests.md
// for more information.
function setAnimationRequiresRaster() {
if (window.testRunner) {