Update "LayoutTests" / "layout test" in docs/*.md.
This CL renames the following md files: - layout_tests_linux -> web_tests_linux.md - how_to_extend_layout_test_framework.md -> how_to_extend_web_test_framework.md - testing/layout_test_baseline_fallback.md -> testing/web_test_baseline_fallback.md - testing/layout_test_expectations.md -> testing/web_test_expectations.md - testing/layout_tests.md -> testing/web_tests.md - testing/layout_tests_in_content_shell.md -> testing/web_tests_in_content_shell.md - testing/layout_tests_tips.md -> testing/web_tests_tips.md - testing/layout_tests_with_manual_fallback.md -> testing/web_tests_with_manual_fallback.md - testing/writing_layout_tests -> testing/writing_web_tests.md No-Try: true Change-Id: Ied34815c92e449888b20818540a5ed0a76107bd1 Bug: 843412 Reviewed-on: https://chromium-review.googlesource.com/c/1350559 Commit-Queue: Kent Tamura <tkent@chromium.org> Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> Cr-Commit-Position: refs/heads/master@{#611024}
This commit is contained in:
docs
README.md
accessibility
android_test_instructions.mdatom.mdhow_to_add_your_feature_flag.mdhow_to_extend_web_test_framework.mdinfra
linux_build_instructions.mdlinux_debugging.mdlinux_eclipse_dev.mdorigin_trials_integration.mdservicification.mdspeed
benchmark
harnesses
testing
identifying_tests_that_depend_on_order.mdjson_test_results_format.mdusing_breakpad_with_content_shell.mdweb_platform_tests.mdweb_test_baseline_fallback.mdweb_test_expectations.mdweb_tests.mdweb_tests_in_content_shell.mdweb_tests_tips.mdweb_tests_with_manual_fallback.mdwriting_web_tests.md
using_a_linux_chroot.mdweb_tests_linux.mdthird_party/blink
renderer
tools
blinkpy
web_tests
web_tests
ASANExpectationsLeakExpectationsMSANExpectationsNeverFixTestsSlowTestsStaleTestExpectationsTestExpectationsW3CImportExpectations
accessibility
fast
harness
http
tests
websocket
resources
@@ -145,20 +145,19 @@ used when committed.
|
|||||||
work.
|
work.
|
||||||
|
|
||||||
### Testing
|
### Testing
|
||||||
* [Running and Debugging Layout Tests](testing/layout_tests.md)
|
* [Running and Debugging Web Tests](testing/web_tests.md)
|
||||||
* [Writing Layout Tests](testing/writing_layout_tests.md) - Layout Tests using
|
* [Writing Web Tests](testing/writing_web_tests.md) - Web Tests using
|
||||||
`content_shell`
|
`content_shell`
|
||||||
* [Layout Test Expectations and Baselines](testing/layout_test_expectations.md) -
|
* [Web Test Expectations and Baselines](testing/web_test_expectations.md) -
|
||||||
Setting expected results of layout tests.
|
Setting expected results of web tests.
|
||||||
* [Layout Tests Tips](testing/layout_tests_tips.md) - Best practices for Layout
|
* [Web Tests Tips](testing/web_tests_tips.md) - Best practices for web tests
|
||||||
Tests
|
* [Web Tests with Manual Fallback](testing/webt_tests_with_manual_fallback.md) -
|
||||||
* [Layout Tests with Manual Fallback](testing/layout_tests_with_manual_fallback.md) -
|
|
||||||
Writing tests that simulate manual interventions
|
Writing tests that simulate manual interventions
|
||||||
* [Extending the Layout Test Framework](how_to_extend_layout_test_framework.md)
|
* [Extending the Web Test Framework](how_to_extend_web_test_framework.md)
|
||||||
* [Fixing Layout Test Flakiness](testing/identifying_tests_that_depend_on_order.md) -
|
* [Fixing Web Test Flakiness](testing/identifying_tests_that_depend_on_order.md) -
|
||||||
Diagnosing and fixing layout test flakiness due to ordering dependencies.
|
Diagnosing and fixing web test flakiness due to ordering dependencies.
|
||||||
* [Running Layout Tests using `content_shell`](testing/layout_tests_in_content_shell.md) -
|
* [Running Web Tests using `content_shell`](testing/web_tests_in_content_shell.md) -
|
||||||
Running layout tests by hand.
|
Running web tests by hand.
|
||||||
* [Web Platform Tests](testing/web_platform_tests.md) - Shared tests across
|
* [Web Platform Tests](testing/web_platform_tests.md) - Shared tests across
|
||||||
browser vendors
|
browser vendors
|
||||||
* [Using Breakpad with `content_shell`](testing/using_breakpad_with_content_shell.md) -
|
* [Using Breakpad with `content_shell`](testing/using_breakpad_with_content_shell.md) -
|
||||||
@@ -217,8 +216,8 @@ used when committed.
|
|||||||
isolate browser components with different privileges.
|
isolate browser components with different privileges.
|
||||||
* [Zygote Process](linux_zygote.md) - How the Linux Zygote process, used to
|
* [Zygote Process](linux_zygote.md) - How the Linux Zygote process, used to
|
||||||
spawn new processes, works.
|
spawn new processes, works.
|
||||||
* [Running Layout Tests on Linux](layout_tests_linux.md) - Linux-specific
|
* [Running Web Tests on Linux](web_tests_linux.md) - Linux-specific
|
||||||
instructions for running layout tests.
|
instructions for running web tests.
|
||||||
* [Linux Sysroot Images](linux_sysroot.md) - How builds use libraries on Linux
|
* [Linux Sysroot Images](linux_sysroot.md) - How builds use libraries on Linux
|
||||||
* [Linux Hardware Video Decoding](linux_hw_video_decode.md) - Enabling
|
* [Linux Hardware Video Decoding](linux_hw_video_decode.md) - Enabling
|
||||||
hardware video decode codepaths on Linux
|
hardware video decode codepaths on Linux
|
||||||
|
@@ -22,7 +22,7 @@ a new one.
|
|||||||
|
|
||||||
Test files:
|
Test files:
|
||||||
```
|
```
|
||||||
third_party/WebKit/LayoutTests/accessibility
|
third_party/blink/web_tests/accessibility
|
||||||
```
|
```
|
||||||
|
|
||||||
Source code to AccessibilityController and WebAXObjectProxy:
|
Source code to AccessibilityController and WebAXObjectProxy:
|
||||||
@@ -30,7 +30,7 @@ Source code to AccessibilityController and WebAXObjectProxy:
|
|||||||
content/shell/test_runner
|
content/shell/test_runner
|
||||||
```
|
```
|
||||||
|
|
||||||
To run all accessibility LayoutTests:
|
To run all accessibility web tests:
|
||||||
```
|
```
|
||||||
autoninja -C out/release blink_tests
|
autoninja -C out/release blink_tests
|
||||||
third_party/blink/tools/run_web_tests.py --build-directory=out --target=release accessibility/
|
third_party/blink/tools/run_web_tests.py --build-directory=out --target=release accessibility/
|
||||||
@@ -39,7 +39,7 @@ third_party/blink/tools/run_web_tests.py --build-directory=out --target=release
|
|||||||
To run just one test by itself without the script:
|
To run just one test by itself without the script:
|
||||||
```
|
```
|
||||||
autoninja -C out/release blink_tests
|
autoninja -C out/release blink_tests
|
||||||
out/release/content_shell --run-web-tests third_party/WebKit/LayoutTests/accessibility/name-calc-inputs.html
|
out/release/content_shell --run-web-tests third_party/blink/web_tests/accessibility/name-calc-inputs.html
|
||||||
```
|
```
|
||||||
|
|
||||||
## DumpAccessibilityTree tests
|
## DumpAccessibilityTree tests
|
||||||
|
@@ -284,9 +284,9 @@ out/Release/bin/java_deobfuscate out/Release/apks/ChromePublicTest.apk.mapping <
|
|||||||
Any stacks produced by test runner output will already be deobfuscated.
|
Any stacks produced by test runner output will already be deobfuscated.
|
||||||
|
|
||||||
|
|
||||||
## Running Blink Layout Tests
|
## Running Blink Web Tests
|
||||||
|
|
||||||
See [Layout Tests](testing/layout_tests.md).
|
See [Web Tests](testing/web_tests.md).
|
||||||
|
|
||||||
## Running GPU tests
|
## Running GPU tests
|
||||||
|
|
||||||
|
@@ -54,7 +54,7 @@ $ ./build.py --clang-completer --system-libclang
|
|||||||
|
|
||||||
## JavaScript lint
|
## JavaScript lint
|
||||||
|
|
||||||
Install JavaScript linter for Blink layout tests.
|
Install JavaScript linter for Blink web tests.
|
||||||
|
|
||||||
```
|
```
|
||||||
$ npm install -g eslint eslint-config-google
|
$ npm install -g eslint eslint-config-google
|
||||||
|
@@ -28,7 +28,7 @@ to see
|
|||||||
4. how to use it in blink
|
4. how to use it in blink
|
||||||
[[1](https://chromium-review.googlesource.com/c/554510/8/third_party/blnk/renderere/core/workers/worker_thread.cc)]
|
[[1](https://chromium-review.googlesource.com/c/554510/8/third_party/blnk/renderere/core/workers/worker_thread.cc)]
|
||||||
|
|
||||||
Also, this patch added a virtual test for running layout tests with the flag.
|
Also, this patch added a virtual test for running web tests with the flag.
|
||||||
When you add a flag, you can consider to use that.
|
When you add a flag, you can consider to use that.
|
||||||
|
|
||||||
## Step 2: Adding the feature flag to the chrome://flags UI.
|
## Step 2: Adding the feature flag to the chrome://flags UI.
|
||||||
|
@@ -1,6 +1,6 @@
|
|||||||
# How to Extend the Layout Test Framework
|
# How to Extend the Web Test Framework
|
||||||
|
|
||||||
The Layout Test Framework that Blink uses is a regression testing tool that is
|
The Web Test Framework that Blink uses is a regression testing tool that is
|
||||||
multi-platform and it has a large amount of tools that help test varying types
|
multi-platform and it has a large amount of tools that help test varying types
|
||||||
of regression, such as pixel diffs, text diffs, etc. The framework is mainly
|
of regression, such as pixel diffs, text diffs, etc. The framework is mainly
|
||||||
used by Blink, however it was made to be extensible so that other projects can
|
used by Blink, however it was made to be extensible so that other projects can
|
||||||
@@ -12,18 +12,17 @@ to help people who want to actually the framework to test whatever they want.
|
|||||||
## Background
|
## Background
|
||||||
|
|
||||||
Before you can start actually extending the framework, you should be familiar
|
Before you can start actually extending the framework, you should be familiar
|
||||||
with how to use it. See the
|
with how to use it. See the [web tests documentation](testing/web_tests.md).
|
||||||
[layout tests documentation](testing/layout_tests.md).
|
|
||||||
|
|
||||||
## How to Extend the Framework
|
## How to Extend the Framework
|
||||||
|
|
||||||
There are two parts to actually extending framework to test a piece of software.
|
There are two parts to actually extending framework to test a piece of software.
|
||||||
The first part is extending certain files in:
|
The first part is extending certain files in:
|
||||||
[/third_party/blink/tools/blinkpy/web_tests/](/third_party/blink/tools/blinkpy/web_tests/)
|
[/third_party/blink/tools/blinkpy/web_tests/](/third_party/blink/tools/blinkpy/web_tests/)
|
||||||
The code in `blinkpy/web_tests` is the layout test framework itself
|
The code in `blinkpy/web_tests` is the web test framework itself
|
||||||
|
|
||||||
The second part is creating a driver (program) to actually communicate the
|
The second part is creating a driver (program) to actually communicate the
|
||||||
layout test framework. This part is significantly more tricky and dependent on
|
web test framework. This part is significantly more tricky and dependent on
|
||||||
what exactly exactly is being tested.
|
what exactly exactly is being tested.
|
||||||
|
|
||||||
### Part 1
|
### Part 1
|
||||||
@@ -31,10 +30,10 @@ what exactly exactly is being tested.
|
|||||||
This part isn’t too difficult. There are basically two classes that need to be
|
This part isn’t too difficult. There are basically two classes that need to be
|
||||||
extended (ideally, just inherited from). These classes are:
|
extended (ideally, just inherited from). These classes are:
|
||||||
|
|
||||||
* `Driver`. Located in `layout_tests/port/driver.py`. Each instance of this is
|
* `Driver`. Located in `web_tests/port/driver.py`. Each instance of this is
|
||||||
the class that will actually an instance of the program that produces the
|
the class that will actually an instance of the program that produces the
|
||||||
test data (program in Part 2).
|
test data (program in Part 2).
|
||||||
* `Port`. Located in `layout_tests/port/base.py`. This class is responsible
|
* `Port`. Located in `web_tests/port/base.py`. This class is responsible
|
||||||
creating drivers with the correct settings, giving access to certain OS
|
creating drivers with the correct settings, giving access to certain OS
|
||||||
functionality to access expected files, etc.
|
functionality to access expected files, etc.
|
||||||
|
|
||||||
@@ -129,7 +128,7 @@ Here are some of the functions that most likely need to be overridden.
|
|||||||
* `layout_tests_dir`
|
* `layout_tests_dir`
|
||||||
* This tells the port where to look for all the and everything associated
|
* This tells the port where to look for all the and everything associated
|
||||||
with them such as resources files.
|
with them such as resources files.
|
||||||
* By default it returns the absolute path to the layout tests directory.
|
* By default it returns the absolute path to the web tests directory.
|
||||||
* If you are planning on running something in the chromium src/ directory,
|
* If you are planning on running something in the chromium src/ directory,
|
||||||
there are helper functions to allow you to return a path relative to the
|
there are helper functions to allow you to return a path relative to the
|
||||||
base of the chromium src directory.
|
base of the chromium src directory.
|
||||||
@@ -137,7 +136,7 @@ Here are some of the functions that most likely need to be overridden.
|
|||||||
The rest of the functions can definitely be overridden for your projects
|
The rest of the functions can definitely be overridden for your projects
|
||||||
specific needs, however these are the bare minimum needed to get it running.
|
specific needs, however these are the bare minimum needed to get it running.
|
||||||
There are also functions you can override to make certain actions that aren’t on
|
There are also functions you can override to make certain actions that aren’t on
|
||||||
by default always take place. For example, the layout test framework always
|
by default always take place. For example, the web test framework always
|
||||||
checks for system dependencies unless you pass in a switch. If you want them
|
checks for system dependencies unless you pass in a switch. If you want them
|
||||||
disabled for your project, just override `check_sys_deps` to always return OK.
|
disabled for your project, just override `check_sys_deps` to always return OK.
|
||||||
This way you don’t need to pass in so many switches.
|
This way you don’t need to pass in so many switches.
|
||||||
@@ -173,7 +172,7 @@ and has stdout, stdin, stderr.
|
|||||||
#### Goals
|
#### Goals
|
||||||
|
|
||||||
Your goal for this part of the project is to create a program (or extend a
|
Your goal for this part of the project is to create a program (or extend a
|
||||||
program) to interface with the layout test framework. The layout test framework
|
program) to interface with the web test framework. The layout test framework
|
||||||
will communicate with this program to tell it what to do and it will accept data
|
will communicate with this program to tell it what to do and it will accept data
|
||||||
from this program to perform the regression testing or create new base line
|
from this program to perform the regression testing or create new base line
|
||||||
files.
|
files.
|
||||||
@@ -189,7 +188,7 @@ This is how your code should be laid out.
|
|||||||
the `run_test()` in the driver.
|
the `run_test()` in the driver.
|
||||||
1. Infinite Loop (!)
|
1. Infinite Loop (!)
|
||||||
* After initialization, your program needs to actually wait for input,
|
* After initialization, your program needs to actually wait for input,
|
||||||
then process that input to carry out the test. In the context of layout
|
then process that input to carry out the test. In the context of web
|
||||||
testing, the `content_shell` needs to wait for an html file to navigate
|
testing, the `content_shell` needs to wait for an html file to navigate
|
||||||
to, render it, then convert that rendering to a PNG. It does this
|
to, render it, then convert that rendering to a PNG. It does this
|
||||||
constantly, until a signal/message is sent to indicate that no more
|
constantly, until a signal/message is sent to indicate that no more
|
||||||
@@ -228,12 +227,12 @@ That’s basically what the skeleton of your program should be.
|
|||||||
### Details
|
### Details
|
||||||
|
|
||||||
This is information about how to do some specific things, such as sending data
|
This is information about how to do some specific things, such as sending data
|
||||||
to the layout test framework.
|
to the web test framework.
|
||||||
|
|
||||||
* Content Blocks
|
* Content Blocks
|
||||||
* The layout test framework accepts output from your program in blocks of
|
* The web test framework accepts output from your program in blocks of
|
||||||
data through stdout. Therefore, printing to stdout is really sending
|
data through stdout. Therefore, printing to stdout is really sending
|
||||||
data to the layout test framework.
|
data to the web test framework.
|
||||||
* Structure of block
|
* Structure of block
|
||||||
* “Header: Data\n”
|
* “Header: Data\n”
|
||||||
* Header indicates what type of data will be sent through. A list
|
* Header indicates what type of data will be sent through. A list
|
@@ -116,18 +116,18 @@ builder.
|
|||||||
* [linux-blink-gen-property-trees](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-blink-gen-property-trees) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux-blink-gen-property-trees)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux-blink-gen-property-trees))
|
* [linux-blink-gen-property-trees](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-blink-gen-property-trees) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux-blink-gen-property-trees)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux-blink-gen-property-trees))
|
||||||
|
|
||||||
Path regular expressions:
|
Path regular expressions:
|
||||||
* [`//third_party/WebKit/LayoutTests/FlagExpectations/enable-blink-gen-property-trees`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/WebKit/LayoutTests/FlagExpectations/enable-blink-gen-property-trees)
|
* [`//third_party/blink/web_tests/FlagExpectations/enable-blink-gen-property-trees`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/FlagExpectations/enable-blink-gen-property-trees)
|
||||||
* [`//third_party/WebKit/LayoutTests/flag-specific/enable-blink-gen-property-trees/.+`](https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/flag-specific/enable-blink-gen-property-trees/)
|
* [`//third_party/blink/web_tests/flag-specific/enable-blink-gen-property-trees/.+`](https://cs.chromium.org/chromium/src/third_party/blink/web_tests/flag-specific/enable-blink-gen-property-trees/)
|
||||||
|
|
||||||
* [linux-blink-rel](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-blink-rel) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux-blink-rel)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux-blink-rel))
|
* [linux-blink-rel](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-blink-rel) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux-blink-rel)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux-blink-rel))
|
||||||
|
|
||||||
Path regular expressions:
|
Path regular expressions:
|
||||||
* [`//cc/.+`](https://cs.chromium.org/chromium/src/cc/)
|
* [`//cc/.+`](https://cs.chromium.org/chromium/src/cc/)
|
||||||
* [`//third_party/WebKit/LayoutTests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees)`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/WebKit/LayoutTests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees))
|
|
||||||
* [`//third_party/WebKit/LayoutTests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/WebKit/LayoutTests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/)
|
|
||||||
* [`//third_party/blink/renderer/core/layout/compositing/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/layout/compositing/)
|
|
||||||
* [`//third_party/blink/renderer/core/(svg|paint)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/renderer/core/(svg|paint)/)
|
* [`//third_party/blink/renderer/core/(svg|paint)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/renderer/core/(svg|paint)/)
|
||||||
|
* [`//third_party/blink/renderer/core/layout/compositing/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/layout/compositing/)
|
||||||
* [`//third_party/blink/renderer/platform/graphics/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/platform/graphics/)
|
* [`//third_party/blink/renderer/platform/graphics/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/platform/graphics/)
|
||||||
|
* [`//third_party/blink/web_tests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees)`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees))
|
||||||
|
* [`//third_party/blink/web_tests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/)
|
||||||
|
|
||||||
* [linux_chromium_dbg_ng](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_dbg_ng) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_chromium_dbg_ng)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_chromium_dbg_ng))
|
* [linux_chromium_dbg_ng](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_dbg_ng) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_chromium_dbg_ng)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_chromium_dbg_ng))
|
||||||
|
|
||||||
@@ -137,27 +137,27 @@ builder.
|
|||||||
* [linux_layout_tests_layout_ng](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_layout_tests_layout_ng) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_layout_tests_layout_ng)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_layout_tests_layout_ng))
|
* [linux_layout_tests_layout_ng](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_layout_tests_layout_ng) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_layout_tests_layout_ng)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_layout_tests_layout_ng))
|
||||||
|
|
||||||
Path regular expressions:
|
Path regular expressions:
|
||||||
* [`//third_party/Webkit/LayoutTests/FlagExpectations/enable-blink-features=LayoutNG`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/Webkit/LayoutTests/FlagExpectations/enable-blink-features=LayoutNG)
|
|
||||||
* [`//third_party/Webkit/LayoutTests/flag-specific/enable-blink-features=LayoutNG/.+`](https://cs.chromium.org/chromium/src/third_party/Webkit/LayoutTests/flag-specific/enable-blink-features=LayoutNG/)
|
|
||||||
* [`//third_party/blink/renderer/core/editing/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/editing/)
|
|
||||||
* [`//third_party/blink/renderer/core/(layout|paint)/ng/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/renderer/core/(layout|paint)/ng/)
|
* [`//third_party/blink/renderer/core/(layout|paint)/ng/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/renderer/core/(layout|paint)/ng/)
|
||||||
|
* [`//third_party/blink/renderer/core/editing/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/editing/)
|
||||||
* [`//third_party/blink/renderer/platform/fonts/shaping/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/platform/fonts/shaping/)
|
* [`//third_party/blink/renderer/platform/fonts/shaping/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/platform/fonts/shaping/)
|
||||||
|
* [`//third_party/blink/web_tests/FlagExpectations/enable-blink-features=LayoutNG`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/FlagExpectations/enable-blink-features=LayoutNG)
|
||||||
|
* [`//third_party/blink/web_tests/flag-specific/enable-blink-features=LayoutNG/.+`](https://cs.chromium.org/chromium/src/third_party/blink/web_tests/flag-specific/enable-blink-features=LayoutNG/)
|
||||||
|
|
||||||
* [linux_layout_tests_slimming_paint_v2](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_layout_tests_slimming_paint_v2) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_layout_tests_slimming_paint_v2)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_layout_tests_slimming_paint_v2))
|
* [linux_layout_tests_slimming_paint_v2](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_layout_tests_slimming_paint_v2) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_layout_tests_slimming_paint_v2)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_layout_tests_slimming_paint_v2))
|
||||||
|
|
||||||
Path regular expressions:
|
Path regular expressions:
|
||||||
* [`//third_party/WebKit/LayoutTests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees)`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/WebKit/LayoutTests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees))
|
|
||||||
* [`//third_party/WebKit/LayoutTests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/WebKit/LayoutTests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/)
|
|
||||||
* [`//third_party/blink/renderer/core/layout/compositing/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/layout/compositing/)
|
* [`//third_party/blink/renderer/core/layout/compositing/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/layout/compositing/)
|
||||||
* [`//third_party/blink/renderer/core/(svg|paint)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/renderer/core/(svg|paint)/)
|
* [`//third_party/blink/renderer/core/(svg|paint)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/renderer/core/(svg|paint)/)
|
||||||
* [`//third_party/blink/renderer/platform/graphics/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/platform/graphics/)
|
* [`//third_party/blink/renderer/platform/graphics/.+`](https://cs.chromium.org/chromium/src/third_party/blink/renderer/platform/graphics/)
|
||||||
|
* [`//third_party/blink/web_tests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees)`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/FlagExpectations/(enable-slimming-paint-v2|enable-blink-gen-property-trees))
|
||||||
|
* [`//third_party/blink/web_tests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/.+`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/flag-specific/(enable-slimming-paint-v2|enable-blink-gen-property-trees)/)
|
||||||
|
|
||||||
* [linux_mojo](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_mojo) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_mojo)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_mojo))
|
* [linux_mojo](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_mojo) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_mojo)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_mojo))
|
||||||
|
|
||||||
Path regular expressions:
|
Path regular expressions:
|
||||||
* [`//services/network/.+`](https://cs.chromium.org/chromium/src/services/network/)
|
* [`//services/network/.+`](https://cs.chromium.org/chromium/src/services/network/)
|
||||||
* [`//testing/buildbot/filters/mojo\\.fyi\\.network_.*`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:testing/buildbot/filters/mojo\\.fyi\\.network_.*)
|
* [`//testing/buildbot/filters/mojo\\.fyi\\.network_.*`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:testing/buildbot/filters/mojo\\.fyi\\.network_.*)
|
||||||
* [`//third_party/WebKit/LayoutTests/FlagExpectations/enable-features=NetworkService`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/WebKit/LayoutTests/FlagExpectations/enable-features=NetworkService)
|
* [`//third_party/blink/web_tests/FlagExpectations/enable-features=NetworkService`](https://cs.chromium.org/search/?q=package:%5Echromium$+file:third_party/blink/web_tests/FlagExpectations/enable-features=NetworkService)
|
||||||
|
|
||||||
* [linux_optional_gpu_tests_rel](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_optional_gpu_tests_rel) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_optional_gpu_tests_rel)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_optional_gpu_tests_rel))
|
* [linux_optional_gpu_tests_rel](https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_optional_gpu_tests_rel) ([`cq.cfg` entry](https://cs.chromium.org/search/?q=package:%5Echromium$+file:cq.cfg+linux_optional_gpu_tests_rel)) ([matching builders](https://cs.chromium.org/search/?q=file:trybots.py+linux_optional_gpu_tests_rel))
|
||||||
|
|
||||||
|
@@ -413,7 +413,7 @@ pulseaudio-libs-devel zlib.i686 httpd mod_ssl php php-cli python-psutil wdiff \
|
|||||||
xorg-x11-server-Xvfb'
|
xorg-x11-server-Xvfb'
|
||||||
```
|
```
|
||||||
|
|
||||||
The fonts needed by Blink's LayoutTests can be obtained by following [these
|
The fonts needed by Blink's web tests can be obtained by following [these
|
||||||
instructions](https://gist.github.com/pwnall/32a3b11c2b10f6ae5c6a6de66c1e12ae).
|
instructions](https://gist.github.com/pwnall/32a3b11c2b10f6ae5c6a6de66c1e12ae).
|
||||||
For the optional packages:
|
For the optional packages:
|
||||||
|
|
||||||
|
@@ -306,7 +306,7 @@ See [linux_minidump_to_core.md](linux_minidump_to_core.md)
|
|||||||
Many of our tests bring up windows on screen. This can be annoying (they steal
|
Many of our tests bring up windows on screen. This can be annoying (they steal
|
||||||
your focus) and hard to debug (they receive extra events as you mouse over them).
|
your focus) and hard to debug (they receive extra events as you mouse over them).
|
||||||
Instead, use `Xvfb` or `Xephyr` to run a nested X session to debug them, as
|
Instead, use `Xvfb` or `Xephyr` to run a nested X session to debug them, as
|
||||||
outlined on [layout_tests_linux.md](layout_tests_linux.md).
|
outlined on [web_tests_linux.md](web_tests_linux.md).
|
||||||
|
|
||||||
### Browser tests
|
### Browser tests
|
||||||
|
|
||||||
@@ -323,10 +323,10 @@ browser process share the outermost process.
|
|||||||
|
|
||||||
To debug a renderer process in this case, use the tips above about renderers.
|
To debug a renderer process in this case, use the tips above about renderers.
|
||||||
|
|
||||||
### Layout tests
|
### Web tests
|
||||||
|
|
||||||
See [layout_tests_linux.md](layout_tests_linux.md) for some tips. In particular,
|
See [web_tests_linux.md](web_tests_linux.md) for some tips. In particular,
|
||||||
note that it's possible to debug a layout test via `ssh`ing to a Linux box; you
|
note that it's possible to debug a web test via `ssh`ing to a Linux box; you
|
||||||
don't need anything on screen if you use `Xvfb`.
|
don't need anything on screen if you use `Xvfb`.
|
||||||
|
|
||||||
### UI tests
|
### UI tests
|
||||||
@@ -464,7 +464,7 @@ See the last section of [Linux Crash Dumping](linux_crash_dumping.md).
|
|||||||
If you break in a debugger during a drag, Chrome will have grabbed your mouse
|
If you break in a debugger during a drag, Chrome will have grabbed your mouse
|
||||||
and keyboard so you won't be able to interact with the debugger! To work around
|
and keyboard so you won't be able to interact with the debugger! To work around
|
||||||
this, run via `Xephyr`. Instructions for how to use `Xephyr` are on the
|
this, run via `Xephyr`. Instructions for how to use `Xephyr` are on the
|
||||||
[Running layout tests on Linux](layout_tests_linux.md) page.
|
[Running web tests on Linux](web_tests_linux.md) page.
|
||||||
|
|
||||||
## Tracking Down Bugs
|
## Tracking Down Bugs
|
||||||
|
|
||||||
@@ -520,7 +520,7 @@ Some strategies are:
|
|||||||
|
|
||||||
To test on various window managers, you can use a nested X server like `Xephyr`.
|
To test on various window managers, you can use a nested X server like `Xephyr`.
|
||||||
Instructions for how to use `Xephyr` are on the
|
Instructions for how to use `Xephyr` are on the
|
||||||
[Running layout tests on Linux](layout_tests_linux.md) page.
|
[Running web tests on Linux](web_tests_linux.md) page.
|
||||||
|
|
||||||
If you need to test something with hardware accelerated compositing
|
If you need to test something with hardware accelerated compositing
|
||||||
(e.g., compiz), you can use `Xgl` (`sudo apt-get install xserver-xgl`). E.g.:
|
(e.g., compiz), you can use `Xgl` (`sudo apt-get install xserver-xgl`). E.g.:
|
||||||
|
@@ -143,8 +143,8 @@ In the Project Explorer on the left side:
|
|||||||
1. Add another filter:
|
1. Add another filter:
|
||||||
* Exclude all
|
* Exclude all
|
||||||
* Folders
|
* Folders
|
||||||
* Name matches `out_.*|\.git|LayoutTests` regular expression
|
* Name matches `out_.*|\.git|web_tests` regular expression
|
||||||
* If you aren't working on WebKit, adding `|WebKit` will remove more
|
* If you aren't working on WebKit, adding `|blink` will remove more
|
||||||
files
|
files
|
||||||
1. Click "Apply and Close"
|
1. Click "Apply and Close"
|
||||||
|
|
||||||
|
@@ -109,7 +109,7 @@ tools/origin_trials/generate_token.py http://localhost:8000 MyFeature
|
|||||||
```
|
```
|
||||||
|
|
||||||
The file `tools/origin_trials/eftest.key` is used by default as the private key
|
The file `tools/origin_trials/eftest.key` is used by default as the private key
|
||||||
for the test keypair used by Origin Trials unit tests and layout tests (i.e. in
|
for the test keypair used by Origin Trials unit tests and web tests (i.e. in
|
||||||
content shell). Tokens generated with this key will **not** work in the browser
|
content shell). Tokens generated with this key will **not** work in the browser
|
||||||
by default (see the [Developer Guide] for instructions on creating real tokens).
|
by default (see the [Developer Guide] for instructions on creating real tokens).
|
||||||
To use a test token with the browser, run Chrome with the command-line flag:
|
To use a test token with the browser, run Chrome with the command-line flag:
|
||||||
@@ -123,8 +123,8 @@ it doesn't work, see [trial_token_unittest.cc]. If you cannot set command-line
|
|||||||
switches (e.g., on Chrome OS), you can also directly modify
|
switches (e.g., on Chrome OS), you can also directly modify
|
||||||
[chrome_origin_trial_policy.cc].
|
[chrome_origin_trial_policy.cc].
|
||||||
|
|
||||||
### Layout Tests
|
### Web Tests
|
||||||
When using the \[OriginTrialEnabled\] IDL attribute, you should add layout tests
|
When using the \[OriginTrialEnabled\] IDL attribute, you should add web tests
|
||||||
to verify that the V8 bindings code is working as expected. Depending on how
|
to verify that the V8 bindings code is working as expected. Depending on how
|
||||||
your feature is exposed, you'll want tests for the exposed interfaces, as well
|
your feature is exposed, you'll want tests for the exposed interfaces, as well
|
||||||
as tests for script-added tokens. For examples, refer to the existing tests in
|
as tests for script-added tokens. For examples, refer to the existing tests in
|
||||||
@@ -135,6 +135,6 @@ as tests for script-added tokens. For examples, refer to the existing tests in
|
|||||||
[generate_token.py]: /tools/origin_trials/generate_token.py
|
[generate_token.py]: /tools/origin_trials/generate_token.py
|
||||||
[Developer Guide]: https://github.com/jpchase/OriginTrials/blob/gh-pages/developer-guide.md
|
[Developer Guide]: https://github.com/jpchase/OriginTrials/blob/gh-pages/developer-guide.md
|
||||||
[OriginTrialEnabled]: /third_party/blink/renderer/bindings/IDLExtendedAttributes.md#_OriginTrialEnabled_i_m_a_c_
|
[OriginTrialEnabled]: /third_party/blink/renderer/bindings/IDLExtendedAttributes.md#_OriginTrialEnabled_i_m_a_c_
|
||||||
[origin_trials/webexposed]: /third_party/WebKit/LayoutTests/http/tests/origin_trials/webexposed/
|
[origin_trials/webexposed]: /third_party/blink/web_tests/http/tests/origin_trials/webexposed/
|
||||||
[runtime\_enabled\_features.json5]: /third_party/blink/renderer/platform/runtime_enabled_features.json5
|
[runtime\_enabled\_features.json5]: /third_party/blink/renderer/platform/runtime_enabled_features.json5
|
||||||
[trial_token_unittest.cc]: /third_party/blink/common/origin_trials/trial_token_unittest.cc
|
[trial_token_unittest.cc]: /third_party/blink/common/origin_trials/trial_token_unittest.cc
|
||||||
|
@@ -195,7 +195,7 @@ Below, we go through strategies for some common challenges encountered when
|
|||||||
servicifying features that have Blink as a client.
|
servicifying features that have Blink as a client.
|
||||||
|
|
||||||
#### Mocking Interface Impls in JS
|
#### Mocking Interface Impls in JS
|
||||||
It is a common pattern in Blink's layout tests to mock a remote Mojo interface
|
It is a common pattern in Blink's web tests to mock a remote Mojo interface
|
||||||
in JS. [This CL](https://codereview.chromium.org/2643713002) illustrates the
|
in JS. [This CL](https://codereview.chromium.org/2643713002) illustrates the
|
||||||
basic pattern for porting such mocking of an interface hosted by
|
basic pattern for porting such mocking of an interface hosted by
|
||||||
//content/browser to an interface hosted by an arbitrary service (see the
|
//content/browser to an interface hosted by an arbitrary service (see the
|
||||||
@@ -308,7 +308,7 @@ To answer this question, there are several different strategies. These
|
|||||||
strategies are not mutually-exclusive; they can and should be combined to
|
strategies are not mutually-exclusive; they can and should be combined to
|
||||||
preserve the full breadth of coverage.
|
preserve the full breadth of coverage.
|
||||||
|
|
||||||
- Blink client-side behavior can be tested via [layout tests](https://codereview.chromium.org/2731953003)
|
- Blink client-side behavior can be tested via [web tests](https://codereview.chromium.org/2731953003)
|
||||||
- To test service impl behavior, create [service tests](https://codereview.chromium.org/2774783003).
|
- To test service impl behavior, create [service tests](https://codereview.chromium.org/2774783003).
|
||||||
- To preserve tests of end-to-end behavior (e.g., that when Blink makes a
|
- To preserve tests of end-to-end behavior (e.g., that when Blink makes a
|
||||||
request via a Web API in JS, the relevant feature impl receives a connection
|
request via a Web API in JS, the relevant feature impl receives a connection
|
||||||
|
@@ -5,7 +5,7 @@
|
|||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
Blink perf tests are used for micro benchmarking the surface of Blink that
|
Blink perf tests are used for micro benchmarking the surface of Blink that
|
||||||
is exposed to the Web. They are the counterpart of [LayoutTests/](../../../docs/testing/layout_tests.md)
|
is exposed to the Web. They are the counterpart of [web_tests/](../../../docs/testing/layout_tests.md)
|
||||||
but for performance coverage.
|
but for performance coverage.
|
||||||
|
|
||||||
## Writing Tests
|
## Writing Tests
|
||||||
|
@@ -1,5 +1,5 @@
|
|||||||
|
|
||||||
# Fixing layout test flakiness
|
# Fixing web test flakiness
|
||||||
|
|
||||||
We'd like to stamp out all the tests that have ordering dependencies. This helps
|
We'd like to stamp out all the tests that have ordering dependencies. This helps
|
||||||
make the tests more reliable and, eventually, will make it so we can run tests
|
make the tests more reliable and, eventually, will make it so we can run tests
|
||||||
|
@@ -14,7 +14,7 @@ output of the test run, and hence the test can be tailored for human readability
|
|||||||
as a result). We also upload the test results to dashboards like the
|
as a result). We also upload the test results to dashboards like the
|
||||||
[Flakiness Dashboard](http://test-results.appspot.com).
|
[Flakiness Dashboard](http://test-results.appspot.com).
|
||||||
|
|
||||||
The test format originated with the Blink layout tests, but has since been
|
The test format originated with the Blink web tests, but has since been
|
||||||
adopted by GTest-based tests and Python unittest-based tests, so we've
|
adopted by GTest-based tests and Python unittest-based tests, so we've
|
||||||
standardized on it for anything related to tracking test flakiness.
|
standardized on it for anything related to tracking test flakiness.
|
||||||
|
|
||||||
@@ -78,7 +78,7 @@ object may appear in any order.
|
|||||||
|------------|-----------|-------------|
|
|------------|-----------|-------------|
|
||||||
| `interrupted` | boolean | **Required.** Whether the test run was interrupted and terminated early (either via the runner bailing out or the user hitting ctrl-C, etc.) If true, this indicates that not all of the tests in the suite were run and the results are at best incomplete and possibly totally invalid. |
|
| `interrupted` | boolean | **Required.** Whether the test run was interrupted and terminated early (either via the runner bailing out or the user hitting ctrl-C, etc.) If true, this indicates that not all of the tests in the suite were run and the results are at best incomplete and possibly totally invalid. |
|
||||||
| `num_failures_by_type` | dict | **Required.** A summary of the totals of each result type. If a test was run more than once, only the first invocation's result is included in the totals. Each key is one of the result types listed below. A missing result type is the same as being present and set to zero (0). |
|
| `num_failures_by_type` | dict | **Required.** A summary of the totals of each result type. If a test was run more than once, only the first invocation's result is included in the totals. Each key is one of the result types listed below. A missing result type is the same as being present and set to zero (0). |
|
||||||
| `path_delimiter` | string | **Optional, will be mandatory.** The separator string to use in between components of a tests name; normally "." for GTest- and Python-based tests and "/" for layout tests; if not present, you should default to "/" for backwards-compatibility. |
|
| `path_delimiter` | string | **Optional, will be mandatory.** The separator string to use in between components of a tests name; normally "." for GTest- and Python-based tests and "/" for web tests; if not present, you should default to "/" for backwards-compatibility. |
|
||||||
| `seconds_since_epoch` | float | **Required.** The start time of the test run expressed as a floating-point offset in seconds from the UNIX epoch. |
|
| `seconds_since_epoch` | float | **Required.** The start time of the test run expressed as a floating-point offset in seconds from the UNIX epoch. |
|
||||||
| `tests` | dict | **Required.** The actual trie of test results. Each directory or module component in the test name is a node in the trie, and the leaf contains the dict of per-test fields as described below. |
|
| `tests` | dict | **Required.** The actual trie of test results. Each directory or module component in the test name is a node in the trie, and the leaf contains the dict of per-test fields as described below. |
|
||||||
| `version` | integer | **Required.** Version of the file format. Current version is 3. |
|
| `version` | integer | **Required.** Version of the file format. Current version is 3. |
|
||||||
@@ -87,10 +87,10 @@ object may appear in any order.
|
|||||||
| `build_number` | string | **Optional.** If this test run was produced on a bot, this should be the build number of the run, e.g., "1234". |
|
| `build_number` | string | **Optional.** If this test run was produced on a bot, this should be the build number of the run, e.g., "1234". |
|
||||||
| `builder_name` | string | **Optional.** If this test run was produced on a bot, this should be the builder name of the bot, e.g., "Linux Tests". |
|
| `builder_name` | string | **Optional.** If this test run was produced on a bot, this should be the builder name of the bot, e.g., "Linux Tests". |
|
||||||
| `chromium_revision` | string | **Optional.** The revision of the current Chromium checkout, if relevant, e.g. "356123". |
|
| `chromium_revision` | string | **Optional.** The revision of the current Chromium checkout, if relevant, e.g. "356123". |
|
||||||
| `has_pretty_patch` | bool | **Optional, layout test specific, deprecated.** Whether the layout tests' output contains PrettyDiff-formatted diffs for test failures. |
|
| `has_pretty_patch` | bool | **Optional, layout test specific, deprecated.** Whether the web tests' output contains PrettyDiff-formatted diffs for test failures. |
|
||||||
| `has_wdiff` | bool | **Optional, layout test specific, deprecated.** Whether the layout tests' output contains wdiff-formatted diffs for test failures. |
|
| `has_wdiff` | bool | **Optional, layout test specific, deprecated.** Whether the web tests' output contains wdiff-formatted diffs for test failures. |
|
||||||
| `layout_tests_dir` | string | **Optional, layout test specific.** Path to the LayoutTests directory for the test run (used so that we can link to the tests used in the run). |
|
| `layout_tests_dir` | string | **Optional, layout test specific.** Path to the web_tests directory for the test run (used so that we can link to the tests used in the run). |
|
||||||
| `pixel_tests_enabled` | bool | **Optional, layout test specific.** Whether the layout tests' were run with the --pixel-tests flag. |
|
| `pixel_tests_enabled` | bool | **Optional, layout test specific.** Whether the web tests' were run with the --pixel-tests flag. |
|
||||||
| `fixable` | integer | **Optional, deprecated.** The number of tests that were run but were expected to fail. |
|
| `fixable` | integer | **Optional, deprecated.** The number of tests that were run but were expected to fail. |
|
||||||
| `num_flaky` | integer | **Optional, deprecated.** The number of tests that were run more than once and produced different results each time. |
|
| `num_flaky` | integer | **Optional, deprecated.** The number of tests that were run more than once and produced different results each time. |
|
||||||
| `num_passes` | integer | **Optional, deprecated.** The number of successful tests; equivalent to `num_failures_by_type["Pass"]` |
|
| `num_passes` | integer | **Optional, deprecated.** The number of successful tests; equivalent to `num_failures_by_type["Pass"]` |
|
||||||
@@ -117,17 +117,17 @@ then is retried and passes, both `is_regression` and `is_unexpected` will be fal
|
|||||||
| `is_unexpected` | bool | **Optional.** If present and true, the test result was unexpected. This might include an unexpected pass, i.e., it is not necessarily a regression. If false (or if the key is not present at all), the test produced the expected result. |
|
| `is_unexpected` | bool | **Optional.** If present and true, the test result was unexpected. This might include an unexpected pass, i.e., it is not necessarily a regression. If false (or if the key is not present at all), the test produced the expected result. |
|
||||||
| `time` | float | **Optional.** If present, the time it took in seconds to execute the first invocation of the test. |
|
| `time` | float | **Optional.** If present, the time it took in seconds to execute the first invocation of the test. |
|
||||||
| `times` | array of floats | **Optional.** If present, the times in seconds of each invocation of the test. |
|
| `times` | array of floats | **Optional.** If present, the times in seconds of each invocation of the test. |
|
||||||
| `has_repaint_overlay` | bool | **Optional, layout test specific.** If present and true, indicates that the test output contains the data needed to draw repaint overlays to help explain the results (only used in layout tests). |
|
| `has_repaint_overlay` | bool | **Optional, web test specific.** If present and true, indicates that the test output contains the data needed to draw repaint overlays to help explain the results (only used in layout tests). |
|
||||||
| `is_missing_audio` | bool | **Optional, layout test specific.** If present and true, the test was supposed to have an audio baseline to compare against, and we didn't find one. |
|
| `is_missing_audio` | bool | **Optional, we test specific.** If present and true, the test was supposed to have an audio baseline to compare against, and we didn't find one. |
|
||||||
| `is_missing_text` | bool | **Optional, layout test specific.** If present and true, the test was supposed to have a text baseline to compare against, and we didn't find one. |
|
| `is_missing_text` | bool | **Optional, web test specific.** If present and true, the test was supposed to have a text baseline to compare against, and we didn't find one. |
|
||||||
| `is_missing_video` | bool | **Optional, layout test specific.** If present and true, the test was supposed to have an image baseline to compare against and we didn't find one. |
|
| `is_missing_video` | bool | **Optional, web test specific.** If present and true, the test was supposed to have an image baseline to compare against and we didn't find one. |
|
||||||
| `is_testharness_test` | bool | **Optional, layout test specific.** If present, indicates that the layout test was written using the w3c's test harness and we don't necessarily have any baselines to compare against. |
|
| `is_testharness_test` | bool | **Optional, web test specific.** If present, indicates that the layout test was written using the w3c's test harness and we don't necessarily have any baselines to compare against. |
|
||||||
| `reftest_type` | string | **Optional, layout test specific.** If present, one of `==` or `!=` to indicate that the test is a "reference test" and the results were expected to match the reference or not match the reference, respectively (only used in layout tests). |
|
| `reftest_type` | string | **Optional, web test specific.** If present, one of `==` or `!=` to indicate that the test is a "reference test" and the results were expected to match the reference or not match the reference, respectively (only used in layout tests). |
|
||||||
|
|
||||||
## Test result types
|
## Test result types
|
||||||
|
|
||||||
Any test may fail in one of several different ways. There are a few generic
|
Any test may fail in one of several different ways. There are a few generic
|
||||||
types of failures, and the layout tests contain a few additional specialized
|
types of failures, and the web tests contain a few additional specialized
|
||||||
failure types.
|
failure types.
|
||||||
|
|
||||||
| Result type | Description |
|
| Result type | Description |
|
||||||
@@ -137,15 +137,15 @@ failure types.
|
|||||||
| `PASS` | The test ran as expected. |
|
| `PASS` | The test ran as expected. |
|
||||||
| `SKIP` | The test was not run. |
|
| `SKIP` | The test was not run. |
|
||||||
| `TIMEOUT` | The test hung (did not complete) and was aborted. |
|
| `TIMEOUT` | The test hung (did not complete) and was aborted. |
|
||||||
| `AUDIO` | **Layout test specific, deprecated.** The test is expected to produce audio output that doesn't match the expected result. Normally you will see `FAIL` instead. |
|
| `AUDIO` | **Web test specific, deprecated.** The test is expected to produce audio output that doesn't match the expected result. Normally you will see `FAIL` instead. |
|
||||||
| `IMAGE` | **Layout test specific, deprecated.** The test produces image (and possibly text output). The image output doesn't match what we'd expect, but the text output, if present, does. Normally you will see `FAIL` instead. |
|
| `IMAGE` | **Web test specific, deprecated.** The test produces image (and possibly text output). The image output doesn't match what we'd expect, but the text output, if present, does. Normally you will see `FAIL` instead. |
|
||||||
| `IMAGE+TEXT` | **Layout test specific, deprecated.** The test produces image and text output, both of which fail to match what we expect. Normally you will see `FAIL` instead. |
|
| `IMAGE+TEXT` | **Web test specific, deprecated.** The test produces image and text output, both of which fail to match what we expect. Normally you will see `FAIL` instead. |
|
||||||
| `LEAK` | **Layout test specific, deprecated.** Memory leaks were detected during the test execution. |
|
| `LEAK` | **Web test specific, deprecated.** Memory leaks were detected during the test execution. |
|
||||||
| `MISSING` | **Layout test specific, deprecated.** The test completed but we could not find an expected baseline to compare against. |
|
| `MISSING` | **Web test specific, deprecated.** The test completed but we could not find an expected baseline to compare against. |
|
||||||
| `NEEDSREBASELINE` | **Layout test specific, deprecated.** The expected test result is out of date and will be ignored (as above); the auto-rebaseline-bot will look for tests of this type and automatically update them. This should never show up as an `actual` result. |
|
| `NEEDSREBASELINE` | **Web test specific, deprecated.** The expected test result is out of date and will be ignored (as above); the auto-rebaseline-bot will look for tests of this type and automatically update them. This should never show up as an `actual` result. |
|
||||||
| `REBASELINE` | **Layout test specific, deprecated.** The expected test result is out of date and will be ignored (any result other than a crash or timeout will be considered as passing). This test result should only ever show up on local test runs, not on bots (it is forbidden to check in a TestExpectations file with this expectation). This should never show up as an "actual" result. |
|
| `REBASELINE` | **Web test specific, deprecated.** The expected test result is out of date and will be ignored (any result other than a crash or timeout will be considered as passing). This test result should only ever show up on local test runs, not on bots (it is forbidden to check in a TestExpectations file with this expectation). This should never show up as an "actual" result. |
|
||||||
| `SLOW` | **Layout test specific, deprecated.** The test is expected to take longer than normal to run. This should never appear as an `actual` result, but may (incorrectly) appear in the expected fields. |
|
| `SLOW` | **Web test specific, deprecated.** The test is expected to take longer than normal to run. This should never appear as an `actual` result, but may (incorrectly) appear in the expected fields. |
|
||||||
| `TEXT` | **Layout test specific, deprecated.** The test is expected to produce a text-only failure (the image, if present, will match). Normally you will see `FAIL` instead. |
|
| `TEXT` | **Web test specific, deprecated.** The test is expected to produce a text-only failure (the image, if present, will match). Normally you will see `FAIL` instead. |
|
||||||
|
|
||||||
Unexpected results, failures, and regressions are different things.
|
Unexpected results, failures, and regressions are different things.
|
||||||
|
|
||||||
@@ -161,7 +161,7 @@ A regression is a result that is both unexpected and a failure.
|
|||||||
|
|
||||||
## `full_results.json` and `failing_results.json`
|
## `full_results.json` and `failing_results.json`
|
||||||
|
|
||||||
The layout tests produce two different variants of the above file. The
|
The web tests produce two different variants of the above file. The
|
||||||
`full_results.json` file matches the above definition and contains every test
|
`full_results.json` file matches the above definition and contains every test
|
||||||
executed in the run. The `failing_results.json` file contains just the tests
|
executed in the run. The `failing_results.json` file contains just the tests
|
||||||
that produced unexpected results, so it is a subset of the `full_results.json`
|
that produced unexpected results, so it is a subset of the `full_results.json`
|
||||||
|
@@ -1,6 +1,6 @@
|
|||||||
# Using breakpad with content shell
|
# Using breakpad with content shell
|
||||||
|
|
||||||
When running layout tests, it is possible to use
|
When running web tests, it is possible to use
|
||||||
[breakpad](../../third_party/breakpad/) to capture stack traces on crashes while
|
[breakpad](../../third_party/breakpad/) to capture stack traces on crashes while
|
||||||
running without a debugger attached and with the sandbox enabled.
|
running without a debugger attached and with the sandbox enabled.
|
||||||
|
|
||||||
|
@@ -10,7 +10,7 @@ test suite where all browser engines are collaborating.
|
|||||||
|
|
||||||
Chromium has a 2-way import/export process with the upstream web-platform-tests
|
Chromium has a 2-way import/export process with the upstream web-platform-tests
|
||||||
repository, where tests are imported into
|
repository, where tests are imported into
|
||||||
[LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
|
[web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt)
|
||||||
and any changes to the imported tests are also exported to web-platform-tests.
|
and any changes to the imported tests are also exported to web-platform-tests.
|
||||||
|
|
||||||
See https://web-platform-tests.org/ for general documentation on
|
See https://web-platform-tests.org/ for general documentation on
|
||||||
@@ -22,7 +22,7 @@ web-platform-tests, including tips for writing and reviewing tests.
|
|||||||
|
|
||||||
Chromium has a [mirror](https://chromium.googlesource.com/external/w3c/web-platform-tests/)
|
Chromium has a [mirror](https://chromium.googlesource.com/external/w3c/web-platform-tests/)
|
||||||
of the GitHub repo and periodically imports a subset of the tests to
|
of the GitHub repo and periodically imports a subset of the tests to
|
||||||
run as part of the regular Blink layout test testing process.
|
run as part of the regular Blink web test testing process.
|
||||||
|
|
||||||
The goals of this process are to be able to run web-platform-tests unmodified
|
The goals of this process are to be able to run web-platform-tests unmodified
|
||||||
locally just as easily as we can run the Blink tests, and ensure that we are
|
locally just as easily as we can run the Blink tests, and ensure that we are
|
||||||
@@ -59,14 +59,14 @@ there are several possible causes, including:
|
|||||||
|
|
||||||
Because these tests are imported from the Web Platform tests, it is better
|
Because these tests are imported from the Web Platform tests, it is better
|
||||||
to have them in the repository (and marked failing) than not, so prefer to
|
to have them in the repository (and marked failing) than not, so prefer to
|
||||||
[add test expectations](layout_test_expectations.md) rather than reverting.
|
[add test expectations](web_test_expectations.md) rather than reverting.
|
||||||
However, if a huge number of tests are failing, please revert the CL so we
|
However, if a huge number of tests are failing, please revert the CL so we
|
||||||
can fix it manually.
|
can fix it manually.
|
||||||
|
|
||||||
### Automatic export process
|
### Automatic export process
|
||||||
|
|
||||||
If you upload a CL with any changes in
|
If you upload a CL with any changes in
|
||||||
[third_party/WebKit/LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt),
|
[third_party/blink/web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt),
|
||||||
once you add reviewers the exporter will create a provisional pull request with
|
once you add reviewers the exporter will create a provisional pull request with
|
||||||
those changes in the [upstream WPT GitHub repository](https://github.com/web-platform-tests/wpt/).
|
those changes in the [upstream WPT GitHub repository](https://github.com/web-platform-tests/wpt/).
|
||||||
|
|
||||||
@@ -101,7 +101,7 @@ For maintainers:
|
|||||||
### Skipped tests
|
### Skipped tests
|
||||||
|
|
||||||
We control which tests are imported via a file called
|
We control which tests are imported via a file called
|
||||||
[W3CImportExpectations](../../third_party/WebKit/LayoutTests/W3CImportExpectations),
|
[W3CImportExpectations](../../third_party/blink/web_tests/W3CImportExpectations),
|
||||||
which has a list of directories to skip while importing.
|
which has a list of directories to skip while importing.
|
||||||
|
|
||||||
In addition to the directories and tests explicitly skipped there, tests may
|
In addition to the directories and tests explicitly skipped there, tests may
|
||||||
@@ -159,21 +159,21 @@ the script.
|
|||||||
## Writing tests
|
## Writing tests
|
||||||
|
|
||||||
To contribute changes to web-platform-tests, just commit your changes directly
|
To contribute changes to web-platform-tests, just commit your changes directly
|
||||||
to [LayoutTests/external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
|
to [web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt)
|
||||||
and the changes will be automatically upstreamed within 24 hours.
|
and the changes will be automatically upstreamed within 24 hours.
|
||||||
|
|
||||||
Changes involving adding, removing or modifying tests can all be upstreamed.
|
Changes involving adding, removing or modifying tests can all be upstreamed.
|
||||||
Any changes outside of
|
Any changes outside of
|
||||||
[external/wpt](../../third_party/WebKit/LayoutTests/external/wpt) will not be
|
[external/wpt](../../third_party/blink/web_tests/external/wpt) will not be
|
||||||
upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`,
|
upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`,
|
||||||
will also not be upstreamed.
|
will also not be upstreamed.
|
||||||
|
|
||||||
Running the layout tests will automatically regenerate MANIFEST.json to pick up
|
Running the web tests will automatically regenerate MANIFEST.json to pick up
|
||||||
any local modifications.
|
any local modifications.
|
||||||
|
|
||||||
Most tests are written using testharness.js, see
|
Most tests are written using testharness.js, see
|
||||||
[Writing Layout Tests](./writing_layout_tests.md) and
|
[Writing Web Tests](./writing_web_tests.md) and
|
||||||
[Layout Tests Tips](./layout_tests_tips.md) for general guidelines.
|
[Web Tests Tips](./web_tests_tips.md) for general guidelines.
|
||||||
|
|
||||||
### Write tests against specifications
|
### Write tests against specifications
|
||||||
|
|
||||||
@@ -193,22 +193,22 @@ and what needs to be resolved to make it non-tentative.
|
|||||||
[testdriver.js](https://web-platform-tests.org/writing-tests/testdriver.html)
|
[testdriver.js](https://web-platform-tests.org/writing-tests/testdriver.html)
|
||||||
provides a means to automate tests that cannot be written purely using web
|
provides a means to automate tests that cannot be written purely using web
|
||||||
platform APIs, similar to `internals.*` and `eventSender.*` in regular Blink
|
platform APIs, similar to `internals.*` and `eventSender.*` in regular Blink
|
||||||
layout tests.
|
web tests.
|
||||||
|
|
||||||
If no testdriver.js API exists, check if it's a
|
If no testdriver.js API exists, check if it's a
|
||||||
[known issue](https://github.com/web-platform-tests/wpt/labels/testdriver.js)
|
[known issue](https://github.com/web-platform-tests/wpt/labels/testdriver.js)
|
||||||
and otherwise consider filing a new issue.
|
and otherwise consider filing a new issue.
|
||||||
|
|
||||||
An alternative is to write manual tests that are automated with scripts from
|
An alternative is to write manual tests that are automated with scripts from
|
||||||
[wpt_automation](../../third_party/WebKit/LayoutTests/external/wpt_automation).
|
[wpt_automation](../../third_party/blink/web_tests/external/wpt_automation).
|
||||||
Injection of JS in manual tests is determined by `loadAutomationScript` in
|
Injection of JS in manual tests is determined by `loadAutomationScript` in
|
||||||
[testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js).
|
[testharnessreport.js](../../third_party/blink/web_tests/resources/testharnessreport.js).
|
||||||
|
|
||||||
Such tests still require case-by-case automation to run for other browser
|
Such tests still require case-by-case automation to run for other browser
|
||||||
engines, but are more valuable than purely manual tests.
|
engines, but are more valuable than purely manual tests.
|
||||||
|
|
||||||
Manual tests that have no automation are still imported, but skipped in
|
Manual tests that have no automation are still imported, but skipped in
|
||||||
[NeverFixTests](../../third_party/WebKit/LayoutTests/NeverFixTests); see
|
[NeverFixTests](../../third_party/blink/web_tests/NeverFixTests); see
|
||||||
[issue 738489](https://crbug.com/738489).
|
[issue 738489](https://crbug.com/738489).
|
||||||
|
|
||||||
### Adding new top-level directories
|
### Adding new top-level directories
|
||||||
@@ -244,15 +244,15 @@ https://web-platform-tests.org/appendix/github-intro.html.
|
|||||||
|
|
||||||
## Running tests
|
## Running tests
|
||||||
|
|
||||||
Same as Blink layout tests, you can use
|
Same as Blink web tests, you can use
|
||||||
[`run_web_tests.py`](layout_tests.md#running-the-tests) to run any WPT test.
|
[`run_web_tests.py`](web_tests.md#running-the-tests) to run any WPT test.
|
||||||
|
|
||||||
One thing to note is that glob patterns for WPT tests are not yet supported.
|
One thing to note is that glob patterns for WPT tests are not yet supported.
|
||||||
|
|
||||||
## Reviewing tests
|
## Reviewing tests
|
||||||
|
|
||||||
Anyone who can review code and tests in Chromium can also review changes in
|
Anyone who can review code and tests in Chromium can also review changes in
|
||||||
[external/wpt](../../third_party/WebKit/LayoutTests/external/wpt)
|
[external/wpt](../../third_party/blink/web_tests/external/wpt)
|
||||||
that will be automatically upstreamed. There will be no additional review in
|
that will be automatically upstreamed. There will be no additional review in
|
||||||
web-platform-tests as part of the export process.
|
web-platform-tests as part of the export process.
|
||||||
|
|
||||||
|
@@ -1,8 +1,8 @@
|
|||||||
# Layout Test Baseline Fallback
|
# Web Test Baseline Fallback
|
||||||
|
|
||||||
|
|
||||||
*** promo
|
*** promo
|
||||||
Read [Layout Test Expectations and Baselines](layout_test_expectations.md) first
|
Read [Web Test Expectations and Baselines](web_test_expectations.md) first
|
||||||
if you have not.
|
if you have not.
|
||||||
***
|
***
|
||||||
|
|
||||||
@@ -18,8 +18,8 @@ optimization and rebaselining.
|
|||||||
## Terminology
|
## Terminology
|
||||||
|
|
||||||
* **Root directory**:
|
* **Root directory**:
|
||||||
[`//src/third_party/WebKit/LayoutTests`](../../third_party/WebKit/LayoutTests)
|
[`//src/third_party/blink/web_tests`](../../third_party/blink/web_tests)
|
||||||
is the root directory (of all the layout tests and baselines). All relative
|
is the root directory (of all the web tests and baselines). All relative
|
||||||
paths in this document start from this directory.
|
paths in this document start from this directory.
|
||||||
* **Test name**: the name of a test is its relative path from the root
|
* **Test name**: the name of a test is its relative path from the root
|
||||||
directory (e.g. `html/dom/foo/bar.html`).
|
directory (e.g. `html/dom/foo/bar.html`).
|
||||||
@@ -29,10 +29,10 @@ optimization and rebaselining.
|
|||||||
`virtual/gpu/html/dom/foo/bar.html` is the virtual variant of
|
`virtual/gpu/html/dom/foo/bar.html` is the virtual variant of
|
||||||
`html/dom/foo/bar.html` in the `gpu` suite. Only the latter file exists on
|
`html/dom/foo/bar.html` in the `gpu` suite. Only the latter file exists on
|
||||||
disk, and is called the base of the virtual test. See
|
disk, and is called the base of the virtual test. See
|
||||||
[Layout Tests#Testing Runtime Flags](layout_tests.md#testing-runtime-flags)
|
[Web Tests#Testing Runtime Flags](web_tests.md#testing-runtime-flags)
|
||||||
for more details.
|
for more details.
|
||||||
* **Platform directory**: each directory under
|
* **Platform directory**: each directory under
|
||||||
[`platform/`](../../third_party/WebKit/LayoutTests/platform) is a platform
|
[`platform/`](../../third_party/blink/web_tests/platform) is a platform
|
||||||
directory that contains baselines (no tests) for that platform. Directory
|
directory that contains baselines (no tests) for that platform. Directory
|
||||||
names are in the form of `PLATFORM-VERSION` (e.g. `mac-mac10.12`), except
|
names are in the form of `PLATFORM-VERSION` (e.g. `mac-mac10.12`), except
|
||||||
for the latest version of a platform which is just `PLATFORM` (e.g. `mac`).
|
for the latest version of a platform which is just `PLATFORM` (e.g. `mac`).
|
@@ -1,14 +1,14 @@
|
|||||||
# Layout Test Expectations and Baselines
|
# Web Test Expectations and Baselines
|
||||||
|
|
||||||
|
|
||||||
The primary function of the LayoutTests is as a regression test suite; this
|
The primary function of the web tests is as a regression test suite; this
|
||||||
means that, while we care about whether a page is being rendered correctly, we
|
means that, while we care about whether a page is being rendered correctly, we
|
||||||
care more about whether the page is being rendered the way we expect it to. In
|
care more about whether the page is being rendered the way we expect it to. In
|
||||||
other words, we look more for changes in behavior than we do for correctness.
|
other words, we look more for changes in behavior than we do for correctness.
|
||||||
|
|
||||||
[TOC]
|
[TOC]
|
||||||
|
|
||||||
All layout tests have "expected results", or "baselines", which may be one of
|
All web tests have "expected results", or "baselines", which may be one of
|
||||||
several forms. The test may produce one or more of:
|
several forms. The test may produce one or more of:
|
||||||
|
|
||||||
* A text file containing JavaScript log messages.
|
* A text file containing JavaScript log messages.
|
||||||
@@ -16,12 +16,12 @@ several forms. The test may produce one or more of:
|
|||||||
* A screen capture of the rendered page as a PNG file.
|
* A screen capture of the rendered page as a PNG file.
|
||||||
* WAV files of the audio output, for WebAudio tests.
|
* WAV files of the audio output, for WebAudio tests.
|
||||||
|
|
||||||
For any of these types of tests, baselines are checked into the LayoutTests
|
For any of these types of tests, baselines are checked into the web_tests
|
||||||
directory. The filename of a baseline is the same as that of the corresponding
|
directory. The filename of a baseline is the same as that of the corresponding
|
||||||
test, but the extension is replaced with `-expected.{txt,png,wav}` (depending on
|
test, but the extension is replaced with `-expected.{txt,png,wav}` (depending on
|
||||||
the type of test output). Baselines usually live alongside tests, with the
|
the type of test output). Baselines usually live alongside tests, with the
|
||||||
exception when baselines vary by platforms; read
|
exception when baselines vary by platforms; read
|
||||||
[Layout Test Baseline Fallback](layout_test_baseline_fallback.md) for more
|
[Web Test Baseline Fallback](web_test_baseline_fallback.md) for more
|
||||||
details.
|
details.
|
||||||
|
|
||||||
Lastly, we also support the concept of "reference tests", which check that two
|
Lastly, we also support the concept of "reference tests", which check that two
|
||||||
@@ -49,10 +49,10 @@ lines to TestExpectations.
|
|||||||
|
|
||||||
Bugs at [crbug.com](https://crbug.com) should track fixing incorrect behavior,
|
Bugs at [crbug.com](https://crbug.com) should track fixing incorrect behavior,
|
||||||
not lines in
|
not lines in
|
||||||
[TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations). If a
|
[TestExpectations](../../third_party/blink/web_tests/TestExpectations). If a
|
||||||
test is never supposed to pass (e.g. it's testing Windows-specific behavior, so
|
test is never supposed to pass (e.g. it's testing Windows-specific behavior, so
|
||||||
can't ever pass on Linux/Mac), move it to the
|
can't ever pass on Linux/Mac), move it to the
|
||||||
[NeverFixTests](../../third_party/WebKit/LayoutTests/NeverFixTests) file. That
|
[NeverFixTests](../../third_party/blink/web_tests/NeverFixTests) file. That
|
||||||
gets it out of the way of the rest of the project.
|
gets it out of the way of the rest of the project.
|
||||||
|
|
||||||
There are some cases where you can't rebaseline and, unfortunately, we don't
|
There are some cases where you can't rebaseline and, unfortunately, we don't
|
||||||
@@ -112,7 +112,7 @@ results from try jobs, by using the command-tool
|
|||||||
|
|
||||||
This way, the new baselines can be reviewed along with the changes, which helps
|
This way, the new baselines can be reviewed along with the changes, which helps
|
||||||
the reviewer verify that the new baselines are correct. It also means that there
|
the reviewer verify that the new baselines are correct. It also means that there
|
||||||
is no period of time when the layout test results are ignored.
|
is no period of time when the web test results are ignored.
|
||||||
|
|
||||||
#### Options
|
#### Options
|
||||||
|
|
||||||
@@ -133,36 +133,36 @@ depends on its arguments.
|
|||||||
|
|
||||||
## Kinds of expectations files
|
## Kinds of expectations files
|
||||||
|
|
||||||
* [TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations): The
|
* [TestExpectations](../../third_party/blink/web_tests/TestExpectations): The
|
||||||
main test failure suppression file. In theory, this should be used for
|
main test failure suppression file. In theory, this should be used for
|
||||||
temporarily marking tests as flaky.
|
temporarily marking tests as flaky.
|
||||||
* [ASANExpectations](../../third_party/WebKit/LayoutTests/ASANExpectations):
|
* [ASANExpectations](../../third_party/blink/web_tests/ASANExpectations):
|
||||||
Tests that fail under ASAN.
|
Tests that fail under ASAN.
|
||||||
* [LeakExpectations](../../third_party/WebKit/LayoutTests/LeakExpectations):
|
* [LeakExpectations](../../third_party/blink/web_tests/LeakExpectations):
|
||||||
Tests that have memory leaks under the leak checker.
|
Tests that have memory leaks under the leak checker.
|
||||||
* [MSANExpectations](../../third_party/WebKit/LayoutTests/MSANExpectations):
|
* [MSANExpectations](../../third_party/blink/web_tests/MSANExpectations):
|
||||||
Tests that fail under MSAN.
|
Tests that fail under MSAN.
|
||||||
* [NeverFixTests](../../third_party/WebKit/LayoutTests/NeverFixTests): Tests
|
* [NeverFixTests](../../third_party/blink/web_tests/NeverFixTests): Tests
|
||||||
that we never intend to fix (e.g. a test for Windows-specific behavior will
|
that we never intend to fix (e.g. a test for Windows-specific behavior will
|
||||||
never be fixed on Linux/Mac). Tests that will never pass on any platform
|
never be fixed on Linux/Mac). Tests that will never pass on any platform
|
||||||
should just be deleted, though.
|
should just be deleted, though.
|
||||||
* [SlowTests](../../third_party/WebKit/LayoutTests/SlowTests): Tests that take
|
* [SlowTests](../../third_party/blink/web_tests/SlowTests): Tests that take
|
||||||
longer than the usual timeout to run. Slow tests are given 5x the usual
|
longer than the usual timeout to run. Slow tests are given 5x the usual
|
||||||
timeout.
|
timeout.
|
||||||
* [SmokeTests](../../third_party/WebKit/LayoutTests/SmokeTests): A small subset
|
* [SmokeTests](../../third_party/blink/web_tests/SmokeTests): A small subset
|
||||||
of tests that we run on the Android bot.
|
of tests that we run on the Android bot.
|
||||||
* [StaleTestExpectations](../../third_party/WebKit/LayoutTests/StaleTestExpectations):
|
* [StaleTestExpectations](../../third_party/blink/web_tests/StaleTestExpectations):
|
||||||
Platform-specific lines that have been in TestExpectations for many months.
|
Platform-specific lines that have been in TestExpectations for many months.
|
||||||
They're moved here to get them out of the way of people doing rebaselines
|
They're moved here to get them out of the way of people doing rebaselines
|
||||||
since they're clearly not getting fixed anytime soon.
|
since they're clearly not getting fixed anytime soon.
|
||||||
* [W3CImportExpectations](../../third_party/WebKit/LayoutTests/W3CImportExpectations):
|
* [W3CImportExpectations](../../third_party/blink/web_tests/W3CImportExpectations):
|
||||||
A record of which W3C tests should be imported or skipped.
|
A record of which W3C tests should be imported or skipped.
|
||||||
|
|
||||||
### Flag-specific expectations files
|
### Flag-specific expectations files
|
||||||
|
|
||||||
It is possible to handle tests that only fail when run with a particular flag
|
It is possible to handle tests that only fail when run with a particular flag
|
||||||
being passed to `content_shell`. See
|
being passed to `content_shell`. See
|
||||||
[LayoutTests/FlagExpectations/README.txt](../../third_party/WebKit/LayoutTests/FlagExpectations/README.txt)
|
[web_tests/FlagExpectations/README.txt](../../third_party/blink/web_tests/FlagExpectations/README.txt)
|
||||||
for more.
|
for more.
|
||||||
|
|
||||||
## Updating the expectations files
|
## Updating the expectations files
|
||||||
@@ -233,14 +233,14 @@ Bug(darin) [ Mac10.9 Debug ] fast/html/keygen.html [ Skip ]
|
|||||||
* `WontFix` implies `Skip` and also indicates that we don't have any plans to
|
* `WontFix` implies `Skip` and also indicates that we don't have any plans to
|
||||||
make the test pass.
|
make the test pass.
|
||||||
* `WontFix` lines always go in the
|
* `WontFix` lines always go in the
|
||||||
[NeverFixTests file](../../third_party/WebKit/LayoutTests/NeverFixTests) as
|
[NeverFixTests file](../../third_party/blink/web_tests/NeverFixTests) as
|
||||||
we never intend to fix them. These are just for tests that only apply to some
|
we never intend to fix them. These are just for tests that only apply to some
|
||||||
subset of the platforms we support.
|
subset of the platforms we support.
|
||||||
* `WontFix` and `Skip` must be used by themselves and cannot be specified
|
* `WontFix` and `Skip` must be used by themselves and cannot be specified
|
||||||
alongside `Crash` or another expectation keyword.
|
alongside `Crash` or another expectation keyword.
|
||||||
* `Slow` causes the test runner to give the test 5x the usual time limit to run.
|
* `Slow` causes the test runner to give the test 5x the usual time limit to run.
|
||||||
`Slow` lines go in the
|
`Slow` lines go in the
|
||||||
[SlowTests file ](../../third_party/WebKit/LayoutTests/SlowTests). A given
|
[SlowTests file ](../../third_party/blink/web_tests/SlowTests). A given
|
||||||
line cannot have both Slow and Timeout.
|
line cannot have both Slow and Timeout.
|
||||||
|
|
||||||
Also, when parsing the file, we use two rules to figure out if an expectation
|
Also, when parsing the file, we use two rules to figure out if an expectation
|
@@ -1,32 +1,32 @@
|
|||||||
# Layout Tests
|
# Web Tests (formly known as "Layout Tests" "LayoutTests")
|
||||||
|
|
||||||
Layout tests are used by Blink to test many components, including but not
|
Web tests are used by Blink to test many components, including but not
|
||||||
limited to layout and rendering. In general, layout tests involve loading pages
|
limited to layout and rendering. In general, web tests involve loading pages
|
||||||
in a test renderer (`content_shell`) and comparing the rendered output or
|
in a test renderer (`content_shell`) and comparing the rendered output or
|
||||||
JavaScript output against an expected output file.
|
JavaScript output against an expected output file.
|
||||||
|
|
||||||
This document covers running and debugging existing layout tests. See the
|
This document covers running and debugging existing web tests. See the
|
||||||
[Writing Layout Tests documentation](./writing_layout_tests.md) if you find
|
[Writing Web Tests documentation](./writing_web_tests.md) if you find
|
||||||
yourself writing layout tests.
|
yourself writing web tests.
|
||||||
|
|
||||||
Note that we're in process of changing the term "layout tests" to "web tests".
|
Note that we changed the term "layout tests" to "web tests".
|
||||||
Please assume these terms mean the identical stuff. We also call it as
|
Please assume these terms mean the identical stuff. We also call it as
|
||||||
"WebKit tests" and "WebKit layout tests".
|
"WebKit tests" and "WebKit layout tests".
|
||||||
|
|
||||||
[TOC]
|
[TOC]
|
||||||
|
|
||||||
## Running Layout Tests
|
## Running Web Tests
|
||||||
|
|
||||||
### Initial Setup
|
### Initial Setup
|
||||||
|
|
||||||
Before you can run the layout tests, you need to build the `blink_tests` target
|
Before you can run the web tests, you need to build the `blink_tests` target
|
||||||
to get `content_shell` and all of the other needed binaries.
|
to get `content_shell` and all of the other needed binaries.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
autoninja -C out/Default blink_tests
|
autoninja -C out/Default blink_tests
|
||||||
```
|
```
|
||||||
|
|
||||||
On **Android** (layout test support
|
On **Android** (web test support
|
||||||
[currently limited to KitKat and earlier](https://crbug.com/567947)) you need to
|
[currently limited to KitKat and earlier](https://crbug.com/567947)) you need to
|
||||||
build and install `content_shell_apk` instead. See also:
|
build and install `content_shell_apk` instead. See also:
|
||||||
[Android Build Instructions](../android_build_instructions.md).
|
[Android Build Instructions](../android_build_instructions.md).
|
||||||
@@ -67,17 +67,17 @@ python third_party/blink/tools/run_web_tests.py -t android --android
|
|||||||
```
|
```
|
||||||
|
|
||||||
Tests marked as `[ Skip ]` in
|
Tests marked as `[ Skip ]` in
|
||||||
[TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations)
|
[TestExpectations](../../third_party/blink/web_tests/TestExpectations)
|
||||||
won't be run at all, generally because they cause some intractable tool error.
|
won't be run at all, generally because they cause some intractable tool error.
|
||||||
To force one of them to be run, either rename that file or specify the skipped
|
To force one of them to be run, either rename that file or specify the skipped
|
||||||
test as the only one on the command line (see below). Read the
|
test as the only one on the command line (see below). Read the
|
||||||
[Layout Test Expectations documentation](./layout_test_expectations.md) to learn
|
[Web Test Expectations documentation](./web_test_expectations.md) to learn
|
||||||
more about TestExpectations and related files.
|
more about TestExpectations and related files.
|
||||||
|
|
||||||
*** promo
|
*** promo
|
||||||
Currently only the tests listed in
|
Currently only the tests listed in
|
||||||
[SmokeTests](../../third_party/WebKit/LayoutTests/SmokeTests)
|
[SmokeTests](../../third_party/blink/web_tests/SmokeTests)
|
||||||
are run on the Android bots, since running all layout tests takes too long on
|
are run on the Android bots, since running all web tests takes too long on
|
||||||
Android (and may still have some infrastructure issues). Most developers focus
|
Android (and may still have some infrastructure issues). Most developers focus
|
||||||
their Blink testing on Linux. We rely on the fact that the Linux and Android
|
their Blink testing on Linux. We rely on the fact that the Linux and Android
|
||||||
behavior is nearly identical for scenarios outside those covered by the smoke
|
behavior is nearly identical for scenarios outside those covered by the smoke
|
||||||
@@ -85,8 +85,8 @@ tests.
|
|||||||
***
|
***
|
||||||
|
|
||||||
To run only some of the tests, specify their directories or filenames as
|
To run only some of the tests, specify their directories or filenames as
|
||||||
arguments to `run_web_tests.py` relative to the layout test directory
|
arguments to `run_web_tests.py` relative to the web test directory
|
||||||
(`src/third_party/WebKit/LayoutTests`). For example, to run the fast form tests,
|
(`src/third_party/blink/web_tests`). For example, to run the fast form tests,
|
||||||
use:
|
use:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -100,7 +100,7 @@ python third_party/blink/tools/run_web_tests.py fast/fo\*
|
|||||||
```
|
```
|
||||||
|
|
||||||
*** promo
|
*** promo
|
||||||
Example: To run the layout tests with a debug build of `content_shell`, but only
|
Example: To run the web tests with a debug build of `content_shell`, but only
|
||||||
test the SVG tests and run pixel tests, you would run:
|
test the SVG tests and run pixel tests, you would run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -119,7 +119,7 @@ as in:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
out/Default/content_shell.exe --run-web-tests --no-sandbox \
|
out/Default/content_shell.exe --run-web-tests --no-sandbox \
|
||||||
c:/chrome/src/third_party/WebKit/LayoutTests/fast/forms/001.html
|
c:/chrome/src/third_party/blink/web_tests/fast/forms/001.html
|
||||||
```
|
```
|
||||||
|
|
||||||
but this requires a manual diff against expected results, because the shell
|
but this requires a manual diff against expected results, because the shell
|
||||||
@@ -135,8 +135,8 @@ python run_web_tests.py --help
|
|||||||
**Linux Note:** We try to match the Windows render tree output exactly by
|
**Linux Note:** We try to match the Windows render tree output exactly by
|
||||||
matching font metrics and widget metrics. If there's a difference in the render
|
matching font metrics and widget metrics. If there's a difference in the render
|
||||||
tree output, we should see if we can avoid rebaselining by improving our font
|
tree output, we should see if we can avoid rebaselining by improving our font
|
||||||
metrics. For additional information on Linux Layout Tests, please see
|
metrics. For additional information on Linux web tests, please see
|
||||||
[docs/layout_tests_linux.md](../layout_tests_linux.md).
|
[docs/web_tests_linux.md](../web_tests_linux.md).
|
||||||
***
|
***
|
||||||
|
|
||||||
*** note
|
*** note
|
||||||
@@ -174,7 +174,7 @@ a diff of the actual and expected results, into
|
|||||||
summary and link to the results/diffs.
|
summary and link to the results/diffs.
|
||||||
|
|
||||||
The expected results for tests are in the
|
The expected results for tests are in the
|
||||||
`src/third_party/WebKit/LayoutTests/platform` or alongside their respective
|
`src/third_party/blink/web_tests/platform` or alongside their respective
|
||||||
tests.
|
tests.
|
||||||
|
|
||||||
*** note
|
*** note
|
||||||
@@ -190,14 +190,14 @@ A row of dots in the script's output indicates one or more tests that passed.
|
|||||||
## Test expectations
|
## Test expectations
|
||||||
|
|
||||||
The
|
The
|
||||||
[TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations) file (and related
|
[TestExpectations](../../third_party/blink/web_tests/TestExpectations) file (and related
|
||||||
files) contains the list of all known layout test failures. See the
|
files) contains the list of all known web test failures. See the
|
||||||
[Layout Test Expectations documentation](./layout_test_expectations.md) for more
|
[Web Test Expectations documentation](./web_test_expectations.md) for more
|
||||||
on this.
|
on this.
|
||||||
|
|
||||||
## Testing Runtime Flags
|
## Testing Runtime Flags
|
||||||
|
|
||||||
There are two ways to run layout tests with additional command-line arguments:
|
There are two ways to run web tests with additional command-line arguments:
|
||||||
|
|
||||||
* Using `--additional-driver-flag`:
|
* Using `--additional-driver-flag`:
|
||||||
|
|
||||||
@@ -209,12 +209,12 @@ There are two ways to run layout tests with additional command-line arguments:
|
|||||||
content_shell binary.
|
content_shell binary.
|
||||||
|
|
||||||
It will also look for flag-specific expectations in
|
It will also look for flag-specific expectations in
|
||||||
`LayoutTests/FlagExpectations/blocking-repaint`, if this file exists. The
|
`web_tests/FlagExpectations/blocking-repaint`, if this file exists. The
|
||||||
suppressions in this file override the main TestExpectations file.
|
suppressions in this file override the main TestExpectations file.
|
||||||
|
|
||||||
* Using a *virtual test suite* defined in
|
* Using a *virtual test suite* defined in
|
||||||
[LayoutTests/VirtualTestSuites](../../third_party/WebKit/LayoutTests/VirtualTestSuites).
|
[web_tests/VirtualTestSuites](../../third_party/blink/web_tests/VirtualTestSuites).
|
||||||
A virtual test suite runs a subset of layout tests under a specific path with
|
A virtual test suite runs a subset of web tests under a specific path with
|
||||||
additional flags. For example, you could test a (hypothetical) new mode for
|
additional flags. For example, you could test a (hypothetical) new mode for
|
||||||
repainting using the following virtual test suite:
|
repainting using the following virtual test suite:
|
||||||
|
|
||||||
@@ -228,7 +228,7 @@ There are two ways to run layout tests with additional command-line arguments:
|
|||||||
|
|
||||||
This will create new "virtual" tests of the form
|
This will create new "virtual" tests of the form
|
||||||
`virtual/blocking_repaint/fast/repaint/...` which correspond to the files
|
`virtual/blocking_repaint/fast/repaint/...` which correspond to the files
|
||||||
under `LayoutTests/fast/repaint` and pass `--blocking-repaint` to
|
under `web_tests/fast/repaint` and pass `--blocking-repaint` to
|
||||||
content_shell when they are run.
|
content_shell when they are run.
|
||||||
|
|
||||||
These virtual tests exist in addition to the original `fast/repaint/...`
|
These virtual tests exist in addition to the original `fast/repaint/...`
|
||||||
@@ -269,7 +269,7 @@ Consider the following when choosing between them:
|
|||||||
|
|
||||||
## Tracking Test Failures
|
## Tracking Test Failures
|
||||||
|
|
||||||
All bugs, associated with layout test failures must have the
|
All bugs, associated with web test failures must have the
|
||||||
[Test-Layout](https://crbug.com/?q=label:Test-Layout) label. Depending on how
|
[Test-Layout](https://crbug.com/?q=label:Test-Layout) label. Depending on how
|
||||||
much you know about the bug, assign the status accordingly:
|
much you know about the bug, assign the status accordingly:
|
||||||
|
|
||||||
@@ -279,7 +279,7 @@ much you know about the bug, assign the status accordingly:
|
|||||||
* **Available** -- You know the root cause of the issue.
|
* **Available** -- You know the root cause of the issue.
|
||||||
* **Assigned** or **Started** -- You will fix this issue.
|
* **Assigned** or **Started** -- You will fix this issue.
|
||||||
|
|
||||||
When creating a new layout test bug, please set the following properties:
|
When creating a new web test bug, please set the following properties:
|
||||||
|
|
||||||
* Components: a sub-component of Blink
|
* Components: a sub-component of Blink
|
||||||
* OS: **All** (or whichever OS the failure is on)
|
* OS: **All** (or whichever OS the failure is on)
|
||||||
@@ -290,9 +290,9 @@ When creating a new layout test bug, please set the following properties:
|
|||||||
You can also use the _Layout Test Failure_ template, which pre-sets these
|
You can also use the _Layout Test Failure_ template, which pre-sets these
|
||||||
labels for you.
|
labels for you.
|
||||||
|
|
||||||
## Debugging Layout Tests
|
## Debugging Web Tests
|
||||||
|
|
||||||
After the layout tests run, you should get a summary of tests that pass or
|
After the web tests run, you should get a summary of tests that pass or
|
||||||
fail. If something fails unexpectedly (a new regression), you will get a
|
fail. If something fails unexpectedly (a new regression), you will get a
|
||||||
`content_shell` window with a summary of the unexpected failures. Or you might
|
`content_shell` window with a summary of the unexpected failures. Or you might
|
||||||
have a failing test in mind to investigate. In any case, here are some steps and
|
have a failing test in mind to investigate. In any case, here are some steps and
|
||||||
@@ -310,7 +310,7 @@ tips for finding the problem.
|
|||||||
ask.
|
ask.
|
||||||
* Some tests only work properly in content_shell, not Chrome, because they
|
* Some tests only work properly in content_shell, not Chrome, because they
|
||||||
rely on extra APIs exposed there.
|
rely on extra APIs exposed there.
|
||||||
* Some tests only work properly when they're run in the layout-test
|
* Some tests only work properly when they're run in the web-test
|
||||||
framework, not when they're loaded into content_shell directly. The test
|
framework, not when they're loaded into content_shell directly. The test
|
||||||
should mention that in its visible text, but not all do. So try that too.
|
should mention that in its visible text, but not all do. So try that too.
|
||||||
See "Running the tests", above.
|
See "Running the tests", above.
|
||||||
@@ -342,15 +342,15 @@ tips for finding the problem.
|
|||||||
tracking down the issue.
|
tracking down the issue.
|
||||||
* Otherwise, you're running in a standard message loop just like in Chrome.
|
* Otherwise, you're running in a standard message loop just like in Chrome.
|
||||||
If you have no other information, set a breakpoint on page load.
|
If you have no other information, set a breakpoint on page load.
|
||||||
* If your test only works in full layout-test mode, or if you find it simpler to
|
* If your test only works in full web-test mode, or if you find it simpler to
|
||||||
debug without all the overhead of an interactive session, start the
|
debug without all the overhead of an interactive session, start the
|
||||||
content_shell with the command-line flag `--run-web-tests`, followed by the
|
content_shell with the command-line flag `--run-web-tests`, followed by the
|
||||||
URL (`file:` or `http:`) to your test. More information about running layout tests
|
URL (`file:` or `http:`) to your test. More information about running web tests
|
||||||
in content_shell can be found [here](./layout_tests_in_content_shell.md).
|
in content_shell can be found [here](./web_tests_in_content_shell.md).
|
||||||
* In VS, you can do this in the Debugging section of the content_shell
|
* In VS, you can do this in the Debugging section of the content_shell
|
||||||
project Properties.
|
project Properties.
|
||||||
* Now you're running with exactly the same API, theme, and other setup that
|
* Now you're running with exactly the same API, theme, and other setup that
|
||||||
the layout tests use.
|
the web tests use.
|
||||||
* Again, if your test contains a JS call, DOM manipulation, or other
|
* Again, if your test contains a JS call, DOM manipulation, or other
|
||||||
distinctive piece of code that you think is failing, search for that in
|
distinctive piece of code that you think is failing, search for that in
|
||||||
the Chrome solution. That's a good place to put a starting breakpoint to
|
the Chrome solution. That's a good place to put a starting breakpoint to
|
||||||
@@ -358,7 +358,7 @@ tips for finding the problem.
|
|||||||
* If you can't find any better place to set a breakpoint, start at the
|
* If you can't find any better place to set a breakpoint, start at the
|
||||||
`TestShell::RunFileTest()` call in `content_shell_main.cc`, or at
|
`TestShell::RunFileTest()` call in `content_shell_main.cc`, or at
|
||||||
`shell->LoadURL() within RunFileTest()` in `content_shell_win.cc`.
|
`shell->LoadURL() within RunFileTest()` in `content_shell_win.cc`.
|
||||||
* Debug as usual. Once you've gotten this far, the failing layout test is just a
|
* Debug as usual. Once you've gotten this far, the failing web test is just a
|
||||||
(hopefully) reduced test case that exposes a problem.
|
(hopefully) reduced test case that exposes a problem.
|
||||||
|
|
||||||
### Debugging HTTP Tests
|
### Debugging HTTP Tests
|
||||||
@@ -370,9 +370,9 @@ cd src/third_party/blink/tools
|
|||||||
python run_blink_httpd.py
|
python run_blink_httpd.py
|
||||||
```
|
```
|
||||||
|
|
||||||
The layout tests are served from `http://127.0.0.1:8000/`. For example, to
|
The web tests are served from `http://127.0.0.1:8000/`. For example, to
|
||||||
run the test
|
run the test
|
||||||
`LayoutTest/http/tests/serviceworker/chromium/service-worker-allowed.html`,
|
`web_tests/http/tests/serviceworker/chromium/service-worker-allowed.html`,
|
||||||
navigate to
|
navigate to
|
||||||
`http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
|
`http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
|
||||||
tests behave differently if you go to `127.0.0.1` vs. `localhost`, so use
|
tests behave differently if you go to `127.0.0.1` vs. `localhost`, so use
|
||||||
@@ -382,7 +382,7 @@ To kill the server, hit any key on the terminal where `run_blink_httpd.py` is
|
|||||||
running, use `taskkill` or the Task Manager on Windows, or `killall` or
|
running, use `taskkill` or the Task Manager on Windows, or `killall` or
|
||||||
Activity Monitor on macOS.
|
Activity Monitor on macOS.
|
||||||
|
|
||||||
The test server sets up an alias to the `LayoutTests/resources` directory. For
|
The test server sets up an alias to the `web_tests/resources` directory. For
|
||||||
example, in HTTP tests, you can access the testing framework using
|
example, in HTTP tests, you can access the testing framework using
|
||||||
`src="/js-test-resources/js-test.js"`.
|
`src="/js-test-resources/js-test.js"`.
|
||||||
|
|
||||||
@@ -441,7 +441,7 @@ NOTE: If the test is an html file, this means it's a legacy test so you need to
|
|||||||
## Bisecting Regressions
|
## Bisecting Regressions
|
||||||
|
|
||||||
You can use [`git bisect`](https://git-scm.com/docs/git-bisect) to find which
|
You can use [`git bisect`](https://git-scm.com/docs/git-bisect) to find which
|
||||||
commit broke (or fixed!) a layout test in a fully automated way. Unlike
|
commit broke (or fixed!) a web test in a fully automated way. Unlike
|
||||||
[bisect-builds.py](http://dev.chromium.org/developers/bisect-builds-py), which
|
[bisect-builds.py](http://dev.chromium.org/developers/bisect-builds-py), which
|
||||||
downloads pre-built Chromium binaries, `git bisect` operates on your local
|
downloads pre-built Chromium binaries, `git bisect` operates on your local
|
||||||
checkout, so it can run tests with `content_shell`.
|
checkout, so it can run tests with `content_shell`.
|
||||||
@@ -449,7 +449,7 @@ checkout, so it can run tests with `content_shell`.
|
|||||||
Bisecting can take several hours, but since it is fully automated you can leave
|
Bisecting can take several hours, but since it is fully automated you can leave
|
||||||
it running overnight and view the results the next day.
|
it running overnight and view the results the next day.
|
||||||
|
|
||||||
To set up an automated bisect of a layout test regression, create a script like
|
To set up an automated bisect of a web test regression, create a script like
|
||||||
this:
|
this:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -461,7 +461,7 @@ autoninja -C out/Debug -j100 blink_tests || exit 125
|
|||||||
|
|
||||||
third_party/blink/tools/run_web_tests.py -t Debug \
|
third_party/blink/tools/run_web_tests.py -t Debug \
|
||||||
--no-show-results --no-retry-failures \
|
--no-show-results --no-retry-failures \
|
||||||
path/to/layout/test.html
|
path/to/web/test.html
|
||||||
```
|
```
|
||||||
|
|
||||||
Modify the `out` directory, ninja args, and test name as appropriate, and save
|
Modify the `out` directory, ninja args, and test name as appropriate, and save
|
||||||
@@ -474,11 +474,11 @@ git bisect run ~/checkrev.sh
|
|||||||
git bisect reset # quit the bisect session
|
git bisect reset # quit the bisect session
|
||||||
```
|
```
|
||||||
|
|
||||||
## Rebaselining Layout Tests
|
## Rebaselining Web Tests
|
||||||
|
|
||||||
*** promo
|
*** promo
|
||||||
To automatically re-baseline tests across all Chromium platforms, using the
|
To automatically re-baseline tests across all Chromium platforms, using the
|
||||||
buildbot results, see [How to rebaseline](./layout_test_expectations.md#How-to-rebaseline).
|
buildbot results, see [How to rebaseline](./web_test_expectations.md#How-to-rebaseline).
|
||||||
Alternatively, to manually run and test and rebaseline it on your workstation,
|
Alternatively, to manually run and test and rebaseline it on your workstation,
|
||||||
read on.
|
read on.
|
||||||
***
|
***
|
||||||
@@ -488,7 +488,7 @@ cd src/third_party/blink
|
|||||||
python tools/run_web_tests.py --reset-results foo/bar/test.html
|
python tools/run_web_tests.py --reset-results foo/bar/test.html
|
||||||
```
|
```
|
||||||
|
|
||||||
If there are current expectation files for `LayoutTests/foo/bar/test.html`,
|
If there are current expectation files for `web_tests/foo/bar/test.html`,
|
||||||
the above command will overwrite the current baselines at their original
|
the above command will overwrite the current baselines at their original
|
||||||
locations with the actual results. The current baseline means the `-expected.*`
|
locations with the actual results. The current baseline means the `-expected.*`
|
||||||
file used to compare the actual result when the test is run locally, i.e. the
|
file used to compare the actual result when the test is run locally, i.e. the
|
||||||
@@ -497,7 +497,7 @@ first file found in the [baseline search path]
|
|||||||
|
|
||||||
If there are no current baselines, the above command will create new baselines
|
If there are no current baselines, the above command will create new baselines
|
||||||
in the platform-independent directory, e.g.
|
in the platform-independent directory, e.g.
|
||||||
`LayoutTests/foo/bar/test-expected.{txt,png}`.
|
`web_tests/foo/bar/test-expected.{txt,png}`.
|
||||||
|
|
||||||
When you rebaseline a test, make sure your commit description explains why the
|
When you rebaseline a test, make sure your commit description explains why the
|
||||||
test is being re-baselined.
|
test is being re-baselined.
|
||||||
@@ -513,7 +513,7 @@ python tools/run_web_tests.py --additional-driver-flag=--enable-flag --reset-res
|
|||||||
```
|
```
|
||||||
|
|
||||||
New baselines will be created in the flag-specific baselines directory, e.g.
|
New baselines will be created in the flag-specific baselines directory, e.g.
|
||||||
`LayoutTests/flag-specific/enable-flag/foo/bar/test-expected.{txt,png}`.
|
`web_tests/flag-specific/enable-flag/foo/bar/test-expected.{txt,png}`.
|
||||||
|
|
||||||
Then you can commit the new baselines and upload the patch for review.
|
Then you can commit the new baselines and upload the patch for review.
|
||||||
|
|
||||||
@@ -539,14 +539,14 @@ files. You can follow the steps below for easier review.
|
|||||||
|
|
||||||
## web-platform-tests
|
## web-platform-tests
|
||||||
|
|
||||||
In addition to layout tests developed and run just by the Blink team, there is
|
In addition to web tests developed and run just by the Blink team, there is
|
||||||
also a shared test suite, see [web-platform-tests](./web_platform_tests.md).
|
also a shared test suite, see [web-platform-tests](./web_platform_tests.md).
|
||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
|
|
||||||
See
|
See
|
||||||
[bugs with the component Blink>Infra](https://bugs.chromium.org/p/chromium/issues/list?can=2&q=component%3ABlink%3EInfra)
|
[bugs with the component Blink>Infra](https://bugs.chromium.org/p/chromium/issues/list?can=2&q=component%3ABlink%3EInfra)
|
||||||
for issues related to Blink tools, include the layout test runner.
|
for issues related to Blink tools, include the web test runner.
|
||||||
|
|
||||||
* If QuickTime is not installed, the plugin tests
|
* If QuickTime is not installed, the plugin tests
|
||||||
`fast/dom/object-embed-plugin-scripting.html` and
|
`fast/dom/object-embed-plugin-scripting.html` and
|
@@ -1,8 +1,8 @@
|
|||||||
# Running layout tests using the content shell
|
# Running web tests using the content shell
|
||||||
|
|
||||||
## Basic usage
|
## Basic usage
|
||||||
|
|
||||||
Layout tests can be run with `content_shell`. To just dump the render tree, use
|
Web tests can be run with `content_shell`. To just dump the render tree, use
|
||||||
the `--run-web-tests` flag:
|
the `--run-web-tests` flag:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -11,13 +11,13 @@ out/Default/content_shell --run-web-tests foo.html
|
|||||||
|
|
||||||
### Compiling
|
### Compiling
|
||||||
|
|
||||||
If you want to run layout tests,
|
If you want to run web tests,
|
||||||
[build the target `blink_tests`](layout_tests.md); this includes all the other
|
[build the target `blink_tests`](web_tests.md); this includes all the other
|
||||||
binaries required to run the tests.
|
binaries required to run the tests.
|
||||||
|
|
||||||
### Running
|
### Running
|
||||||
|
|
||||||
You can run layout tests using `run_web_tests.py` (in
|
You can run web tests using `run_web_tests.py` (in
|
||||||
`src/third_party/blink/tools`).
|
`src/third_party/blink/tools`).
|
||||||
|
|
||||||
```bash
|
```bash
|
@@ -1,18 +1,18 @@
|
|||||||
# Layout Tests Tips
|
# Web Tests Tips
|
||||||
|
|
||||||
The recommendations here are intended to help you write new tests that go
|
The recommendations here are intended to help you write new tests that go
|
||||||
through code review with a minimal number of round trips, remain useful as Blink
|
through code review with a minimal number of round trips, remain useful as Blink
|
||||||
evolves, and serve as an asset (rather than a liability) for the team.
|
evolves, and serve as an asset (rather than a liability) for the team.
|
||||||
|
|
||||||
While reading existing layout tests, please keep in mind that they represent
|
While reading existing web tests, please keep in mind that they represent
|
||||||
snapshots taken over many years of an ever-evolving collective opinion of what
|
snapshots taken over many years of an ever-evolving collective opinion of what
|
||||||
good Web pages and solid tests should look like. Thus, it should not come as a
|
good Web pages and solid tests should look like. Thus, it should not come as a
|
||||||
surprise that most existing layout tests are not consistent with these
|
surprise that most existing web tests are not consistent with these
|
||||||
recommendations, and are not even consistent with each other.
|
recommendations, and are not even consistent with each other.
|
||||||
|
|
||||||
*** note
|
*** note
|
||||||
This document intentionally uses _should_ a lot more than _must_, as defined in
|
This document intentionally uses _should_ a lot more than _must_, as defined in
|
||||||
[RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a
|
[RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing web tests is a
|
||||||
careful act of balancing many concerns, and this humble document cannot possibly
|
careful act of balancing many concerns, and this humble document cannot possibly
|
||||||
capture the context that rests in the head of an experienced Blink engineer.
|
capture the context that rests in the head of an experienced Blink engineer.
|
||||||
***
|
***
|
||||||
@@ -51,7 +51,7 @@ test results less reliable.
|
|||||||
### Fast
|
### Fast
|
||||||
|
|
||||||
Tests should be as **fast** as possible, without compromising on the principles
|
Tests should be as **fast** as possible, without compromising on the principles
|
||||||
below. Blink has several thousand layout tests that are run in parallel, and
|
below. Blink has several thousand web tests that are run in parallel, and
|
||||||
avoiding unnecessary delays is crucial to keeping our Commit Queue in good
|
avoiding unnecessary delays is crucial to keeping our Commit Queue in good
|
||||||
shape.
|
shape.
|
||||||
|
|
||||||
@@ -72,7 +72,7 @@ and the Commit Queue.
|
|||||||
on a fast system, tests that rely on fixed timeouts can fail when on systems
|
on a fast system, tests that rely on fixed timeouts can fail when on systems
|
||||||
that are slower than expected.
|
that are slower than expected.
|
||||||
|
|
||||||
When adding or significantly modifying a layout test, use the command below to
|
When adding or significantly modifying a web test, use the command below to
|
||||||
assess its flakiness. While not foolproof, this approach gives you some
|
assess its flakiness. While not foolproof, this approach gives you some
|
||||||
confidence, and giving up CPU cycles for mental energy is a pretty good trade.
|
confidence, and giving up CPU cycles for mental energy is a pretty good trade.
|
||||||
|
|
||||||
@@ -81,7 +81,7 @@ third_party/blink/tools/run_web_tests.py path/to/test.html --repeat-each=100
|
|||||||
```
|
```
|
||||||
|
|
||||||
The
|
The
|
||||||
[PSA on writing reliable layout tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit).
|
[PSA on writing reliable web tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit).
|
||||||
also has good guidelines for writing reliable tests.
|
also has good guidelines for writing reliable tests.
|
||||||
|
|
||||||
### Self-Describing
|
### Self-Describing
|
||||||
@@ -92,7 +92,7 @@ feature being tested.
|
|||||||
|
|
||||||
`testharness.js` makes a test self-describing when used correctly. Other types
|
`testharness.js` makes a test self-describing when used correctly. Other types
|
||||||
of tests, such as reference tests and
|
of tests, such as reference tests and
|
||||||
[tests with manual fallback](./layout_tests_with_manual_fallback.md),
|
[tests with manual fallback](./web_tests_with_manual_fallback.md),
|
||||||
[must be carefully designed](https://web-platform-tests.org/writing-tests/manual.html#requirements-for-a-manual-test)
|
[must be carefully designed](https://web-platform-tests.org/writing-tests/manual.html#requirements-for-a-manual-test)
|
||||||
to be self-describing.
|
to be self-describing.
|
||||||
|
|
||||||
@@ -179,9 +179,9 @@ the doctype.
|
|||||||
|
|
||||||
## Coding Style
|
## Coding Style
|
||||||
|
|
||||||
No coding style is enforced for layout tests. This section highlights coding
|
No coding style is enforced for web tests. This section highlights coding
|
||||||
style aspects that are not consistent across our layout tests, and suggests some
|
style aspects that are not consistent across our web tests, and suggests some
|
||||||
defaults for unopinionated developers. When writing layout tests for a new part
|
defaults for unopinionated developers. When writing web tests for a new part
|
||||||
of the codebase, you can minimize review latency by taking a look at existing
|
of the codebase, you can minimize review latency by taking a look at existing
|
||||||
tests, and pay particular attention to these issues. Also beware of per-project
|
tests, and pay particular attention to these issues. Also beware of per-project
|
||||||
style guides, such as the
|
style guides, such as the
|
||||||
@@ -192,7 +192,7 @@ style guides, such as the
|
|||||||
[Google's JavaScript Style Guide](https://google.github.io/styleguide/jsguide.html)
|
[Google's JavaScript Style Guide](https://google.github.io/styleguide/jsguide.html)
|
||||||
and
|
and
|
||||||
[Google's HTML/CSS Style Guide](https://google.github.io/styleguide/htmlcssguide.xml)
|
[Google's HTML/CSS Style Guide](https://google.github.io/styleguide/htmlcssguide.xml)
|
||||||
are a reasonable baseline for coding style defaults, with the caveat that layout
|
are a reasonable baseline for coding style defaults, with the caveat that web
|
||||||
tests do not use Google Closure or JSDoc.
|
tests do not use Google Closure or JSDoc.
|
||||||
|
|
||||||
### == vs ===
|
### == vs ===
|
||||||
@@ -273,7 +273,7 @@ more familiar to developers coming from other programming languages.
|
|||||||
|
|
||||||
A good default is to prefer classes over other OOP constructs, as they will make
|
A good default is to prefer classes over other OOP constructs, as they will make
|
||||||
the code easier to read for many of your fellow Chrome developers. At the same
|
the code easier to read for many of your fellow Chrome developers. At the same
|
||||||
time, most layout tests are simple enough that OOP is not justified.
|
time, most web tests are simple enough that OOP is not justified.
|
||||||
|
|
||||||
### Character Encoding
|
### Character Encoding
|
||||||
|
|
||||||
@@ -286,7 +286,7 @@ browser-specific heuristics.
|
|||||||
|
|
||||||
The easiest way to not have to think about any of this is to add
|
The easiest way to not have to think about any of this is to add
|
||||||
`<meta charset="utf-8">` to all your tests. This is easier to remember if you
|
`<meta charset="utf-8">` to all your tests. This is easier to remember if you
|
||||||
use a template for your layout tests, rather than writing them from scratch.
|
use a template for your web tests, rather than writing them from scratch.
|
||||||
|
|
||||||
## Tests with Manual Feedback
|
## Tests with Manual Feedback
|
||||||
|
|
||||||
@@ -296,6 +296,6 @@ having the tests gracefully degrade to manual tests in the absence of the
|
|||||||
testing APIs.
|
testing APIs.
|
||||||
|
|
||||||
The
|
The
|
||||||
[document on layout tests with manual feedback](./layout_tests_with_manual_fallback.md)
|
[document on web tests with manual feedback](./web_tests_with_manual_fallback.md)
|
||||||
describes the approach in detail and highlights the trade-off between added test
|
describes the approach in detail and highlights the trade-off between added test
|
||||||
weight and ease of debugging.
|
weight and ease of debugging.
|
8
docs/testing/layout_tests_with_manual_fallback.md → docs/testing/web_tests_with_manual_fallback.md
8
docs/testing/layout_tests_with_manual_fallback.md → docs/testing/web_tests_with_manual_fallback.md
@@ -1,4 +1,4 @@
|
|||||||
# Layout Tests with Manual Fallback
|
# Web Tests with Manual Fallback
|
||||||
|
|
||||||
Some Blink features cannot be automatically tested using the Web Platform. Prime
|
Some Blink features cannot be automatically tested using the Web Platform. Prime
|
||||||
examples are the APIs that require
|
examples are the APIs that require
|
||||||
@@ -17,7 +17,7 @@ when loaded in an environment that exposes the needed testing APIs.
|
|||||||
|
|
||||||
## Motivation
|
## Motivation
|
||||||
|
|
||||||
Layout tests that degrade to manual tests in the absence of testing APIs have
|
Web tests that degrade to manual tests in the absence of testing APIs have
|
||||||
the following benefits.
|
the following benefits.
|
||||||
|
|
||||||
* The manual test component can be debugged in a normal browser session, using
|
* The manual test component can be debugged in a normal browser session, using
|
||||||
@@ -25,7 +25,7 @@ the following benefits.
|
|||||||
without a manual fallback can only be debugged in the test runner.
|
without a manual fallback can only be debugged in the test runner.
|
||||||
* The manual tests can run in other browsers, making it easy to check whether
|
* The manual tests can run in other browsers, making it easy to check whether
|
||||||
our behavior matches other browsers.
|
our behavior matches other browsers.
|
||||||
* The layout tests can form the basis for manual tests that are contributed to
|
* The web tests can form the basis for manual tests that are contributed to
|
||||||
[web-platform-tests](./web_platform_tests.md).
|
[web-platform-tests](./web_platform_tests.md).
|
||||||
|
|
||||||
Therefore, the desirability of adding a manual fallback to a test heavily
|
Therefore, the desirability of adding a manual fallback to a test heavily
|
||||||
@@ -36,7 +36,7 @@ and the size and complexity introduced by the manual fallback.
|
|||||||
|
|
||||||
## Development Tips
|
## Development Tips
|
||||||
|
|
||||||
A natural workflow for writing a layout test that gracefully degrades to a
|
A natural workflow for writing a web test that gracefully degrades to a
|
||||||
manual test is to first develop the manual test in a browser, and then add code
|
manual test is to first develop the manual test in a browser, and then add code
|
||||||
that feature-checks for testing APIs, and uses them to automate the test's
|
that feature-checks for testing APIs, and uses them to automate the test's
|
||||||
manual steps.
|
manual steps.
|
@@ -1,16 +1,10 @@
|
|||||||
# Writing Layout Tests
|
# Writing Web Tests
|
||||||
|
|
||||||
_Layout tests_ is a bit of a misnomer. This term is
|
|
||||||
[a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory/),
|
|
||||||
and we use it to refer to every test that is written as a Web page (HTML, SVG,
|
|
||||||
or XHTML) and lives in
|
|
||||||
[third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests).
|
|
||||||
|
|
||||||
[TOC]
|
[TOC]
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
Layout tests should be used to accomplish one of the following goals:
|
Web tests should be used to accomplish one of the following goals:
|
||||||
|
|
||||||
1. The entire surface of Blink that is exposed to the Web should be covered by
|
1. The entire surface of Blink that is exposed to the Web should be covered by
|
||||||
tests that we contribute to [web-platform-tests](./web_platform_tests.md)
|
tests that we contribute to [web-platform-tests](./web_platform_tests.md)
|
||||||
@@ -22,12 +16,12 @@ Layout tests should be used to accomplish one of the following goals:
|
|||||||
2. When a Blink feature cannot be tested using the tools provided by WPT, and
|
2. When a Blink feature cannot be tested using the tools provided by WPT, and
|
||||||
cannot be easily covered by
|
cannot be easily covered by
|
||||||
[C++ unit tests](https://cs.chromium.org/chromium/src/third_party/blink/renderer/web/tests/?q=webframetest&sq=package:chromium&type=cs),
|
[C++ unit tests](https://cs.chromium.org/chromium/src/third_party/blink/renderer/web/tests/?q=webframetest&sq=package:chromium&type=cs),
|
||||||
the feature must be covered by layout tests, to avoid unexpected regressions.
|
the feature must be covered by web tests, to avoid unexpected regressions.
|
||||||
These tests will use Blink-specific testing APIs that are only available in
|
These tests will use Blink-specific testing APIs that are only available in
|
||||||
[content_shell](./layout_tests_in_content_shell.md).
|
[content_shell](./web_tests_in_content_shell.md).
|
||||||
|
|
||||||
*** promo
|
*** promo
|
||||||
If you know that Blink layout tests are upstreamed to other projects, such as
|
If you know that Blink web tests are upstreamed to other projects, such as
|
||||||
[test262](https://github.com/tc39/test262), please update this document. Most
|
[test262](https://github.com/tc39/test262), please update this document. Most
|
||||||
importantly, our guidelines should to make it easy for our tests to be
|
importantly, our guidelines should to make it easy for our tests to be
|
||||||
upstreamed. The
|
upstreamed. The
|
||||||
@@ -38,9 +32,9 @@ repositories.
|
|||||||
|
|
||||||
### Test Types
|
### Test Types
|
||||||
|
|
||||||
There are four broad types of layout tests, listed in the order of preference.
|
There are four broad types of web tests, listed in the order of preference.
|
||||||
|
|
||||||
* *JavaScript Tests* are the layout test implementation of
|
* *JavaScript Tests* are the web test implementation of
|
||||||
[xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain
|
[xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain
|
||||||
assertions written in JavaScript, and pass if the assertions evaluate to
|
assertions written in JavaScript, and pass if the assertions evaluate to
|
||||||
true.
|
true.
|
||||||
@@ -57,7 +51,7 @@ There are four broad types of layout tests, listed in the order of preference.
|
|||||||
system settings. For this reason, it is common for a pixel test to have a
|
system settings. For this reason, it is common for a pixel test to have a
|
||||||
different reference image for each platform that Blink is tested on, and
|
different reference image for each platform that Blink is tested on, and
|
||||||
the reference images are
|
the reference images are
|
||||||
[quite cumbersome to manage](./layout_test_expectations.md). You
|
[quite cumbersome to manage](./web_test_expectations.md). You
|
||||||
should only write a pixel test if you cannot use a reference test.
|
should only write a pixel test if you cannot use a reference test.
|
||||||
* *Text Tests* output pure text which represents the DOM tree, the DOM inner
|
* *Text Tests* output pure text which represents the DOM tree, the DOM inner
|
||||||
text, internal data structure of Blink like layout tree or graphics layer
|
text, internal data structure of Blink like layout tree or graphics layer
|
||||||
@@ -84,11 +78,11 @@ to the WPT project. To this end, tests should follow the
|
|||||||
[WPT guidelines](https://web-platform-tests.org/writing-tests/).
|
[WPT guidelines](https://web-platform-tests.org/writing-tests/).
|
||||||
|
|
||||||
|
|
||||||
There is no style guide that applies to all layout tests. However, some projects
|
There is no style guide that applies to all web tests. However, some projects
|
||||||
have adopted style guides, such as the
|
have adopted style guides, such as the
|
||||||
[ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/testing).
|
[ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/testing).
|
||||||
|
|
||||||
Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most
|
Our [document on web tests tips](./web_tests_tips.md) summarizes the most
|
||||||
important WPT guidelines and highlights some JavaScript concepts that are worth
|
important WPT guidelines and highlights some JavaScript concepts that are worth
|
||||||
paying attention to when trying to infer style rules from existing tests. If
|
paying attention to when trying to infer style rules from existing tests. If
|
||||||
you're unopinionated and looking for a style guide to follow, the document also
|
you're unopinionated and looking for a style guide to follow, the document also
|
||||||
@@ -110,8 +104,8 @@ are more accessible to browser developers.
|
|||||||
See the [API documentation](https://web-platform-tests.org/writing-tests/testharness-api.html)
|
See the [API documentation](https://web-platform-tests.org/writing-tests/testharness-api.html)
|
||||||
for a thorough introduction to `testharness.js`.
|
for a thorough introduction to `testharness.js`.
|
||||||
|
|
||||||
Layout tests should follow the recommendations of the above documentation.
|
Web tests should follow the recommendations of the above documentation.
|
||||||
Furthermore, layout tests should include relevant
|
Furthermore, web tests should include relevant
|
||||||
[metadata](https://web-platform-tests.org/writing-tests/css-metadata.html). The
|
[metadata](https://web-platform-tests.org/writing-tests/css-metadata.html). The
|
||||||
specification URL (in `<link rel="help">`) is almost always relevant, and is
|
specification URL (in `<link rel="help">`) is almost always relevant, and is
|
||||||
incredibly helpful to a developer who needs to understand the test quickly.
|
incredibly helpful to a developer who needs to understand the test quickly.
|
||||||
@@ -190,14 +184,14 @@ Some points that are not immediately obvious from the example:
|
|||||||
`t.done()` call.
|
`t.done()` call.
|
||||||
|
|
||||||
*** promo
|
*** promo
|
||||||
Layout tests that load from `file://` origins must currently use relative paths
|
Web tests that load from `file://` origins must currently use relative paths
|
||||||
to point to
|
to point to
|
||||||
[/resources/testharness.js](../../third_party/WebKit/LayoutTests/resources/testharness.js)
|
[/resources/testharness.js](../../third_party/blink/web_tests/resources/testharness.js)
|
||||||
and
|
and
|
||||||
[/resources/testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js).
|
[/resources/testharnessreport.js](../../third_party/blink/web_tests/resources/testharnessreport.js).
|
||||||
This is contrary to the WPT guidelines, which call for absolute paths.
|
This is contrary to the WPT guidelines, which call for absolute paths.
|
||||||
This limitation does not apply to the tests in `LayoutTests/http`, which rely on
|
This limitation does not apply to the tests in `web_tests/http`, which rely on
|
||||||
an HTTP server, or to the tests in `LayoutTests/external/wpt`, which are
|
an HTTP server, or to the tests in `web_tests/external/wpt`, which are
|
||||||
imported from the [WPT repository](https://github.com/web-platform-tests/wpt).
|
imported from the [WPT repository](https://github.com/web-platform-tests/wpt).
|
||||||
***
|
***
|
||||||
|
|
||||||
@@ -210,7 +204,7 @@ via supplemental testing APIs.
|
|||||||
|
|
||||||
When writing tests that rely on supplemental testing APIs, please consider the
|
When writing tests that rely on supplemental testing APIs, please consider the
|
||||||
cost and benefits of having the tests
|
cost and benefits of having the tests
|
||||||
[gracefully degrade to manual tests](./layout_tests_with_manual_fallback.md) in
|
[gracefully degrade to manual tests](./web_tests_with_manual_fallback.md) in
|
||||||
the absence of the testing APIs.
|
the absence of the testing APIs.
|
||||||
|
|
||||||
*** promo
|
*** promo
|
||||||
@@ -223,7 +217,7 @@ event handling tests can use
|
|||||||
|
|
||||||
Tests that cannot be expressed using the Web Platform APIs or WPT's testing APIs
|
Tests that cannot be expressed using the Web Platform APIs or WPT's testing APIs
|
||||||
use Blink-specific testing APIs. These APIs are only available in
|
use Blink-specific testing APIs. These APIs are only available in
|
||||||
[content_shell](./layout_tests_in_content_shell.md), and should only be used as
|
[content_shell](./web_tests_in_content_shell.md), and should only be used as
|
||||||
a last resort.
|
a last resort.
|
||||||
|
|
||||||
A downside of Blink-specific APIs is that they are not as well documented as the
|
A downside of Blink-specific APIs is that they are not as well documented as the
|
||||||
@@ -261,7 +255,7 @@ APIs, we should use the qualified name in the `if` statement:
|
|||||||
by tests that stick to Web Platform APIs. The `testharnessreport.js` file in
|
by tests that stick to Web Platform APIs. The `testharnessreport.js` file in
|
||||||
`testharness.js` is specifically designated to hold glue code that connects
|
`testharness.js` is specifically designated to hold glue code that connects
|
||||||
`testharness.js` to the testing environment. Our implementation is in
|
`testharness.js` to the testing environment. Our implementation is in
|
||||||
[third_party/WebKit/LayoutTests/resources/testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js),
|
[third_party/blink/web_tests/resources/testharnessreport.js](../../third_party/blink/web_tests/resources/testharnessreport.js),
|
||||||
and uses the `testRunner` API.
|
and uses the `testRunner` API.
|
||||||
***
|
***
|
||||||
|
|
||||||
@@ -288,8 +282,8 @@ In these situations, a test file will be accompanied by a baseline, which is an
|
|||||||
`-expected.txt` file that contains the test's expected output.
|
`-expected.txt` file that contains the test's expected output.
|
||||||
|
|
||||||
The baselines are generated automatically when appropriate by
|
The baselines are generated automatically when appropriate by
|
||||||
`run_web_tests.py`, which is described [here](./layout_tests.md), and by the
|
`run_web_tests.py`, which is described [here](./web_tests.md), and by the
|
||||||
[rebaselining tools](./layout_test_expectations.md).
|
[rebaselining tools](./web_test_expectations.md).
|
||||||
|
|
||||||
Text baselines for `testharness.js` should be avoided, as having a text baseline
|
Text baselines for `testharness.js` should be avoided, as having a text baseline
|
||||||
associated with a `testharness.js` indicates the presence of a bug. For this
|
associated with a `testharness.js` indicates the presence of a bug. For this
|
||||||
@@ -301,7 +295,7 @@ text expectations.
|
|||||||
behavior does not match the specification that is being tested, a text
|
behavior does not match the specification that is being tested, a text
|
||||||
baseline is necessary. Remember to create an issue tracking the expectation's
|
baseline is necessary. Remember to create an issue tracking the expectation's
|
||||||
removal, and to link the issue in the CL description.
|
removal, and to link the issue in the CL description.
|
||||||
* Layout tests that cannot be upstreamed to WPT should use JavaScript to
|
* Web tests that cannot be upstreamed to WPT should use JavaScript to
|
||||||
document Blink's current behavior, rather than using JavaScript to document
|
document Blink's current behavior, rather than using JavaScript to document
|
||||||
desired behavior and a text file to document current behavior.
|
desired behavior and a text file to document current behavior.
|
||||||
|
|
||||||
@@ -314,7 +308,7 @@ This harness is **deprecated**, and should not be used for new tests.
|
|||||||
|
|
||||||
If you need to understand old tests, the best `js-test` documentation is its
|
If you need to understand old tests, the best `js-test` documentation is its
|
||||||
implementation at
|
implementation at
|
||||||
[third_party/WebKit/LayoutTests/resources/js-test.js](../../third_party/WebKit/LayoutTests/resources/js-test.js).
|
[third_party/blink/web_tests/resources/js-test.js](../../third_party/blink/web_tests/resources/js-test.js).
|
||||||
|
|
||||||
`js-test` tests lean heavily on the Blink-specific `testRunner` testing API.
|
`js-test` tests lean heavily on the Blink-specific `testRunner` testing API.
|
||||||
In a nutshell, the tests call `testRunner.dumpAsText()` to signal that the page
|
In a nutshell, the tests call `testRunner.dumpAsText()` to signal that the page
|
||||||
@@ -329,7 +323,7 @@ By default, tests are loaded as if via `file:` URLs. Some web platform features
|
|||||||
require tests served via HTTP or HTTPS, for example absolute paths (`src=/foo`)
|
require tests served via HTTP or HTTPS, for example absolute paths (`src=/foo`)
|
||||||
or features restricted to secure protocols.
|
or features restricted to secure protocols.
|
||||||
|
|
||||||
HTTP tests are those under `LayoutTests/http/tests` (or virtual variants). Use a
|
HTTP tests are those under `web_tests/http/tests` (or virtual variants). Use a
|
||||||
locally running HTTP server (Apache) to run them. Tests are served off of ports
|
locally running HTTP server (Apache) to run them. Tests are served off of ports
|
||||||
8000 and 8080 for HTTP, and 8443 for HTTPS. If you run the tests using
|
8000 and 8080 for HTTP, and 8443 for HTTPS. If you run the tests using
|
||||||
`run_web_tests.py`, the server will be started automatically. To run the server
|
`run_web_tests.py`, the server will be started automatically. To run the server
|
||||||
@@ -340,7 +334,7 @@ cd src/third_party/blink/tools
|
|||||||
./run_blink_httpd.py
|
./run_blink_httpd.py
|
||||||
```
|
```
|
||||||
|
|
||||||
The layout tests will be served from `http://127.0.0.1:8000`. For example, to
|
The web tests will be served from `http://127.0.0.1:8000`. For example, to
|
||||||
run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`,
|
run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`,
|
||||||
navigate to
|
navigate to
|
||||||
`http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
|
`http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
|
||||||
@@ -351,12 +345,12 @@ To kill the server, hit any key on the terminal where `run_blink_httpd.py` is
|
|||||||
running, or just use `taskkill` or the Task Manager on Windows, and `killall` or
|
running, or just use `taskkill` or the Task Manager on Windows, and `killall` or
|
||||||
Activity Monitor on MacOS.
|
Activity Monitor on MacOS.
|
||||||
|
|
||||||
The test server sets up an alias to the `LayoutTests/resources` directory. In
|
The test server sets up an alias to the `web_tests/resources` directory. In
|
||||||
HTTP tests, you can access the testing framework at e.g.
|
HTTP tests, you can access the testing framework at e.g.
|
||||||
`src="/resources/testharness.js"`.
|
`src="/resources/testharness.js"`.
|
||||||
|
|
||||||
TODO: Document [wptserve](http://wptserve.readthedocs.io/) when we are in a
|
TODO: Document [wptserve](http://wptserve.readthedocs.io/) when we are in a
|
||||||
position to use it to run layout tests.
|
position to use it to run web tests.
|
||||||
|
|
||||||
## Reference Tests (Reftests)
|
## Reference Tests (Reftests)
|
||||||
|
|
||||||
@@ -470,7 +464,7 @@ The recommendation of using Ahem in pixel tests is being discussed on
|
|||||||
[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
|
[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
|
||||||
***
|
***
|
||||||
|
|
||||||
The following snippet includes the Ahem font in a layout test.
|
The following snippet includes the Ahem font in a web test.
|
||||||
|
|
||||||
```html
|
```html
|
||||||
<style>
|
<style>
|
||||||
@@ -482,31 +476,31 @@ body {
|
|||||||
```
|
```
|
||||||
|
|
||||||
*** promo
|
*** promo
|
||||||
Tests outside `LayoutTests/http` and `LayoutTests/external/wpt` currently need
|
Tests outside `web_tests/http` and `web_tests/external/wpt` currently need
|
||||||
to use a relative path to
|
to use a relative path to
|
||||||
[/third_party/WebKit/LayoutTests/resources/ahem.js](../../third_party/WebKit/LayoutTests/resources/ahem.js)
|
[/third_party/blink/web_tests/resources/ahem.js](../../third_party/blink/web_tests/resources/ahem.js)
|
||||||
***
|
***
|
||||||
|
|
||||||
### Tests that need to paint, raster, or draw a frame of intermediate output
|
### Tests that need to paint, raster, or draw a frame of intermediate output
|
||||||
|
|
||||||
A layout test does not actually draw frames of output until the test exits.
|
A web test does not actually draw frames of output until the test exits.
|
||||||
Tests that need to generate a painted frame can use `runAfterLayoutAndPaint()`
|
Tests that need to generate a painted frame can use `runAfterLayoutAndPaint()`
|
||||||
defined in [third_party/WebKit/LayoutTests/resources/run-after-layout-and-paint.js](../../third_party/WebKit/LayoutTests/resources/run-after-layout-and-paint.js)
|
defined in [third_party/blink/web_tests/resources/run-after-layout-and-paint.js](../../third_party/blink/web_tests/resources/run-after-layout-and-paint.js)
|
||||||
which will run the machinery to put up a frame, then call the passed callback.
|
which will run the machinery to put up a frame, then call the passed callback.
|
||||||
There is also a library at
|
There is also a library at
|
||||||
[third_party/WebKit/LayoutTests/paint/invalidation/resources/text-based-repaint.js](../../third_party/WebKit/LayoutTests/paint/invalidation/resources/text-based-repaint.js)
|
[third_party/blink/web_tests/paint/invalidation/resources/text-based-repaint.js](../../third_party/blink/web_tests/paint/invalidation/resources/text-based-repaint.js)
|
||||||
to help with writing paint invalidation and repaint tests.
|
to help with writing paint invalidation and repaint tests.
|
||||||
|
|
||||||
### Tests for scrolling animations
|
### Tests for scrolling animations
|
||||||
|
|
||||||
Some layout tests need to ensure animations such as middle-click auto-scroll,
|
Some web tests need to ensure animations such as middle-click auto-scroll,
|
||||||
fling, etc. get performed properly. When testing in display compositor pixel
|
fling, etc. get performed properly. When testing in display compositor pixel
|
||||||
dump mode (now the standard), the standard behavior for tests is to
|
dump mode (now the standard), the standard behavior for tests is to
|
||||||
synchronously composite without rastering (to save time). However, animations
|
synchronously composite without rastering (to save time). However, animations
|
||||||
run upon surface activation, which only happens once rasterization is performed.
|
run upon surface activation, which only happens once rasterization is performed.
|
||||||
Therefore, for these tests, an additional setting needs to be set. Near the
|
Therefore, for these tests, an additional setting needs to be set. Near the
|
||||||
beginning of these tests, call `setAnimationRequiresRaster()` defined in
|
beginning of these tests, call `setAnimationRequiresRaster()` defined in
|
||||||
[third_party/WebKit/LayoutTests/resources/compositor-controls.js](../../third_party/WebKit/LayoutTests/resources/compositor-controls.js)
|
[third_party/blink/web_tests/resources/compositor-controls.js](../../third_party/blink/web_tests/resources/compositor-controls.js)
|
||||||
which will enable full rasterization during the test.
|
which will enable full rasterization during the test.
|
||||||
|
|
||||||
## Text tests
|
## Text tests
|
||||||
@@ -629,7 +623,7 @@ For a test that is both a pixel/reference test and a text test, both pixel and
|
|||||||
text results will be compared to baselines, and the test passes if each result
|
text results will be compared to baselines, and the test passes if each result
|
||||||
matches the corresponding baseline.
|
matches the corresponding baseline.
|
||||||
|
|
||||||
Many of the [paint invalidation tests](../../third_party/WebKit/LayoutTests/paint/invalidation)
|
Many of the [paint invalidation tests](../../third_party/blink/web_tests/paint/invalidation)
|
||||||
are of this type. The pixel results (compared against `-expected.png` or
|
are of this type. The pixel results (compared against `-expected.png` or
|
||||||
`-expected.html`) ensure correct rendering, and the text results (compared
|
`-expected.html`) ensure correct rendering, and the text results (compared
|
||||||
against `-expected.txt`) ensure correct compositing and raster invalidation
|
against `-expected.txt`) ensure correct compositing and raster invalidation
|
||||||
@@ -643,7 +637,7 @@ which outputs you really care about.
|
|||||||
|
|
||||||
## Directory Structure
|
## Directory Structure
|
||||||
|
|
||||||
The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently
|
The [web_tests directory](../../third_party/blink/web_tests) currently
|
||||||
lacks a strict, formal structure. The following directories have special
|
lacks a strict, formal structure. The following directories have special
|
||||||
meaning:
|
meaning:
|
||||||
|
|
||||||
@@ -652,7 +646,7 @@ meaning:
|
|||||||
as media files, and code that is shared by multiple test files.
|
as media files, and code that is shared by multiple test files.
|
||||||
|
|
||||||
*** note
|
*** note
|
||||||
Some layout tests consist of a minimal HTML page that references a JavaScript
|
Some web tests consist of a minimal HTML page that references a JavaScript
|
||||||
file in `resources/`. Please do not use this pattern for new tests, as it goes
|
file in `resources/`. Please do not use this pattern for new tests, as it goes
|
||||||
against the minimality principle. JavaScript and CSS files should only live in
|
against the minimality principle. JavaScript and CSS files should only live in
|
||||||
`resources/` if they are shared by at least two test files.
|
`resources/` if they are shared by at least two test files.
|
@@ -1,6 +1,6 @@
|
|||||||
# Using a chroot
|
# Using a chroot
|
||||||
|
|
||||||
If you want to run layout tests and you're not running Lucid, you'll get errors
|
If you want to run web tests and you're not running Lucid, you'll get errors
|
||||||
due to version differences in libfreetype. To work around this, you can use a
|
due to version differences in libfreetype. To work around this, you can use a
|
||||||
chroot.
|
chroot.
|
||||||
|
|
||||||
@@ -65,16 +65,16 @@ fi
|
|||||||
### Running X apps
|
### Running X apps
|
||||||
|
|
||||||
I also have `DISPLAY=:0` in my `$debian_chroot` section so I can run test_shell
|
I also have `DISPLAY=:0` in my `$debian_chroot` section so I can run test_shell
|
||||||
or layout tests without manually setting my display every time. Your display
|
or web tests without manually setting my display every time. Your display
|
||||||
number may vary (`echo $DISPLAY` outside the chroot to see what your display
|
number may vary (`echo $DISPLAY` outside the chroot to see what your display
|
||||||
number is).
|
number is).
|
||||||
|
|
||||||
You can also use `Xvfb` if you only want to
|
You can also use `Xvfb` if you only want to
|
||||||
[run tests headless](layout_tests_linux.md#Using-an-embedded-X-server).
|
[run tests headless](web_tests_linux.md#Using-an-embedded-X-server).
|
||||||
|
|
||||||
### Having layout test results open in a browser
|
### Having web test results open in a browser
|
||||||
|
|
||||||
After running layout tests, you should get a new browser tab or window that
|
After running web tests, you should get a new browser tab or window that
|
||||||
opens results.html. If you get an error "Failed to open
|
opens results.html. If you get an error "Failed to open
|
||||||
file:///path/to/results.html, check the
|
file:///path/to/results.html, check the
|
||||||
following conditions.
|
following conditions.
|
||||||
|
@@ -1,16 +1,16 @@
|
|||||||
# Running layout tests on Linux
|
# Running web tests on Linux
|
||||||
|
|
||||||
1. Build `blink_tests` (see [Linux-specific build instructions](https://chromium.googlesource.com/chromium/src/+/master/docs/linux_build_instructions.md))
|
1. Build `blink_tests` (see [Linux-specific build instructions](https://chromium.googlesource.com/chromium/src/+/master/docs/linux_build_instructions.md))
|
||||||
1. Checkout the layout tests
|
1. Checkout the web tests
|
||||||
* If you have an entry in your `.gclient` file that includes
|
* If you have an entry in your `.gclient` file that includes
|
||||||
"LayoutTests", you may need to comment it out and sync.
|
"web_tests", you may need to comment it out and sync.
|
||||||
* You can run a subset of the tests by passing in a path relative to
|
* You can run a subset of the tests by passing in a path relative to
|
||||||
`src/third_party/WebKit/LayoutTests/`. For example,
|
`src/third_party/blink/web_tests/`. For example,
|
||||||
`third_party/blink/tools/run_web_tests.py fast` will only run the tests under
|
`third_party/blink/tools/run_web_tests.py fast` will only run the tests under
|
||||||
`src/third_party/WebKit/LayoutTests/fast/`.
|
`src/third_party/blink/web_tests/fast/`.
|
||||||
1. When the tests finish, any unexpected results should be displayed.
|
1. When the tests finish, any unexpected results should be displayed.
|
||||||
|
|
||||||
See [Layout Tests](testing/layout_tests.md)
|
See [Web Tests](testing/web_tests.md)
|
||||||
for full documentation about set up and available options.
|
for full documentation about set up and available options.
|
||||||
|
|
||||||
## Pixel Tests
|
## Pixel Tests
|
||||||
@@ -61,15 +61,15 @@ If `fast/dom/object-plugin-hides-properties.html` and
|
|||||||
set up a Lucid 64 chroot. Learn more about
|
set up a Lucid 64 chroot. Learn more about
|
||||||
[using a linux chroot](using_a_linux_chroot.md).
|
[using a linux chroot](using_a_linux_chroot.md).
|
||||||
|
|
||||||
## Getting a layout test into a debugger
|
## Getting a web test into a debugger
|
||||||
|
|
||||||
There are two ways:
|
There are two ways:
|
||||||
|
|
||||||
1. Run `content_shell` directly rather than using `run_web_tests.py`. You
|
1. Run `content_shell` directly rather than using `run_web_tests.py`. You
|
||||||
will need to pass some options:
|
will need to pass some options:
|
||||||
* `--no-timeout` to give you plenty of time to debug
|
* `--no-timeout` to give you plenty of time to debug
|
||||||
* the fully qualified path of the layout test (rather than relative to
|
* the fully qualified path of the web test (rather than relative to
|
||||||
`WebKit/LayoutTests`).
|
`blink/web_tests`).
|
||||||
1. Or, run as normal but with the
|
1. Or, run as normal but with the
|
||||||
`--additional-drt-flag=--renderer-startup-dialog
|
`--additional-drt-flag=--renderer-startup-dialog
|
||||||
--additional-drt-flag=--no-timeout --time-out-ms=86400000` flags. The first
|
--additional-drt-flag=--no-timeout --time-out-ms=86400000` flags. The first
|
||||||
@@ -86,7 +86,7 @@ create a separate X server for running the tests.
|
|||||||
|
|
||||||
1. Install Xephyr (`sudo apt-get install xserver-xephyr`)
|
1. Install Xephyr (`sudo apt-get install xserver-xephyr`)
|
||||||
1. Start Xephyr as display 4: `Xephyr :4 -screen 1024x768x24`
|
1. Start Xephyr as display 4: `Xephyr :4 -screen 1024x768x24`
|
||||||
1. Run the layout tests in the Xephyr: `DISPLAY=:4 run_web_tests.py`
|
1. Run the web tests in the Xephyr: `DISPLAY=:4 run_web_tests.py`
|
||||||
|
|
||||||
Xephyr supports debugging repainting. See the
|
Xephyr supports debugging repainting. See the
|
||||||
[Xephyr README](http://cgit.freedesktop.org/xorg/xserver/tree/hw/kdrive/ephyr/README)
|
[Xephyr README](http://cgit.freedesktop.org/xorg/xserver/tree/hw/kdrive/ephyr/README)
|
||||||
@@ -99,11 +99,11 @@ If you don't want to see anything at all, you can use Xvfb (should already be
|
|||||||
installed).
|
installed).
|
||||||
|
|
||||||
1. Start Xvfb as display 4: `Xvfb :4 -screen 0 1024x768x24`
|
1. Start Xvfb as display 4: `Xvfb :4 -screen 0 1024x768x24`
|
||||||
1. Run the layout tests in the Xvfb: `DISPLAY=:4 run_web_tests.py`
|
1. Run the web tests in the Xvfb: `DISPLAY=:4 run_web_tests.py`
|
||||||
|
|
||||||
## Tiling Window managers
|
## Tiling Window managers
|
||||||
|
|
||||||
The layout tests want to run with the window at a particular size down to the
|
The web tests want to run with the window at a particular size down to the
|
||||||
pixel level. This means if your window manager resizes the window it'll cause
|
pixel level. This means if your window manager resizes the window it'll cause
|
||||||
test failures. This is another good reason to use an embedded X server.
|
test failures. This is another good reason to use an embedded X server.
|
||||||
|
|
@@ -341,7 +341,7 @@ pointers for how to get started. If Chrome does not correctly implement the
|
|||||||
spec, add a corresponding -expected.txt file with your test listing the expected
|
spec, add a corresponding -expected.txt file with your test listing the expected
|
||||||
failure in Chrome.
|
failure in Chrome.
|
||||||
|
|
||||||
[Layout tests](../../../../../docs/testing/writing_layout_tests.md) are located
|
[Web tests](../../../../../docs/testing/writing_web_tests.md) are located
|
||||||
in [third_party/blink/web_tests][]. These should be written when needing end
|
in [third_party/blink/web_tests][]. These should be written when needing end
|
||||||
to end testing but either when testing chrome specific features (i.e.
|
to end testing but either when testing chrome specific features (i.e.
|
||||||
non-standardized) such as compositing or when the test requires access to chrome
|
non-standardized) such as compositing or when the test requires access to chrome
|
||||||
|
@@ -91,7 +91,7 @@ npm test -- -f --child-processes=16
|
|||||||
npm test -- --time-out-ms=6000000 <test_path>
|
npm test -- --time-out-ms=6000000 <test_path>
|
||||||
```
|
```
|
||||||
|
|
||||||
> **Tip**: [Learn about the test harness flags](https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_tests.md#Test-Harness-Options)
|
> **Tip**: [Learn about the test harness flags](https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_tests.md#Test-Harness-Options)
|
||||||
|
|
||||||
#### `--fetch-content-shell`
|
#### `--fetch-content-shell`
|
||||||
```
|
```
|
||||||
|
@@ -499,7 +499,7 @@ def parse_args(args):
|
|||||||
option_parser = optparse.OptionParser(
|
option_parser = optparse.OptionParser(
|
||||||
prog='run_web_tests.py',
|
prog='run_web_tests.py',
|
||||||
usage='%prog [options] [tests]',
|
usage='%prog [options] [tests]',
|
||||||
description='Runs Blink layout tests as described in docs/testing/layout_tests.md')
|
description='Runs Blink layout tests as described in docs/testing/web_tests.md')
|
||||||
|
|
||||||
for group_name, group_options in option_group_definitions:
|
for group_name, group_options in option_group_definitions:
|
||||||
option_group = optparse.OptionGroup(option_parser, group_name)
|
option_group = optparse.OptionGroup(option_parser, group_name)
|
||||||
|
2
third_party/blink/web_tests/ASANExpectations
vendored
2
third_party/blink/web_tests/ASANExpectations
vendored
@@ -1,7 +1,7 @@
|
|||||||
# This file is used for tests that only need to be suppressed on
|
# This file is used for tests that only need to be suppressed on
|
||||||
# the Chromium WebKit ASAN bots.
|
# the Chromium WebKit ASAN bots.
|
||||||
# Further documentation:
|
# Further documentation:
|
||||||
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
|
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
|
||||||
|
|
||||||
# NULL deref in libflashplayer.so, http://crbug.com/145940
|
# NULL deref in libflashplayer.so, http://crbug.com/145940
|
||||||
# ==12527== ERROR: AddressSanitizer: SEGV on unknown address 0x7f628058b38b
|
# ==12527== ERROR: AddressSanitizer: SEGV on unknown address 0x7f628058b38b
|
||||||
|
2
third_party/blink/web_tests/LeakExpectations
vendored
2
third_party/blink/web_tests/LeakExpectations
vendored
@@ -1,7 +1,7 @@
|
|||||||
# This file is used for tests that only need to be suppressed on the Chromium
|
# This file is used for tests that only need to be suppressed on the Chromium
|
||||||
# WebKit Leak bots.
|
# WebKit Leak bots.
|
||||||
# Further documentation:
|
# Further documentation:
|
||||||
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
|
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
|
||||||
|
|
||||||
###########################################################################
|
###########################################################################
|
||||||
# WARNING: Memory leaks must be fixed asap. Sheriff is expected to revert #
|
# WARNING: Memory leaks must be fixed asap. Sheriff is expected to revert #
|
||||||
|
2
third_party/blink/web_tests/MSANExpectations
vendored
2
third_party/blink/web_tests/MSANExpectations
vendored
@@ -1,7 +1,7 @@
|
|||||||
# This file is used for tests that only need to be suppressed on
|
# This file is used for tests that only need to be suppressed on
|
||||||
# the Chromium Blink MSan bots.
|
# the Chromium Blink MSan bots.
|
||||||
# Further documentation:
|
# Further documentation:
|
||||||
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
|
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
|
||||||
|
|
||||||
# Those stress V8's simulator, making them very slow under MSan.
|
# Those stress V8's simulator, making them very slow under MSan.
|
||||||
crbug.com/420198 [ Linux ] fast/js/regress [ Skip ]
|
crbug.com/420198 [ Linux ] fast/js/regress [ Skip ]
|
||||||
|
2
third_party/blink/web_tests/NeverFixTests
vendored
2
third_party/blink/web_tests/NeverFixTests
vendored
@@ -8,7 +8,7 @@
|
|||||||
# 6. Tests that fail due to Debug rendering variations from Release behavior.
|
# 6. Tests that fail due to Debug rendering variations from Release behavior.
|
||||||
# 7. Tests that are not meant to be run automatically (manual tests).
|
# 7. Tests that are not meant to be run automatically (manual tests).
|
||||||
# 8. Tests that are automatically imported but not intended to be run.
|
# 8. Tests that are automatically imported but not intended to be run.
|
||||||
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
|
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
|
||||||
|
|
||||||
# FIXME: Remove the WontFix keyword and have that be implied for tests in this file.
|
# FIXME: Remove the WontFix keyword and have that be implied for tests in this file.
|
||||||
|
|
||||||
|
2
third_party/blink/web_tests/SlowTests
vendored
2
third_party/blink/web_tests/SlowTests
vendored
@@ -6,7 +6,7 @@
|
|||||||
# than 2 seconds in Release mode or 6 seconds in Debug mode should be listed here.
|
# than 2 seconds in Release mode or 6 seconds in Debug mode should be listed here.
|
||||||
#
|
#
|
||||||
# Further documentation:
|
# Further documentation:
|
||||||
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
|
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
|
||||||
|
|
||||||
crbug.com/24182 storage/indexeddb/objectstore-cursor.html [ Slow ]
|
crbug.com/24182 storage/indexeddb/objectstore-cursor.html [ Slow ]
|
||||||
crbug.com/24182 storage/indexeddb/mozilla/test_objectStore_openKeyCursor.html [ Slow ]
|
crbug.com/24182 storage/indexeddb/mozilla/test_objectStore_openKeyCursor.html [ Slow ]
|
||||||
|
@@ -5,7 +5,7 @@
|
|||||||
# Feel free to take ownership of any of these lines and get the test passing.
|
# Feel free to take ownership of any of these lines and get the test passing.
|
||||||
#
|
#
|
||||||
# Further documentation:
|
# Further documentation:
|
||||||
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
|
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
|
||||||
|
|
||||||
# We could fix this test for us and upstream it if the test shell user agent
|
# We could fix this test for us and upstream it if the test shell user agent
|
||||||
# would let us differentiate test_shell and WebKit DumpTreeNode.
|
# would let us differentiate test_shell and WebKit DumpTreeNode.
|
||||||
|
2
third_party/blink/web_tests/TestExpectations
vendored
2
third_party/blink/web_tests/TestExpectations
vendored
@@ -1,6 +1,6 @@
|
|||||||
# This is the main failure suppression file for Blink LayoutTests.
|
# This is the main failure suppression file for Blink LayoutTests.
|
||||||
# Further documentation:
|
# Further documentation:
|
||||||
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
|
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
|
||||||
|
|
||||||
# Intentional failures to test the layout test system.
|
# Intentional failures to test the layout test system.
|
||||||
Bug(intentional) harness-tests/crash.html [ Crash ]
|
Bug(intentional) harness-tests/crash.html [ Crash ]
|
||||||
|
@@ -7,7 +7,7 @@
|
|||||||
# used to make it easier to work on getting them running and passing.
|
# used to make it easier to work on getting them running and passing.
|
||||||
#
|
#
|
||||||
# Further documentation:
|
# Further documentation:
|
||||||
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
|
# https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
|
||||||
|
|
||||||
external/wpt/.codecov.yml [ Skip ]
|
external/wpt/.codecov.yml [ Skip ]
|
||||||
external/wpt/.gitmodules [ Skip ]
|
external/wpt/.gitmodules [ Skip ]
|
||||||
|
@@ -1,13 +1,13 @@
|
|||||||
# LayoutTests for Accessibility
|
# Web Tests for Accessibility
|
||||||
|
|
||||||
## General Info on LayoutTests: Building and Running the Tests
|
## General Info on web tests: Building and Running the Tests
|
||||||
|
|
||||||
See [Layout Tests](/docs/testing/layout_tests.md) for general
|
See [Web Tests](/docs/testing/web_tests.md) for general
|
||||||
info on how to build and run layout tests.
|
info on how to build and run web tests.
|
||||||
|
|
||||||
## Old vs. New
|
## Old vs. New
|
||||||
|
|
||||||
There are two styles of accessibility layout tests:
|
There are two styles of accessibility web tests:
|
||||||
|
|
||||||
* Using a ```-expected.txt``` (now deprecated)
|
* Using a ```-expected.txt``` (now deprecated)
|
||||||
* Unit-style tests with assertions
|
* Unit-style tests with assertions
|
||||||
|
@@ -232,7 +232,7 @@ p, h2, h3, h4 {
|
|||||||
<h4>2. TestExpectations</a></h4>
|
<h4>2. TestExpectations</a></h4>
|
||||||
<pre><a href="#">crbug.com/bug</a> layout/test/path/<a href="#">test.html</a> [ Status ]</pre>
|
<pre><a href="#">crbug.com/bug</a> layout/test/path/<a href="#">test.html</a> [ Status ]</pre>
|
||||||
<p>TestExpectationsshow lines as they'd appear in <a
|
<p>TestExpectationsshow lines as they'd appear in <a
|
||||||
href="https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md">TestExpectations</a> file.</p>
|
href="https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md">TestExpectations</a> file.</p>
|
||||||
|
|
||||||
<p>The interesting part here is [ Status ]. Inside TestExpectations file, [ Status ]
|
<p>The interesting part here is [ Status ]. Inside TestExpectations file, [ Status ]
|
||||||
can have multiple values, representing all expected results. For example:</p>
|
can have multiple values, representing all expected results. For example:</p>
|
||||||
@@ -548,7 +548,7 @@ const Report = {
|
|||||||
|
|
||||||
printExpectation: (test, path, traversal) => {
|
printExpectation: (test, path, traversal) => {
|
||||||
// TestExpectations file format is documented at:
|
// TestExpectations file format is documented at:
|
||||||
// https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md
|
// https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_test_expectations.md
|
||||||
|
|
||||||
let pathParser = new PathParser(path);
|
let pathParser = new PathParser(path);
|
||||||
// Print directory header if this test's directory is different from the last.
|
// Print directory header if this test's directory is different from the last.
|
||||||
|
@@ -14,7 +14,7 @@
|
|||||||
// These tests will not function correctly without the --run-web-tests
|
// These tests will not function correctly without the --run-web-tests
|
||||||
// option. To debug them interactively you can follow the instructions for a
|
// option. To debug them interactively you can follow the instructions for a
|
||||||
// "legacy test" in
|
// "legacy test" in
|
||||||
// https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_tests.md#Debugging-DevTools-Tests.
|
// https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_tests.md#Debugging-DevTools-Tests.
|
||||||
|
|
||||||
async_test(t => {
|
async_test(t => {
|
||||||
const startTime = Date.now();
|
const startTime = Date.now();
|
||||||
|
@@ -1,6 +1,6 @@
|
|||||||
// To make sure animation events get processed, periodic lifecycle phase runs
|
// To make sure animation events get processed, periodic lifecycle phase runs
|
||||||
// will include rasterization. See
|
// will include rasterization. See
|
||||||
// https://chromium.googlesource.com/chromium/src/+/master/docs/testing/writing_layout_tests.md
|
// https://chromium.googlesource.com/chromium/src/+/master/docs/testing/writing_web_tests.md
|
||||||
// for more information.
|
// for more information.
|
||||||
function setAnimationRequiresRaster() {
|
function setAnimationRequiresRaster() {
|
||||||
if (window.testRunner) {
|
if (window.testRunner) {
|
||||||
|
Reference in New Issue
Block a user