chromium/src
0
Fork 0
Code Activity
Files
94d30b8208a95d99fd32effb1afe5cd35760b9aa
src/docs/accessibility/browser/tests.md
Benjamin Beaudry 47b8a8f22d Introduce manual testing section in docs/accessibility 
This CL introduces a new "Manual testing" section for accessibility. The
goal is to create a live document that is continuously updated with a
list of all features to test for the most used ATs to guide all Chromium
contributors to perform thorough manual testing when making a potentially
breaking change.

For now, we start with testing guidelines about Narrator. This section
will eventually be updated to include all popular ATs. I  recommend that
we update it whenever we fix a bug that our automated tests missed (in
addition to writing a new test that will prevent the regression in
the future).

Change-Id: I69311fd986e7b5e37b1d30a139a6ed8a477f8c91
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6362795
Reviewed-by: Jacques Newman <janewman@microsoft.com>
Commit-Queue: Benjamin Beaudry <benjamin.beaudry@microsoft.com>
Cr-Commit-Position: refs/heads/main@{#1433802}
2025-03-17 15:21:32 -07:00

206 lines
8.0 KiB
Markdown
Raw Blame History

# Accessibility
Here's a quick overview of all of the locations in the codebase where you'll
find accessibility tests, and a brief overview of the purpose of all of them.
## Web Tests
This is the primary place where we test accessibility code in Blink. This code
should have no platform-specific code. Use this to test anything where there's
any interesting web platform logic, or where you need to be able to query things
synchronously from the renderer thread to test them.
Don't add tests for trivial features like ARIA attributes that we just expose
directly to the next layer up. In those cases the Blink tests are trivial and
it's more valuable to test these features at a higher level where we can ensure
they actually work.
Note that many of these tests are inherited from WebKit and the coding style has
evolved a lot since then. Look for more recent tests as a guide if writing a new
one.
Test files:
[third_party/blink/web_tests/accessibility](https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/accessibility/)
Source code to AccessibilityController and WebAXObjectProxy:
[content/web_test/renderer](https://source.chromium.org/chromium/chromium/src/+/main:content/web_test/renderer/)
First, you'll need to build the tests:
```
autoninja -C out/release blink_tests
```
Then, to run all accessibility web tests:
```
third_party/blink/tools/run_web_tests.py --build-directory=out --target=release accessibility/
```
Or, to run just one test by itself, without invoking the `run_web_tests.py` script:
```
out/release/content_shell \
--run-web-tests third_party/blink/web_tests/accessibility/name-calc-inputs.html
```
For information on modifying or adding web tests, see the main
[web tests documentation](../../testing/web_tests.md).
## DumpAccessibilityTree tests
This is the best way to write both cross-platform and platform-specific tests
using only an input HTML file, some magic strings to describe what attributes
you're interested in, and one or more expectation files to enable checking
whether the resulting accessibility tree is correct or not. In particular,
almost no test code is required.
[More documentation on DumpAccessibilityTree](../../../content/test/data/accessibility/readme.md)
Test files:
[content/test/data/accessibility](https://source.chromium.org/chromium/chromium/src/+/main:content/test/data/accessibility/)
Test runner:
[content/browser/accessibility/dump_accessibility_tree_browsertest.h](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/dump_accessibility_tree_browsertest.h)
To run all tests:
```
autoninja -C out/release content_browsertests && \
out/release/content_browsertests --gtest_filter="All/DumpAccessibilityTree*"
```
Expectation baselines for each OS can be generated via:
```
tools/accessibility/rebase_dump_accessibility_tree_tests.py
```
## Other content_browsertests
There are many other tests in content/ that relate to accessibility. All of
these tests work by launching a full multi-process browser shell, loading a web
page in a renderer, then accessing the resulting accessibility tree from the
browser process, and running some test from there.
To run all tests:
```
autoninja -C out/release content_browsertests && \
out/release/content_browsertests --gtest_filter="*ccessib*"
```
## Accessibility unittests
This tests the core accessibility code that's shared by both web and non-web
accessibility infrastructure.
Code location:
[ui/accessibility](https://source.chromium.org/chromium/chromium/src/+/main:ui/accessibility/)
and subdirectories; check the
[`*_unittest.cc` files](https://source.chromium.org/search?q=path:accessibility%20path:_unittest&sq=&ss=chromium%2Fchromium%2Fsrc:ui%2Faccessibility%2F),
next to every file that it's being tested.
List of sources:
[ui/accessibility/BUILD.gn](https://source.chromium.org/chromium/chromium/src/+/main:ui/accessibility/BUILD.gn?q=%22test(%22accessibility_unittests%22)%20%7B%22)
To run all tests:
```
autoninja -C out/release accessibility_unittests && \
out/release/accessibility_unittests
```
## ChromeVox tests
ChromeVox tests are part of the browser_tests suite. You must build with
`target_os = "chromeos"` in your GN args.
To run all tests:
```
autoninja -C out/release browser_tests && \
out/release/browser_tests --test-launcher-jobs=20 --gtest_filter=ChromeVox*
```
### Select-To-Speak tests
```
autoninja -C out/Default unit_tests browser_tests && \
out/Default/unit_tests --gtest_filter=*SelectToSpeak* && \
out/Default/browser_tests --gtest_filter=*SelectToSpeak*
```
## Performance tests
We also have a page on [Performance Tests](./perf.md).
## Other locations of accessibility tests:
Even this isn't a complete list. The tests described above cover more than 90%
of the accessibility tests, and the remainder are scattered throughout the
codebase. Here are a few other locations to check:
* [chrome/android/javatests/src/org/chromium/chrome/browser/accessibility](https://source.chromium.org/chromium/chromium/src/+/main:chrome/android/javatests/src/org/chromium/chrome/browser/accessibility/)
* [chrome/browser/accessibility](https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/accessibility/)
* [chrome/browser/ash/accessibility/](https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/ash/accessibility/)
* [ui/chromeos](https://source.chromium.org/chromium/chromium/src/+/main:ui/chromeos/)
* [ui/views/accessibility](https://source.chromium.org/chromium/chromium/src/+/main:ui/views/accessibility/)
## Helpful flags:
Across all tests there are some helpful flags that will make testing easier.
* Run tests multiple times in parallel (useful for finding flakes):
`--test-launcher-jobs=10`
* Filter which tests to run: `--gtest_filter="*Cats*"`
## Manual testing:
The Chromium accessibility pipeline supports an ecosystem of various assistive
technologies through the platform accessibility APIs. When we make changes that
modify the output of a platform accessibility interface, we should test the
major assistive technologies (JAWS, NVDA, Narrator, VoiceOver, etc.) that rely
on those interfaces to reduce the risk of introducing regressions. Below is a
list of key features to validate for each AT:
### Narrator:
Narrator relies only on UIA -- it does not use MSAA/IA2. When modifying the UIA
APIs, we should ensure that these key features still work:
* Scan Mode (toggle it with Narrator + space)
* Navigation by text unit (Narrator + CTRL + up/down arrow to change the text
unit, then Narrator + left/right arrow to navigate backward or forward).
Here are the text units to validate:
* Character navigation
* Word navigation
* Line navigation
* Sentence navigation
* Paragraph navigation
* Item navigation (as in list items, bullet points)
* Heading navigation
* Link navigation
* Form field navigation
* Landmark navigation
* Table navigation
* Search (Narrator + CTRL + F)
* List of links (Narrator + F7)
* List of headings (Narrator + F6)
* List of landmarks (Narrator + F5)
* Caret navigation (with Scan Mode off) in a text field
* The character that follows the caret should be read.
* "Blank" or similar should be announced when moving the character after the
last character of the text field.
* Spelling and grammar errors should be announced properly when moving using
the arrow keys.
* For web content, errors can be added using the aria-invalid attribute or
CSS highlights. Chromium's implementations for each of them is different
for UIA, so both should be tested.
* Errors should be announced when the caret is positioned at the very
beginning of an error range. We should test arrow key navigations and word
navigation (CTRL + right/left arrow key).
* All errors in the same line should be announced when arrowing up/down at
the start of a line.
This section is continuously updated. Feel free to edit it to add more core
features to test when making a potentially breaking change, or to add testing
instructions for other assistive technologies.
View Git Blame Copy Permalink