0

Consolidate Android testing docs /testing/android/docs -> /docs/testing

This should hopefully make them more discoverable.

Bug: None
Change-Id: I65885c50b655a8b867cdc9224bd1be59822dcb14
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4908056
Commit-Queue: Andrew Grieve <agrieve@chromium.org>
Owners-Override: Andrew Grieve <agrieve@chromium.org>
Reviewed-by: Haiyang Pan <hypan@google.com>
Reviewed-by: James Shen <zhiyuans@google.com>
Reviewed-by: Henrique Nakashima <hnakashima@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1205217}
This commit is contained in:
Andrew Grieve
2023-10-04 13:55:51 +00:00
committed by Chromium LUCI CQ
parent 58260cf066
commit b5b3bb3d3e
13 changed files with 352 additions and 825 deletions

@ -16,7 +16,7 @@ tests are organized by overall functionality instead). Put new tests into an
existing test class or create a new one if a suitable one isn't available.
#### How to write intrumentation tests?
See the [instrumentation test doc](/testing/android/docs/instrumentation.md).
See the [instrumentation test doc](/docs/testing/android_instrumentation_tests.md).
#### How do tests interact with WebView?
Tests are written as JUnit4 tests. [AwActivityTestRule] is used to create and
@ -49,7 +49,7 @@ Java code.
The tests are located in the [junit directory](/android_webview/junit/src/org/chromium/android_webview/robolectric/).
#### How to write junit tests?
See the [JUnit doc](/testing/android/docs/junit.md#junit-tests-with-robolectric).
See the [JUnit doc](/docs/testing/android_robolectric_tests.md).
## Native unittests
These are on-device gtests that only exercise android\_webview native code.
@ -59,7 +59,7 @@ The tests are located alongside respective source code files under
[android\_webview directory](/android_webview/).
#### How to write gtests?
See the [GTest doc](/testing/android/docs/gtest_implementation.md).
See the [GTest doc](/docs/testing/android_gtests.md).
## How to run tests?
Running tests is covered in [WebView Test Instructions](/android_webview/docs/test-instructions.md).

@ -352,7 +352,7 @@ for more on debugging, including how to debug Java code.
### Testing
For information on running tests, see
[Android Test Instructions](/testing/android/docs/README.md)
[Android Test Instructions](/docs/testing/android_test_instructions.md)
### Faster Edit/Deploy

@ -44,7 +44,7 @@ After writing a test, you can run it by:
Typically, the "relevant gn target" is simply the target containing the
other files in the same directory.
- Rebuild.
- Run the test using the process described [here](/testing/android/docs/todo.md).
- Run the test
## Instrumentation test features
@ -278,15 +278,10 @@ test cases according to the feature they're testing.
@Feature doesn't have an inherent function, but it can be used to filter tests
via the test runner's
[-A/--annotation](/testing/android/docs/todo.md) and
[-E/--exclude-annotation](/testing/android/docs/todo.md) flags. For example,
this would run only the tests with @Feature annotations containing at least
"Sync" in `chrome_public_test_apk`:
`-A/--annotation` and `-E/--exclude-annotation` flags. For example, this would
run only the tests with @Feature annotations containing at least "Sync" in
`chrome_public_test_apk`:
```bash
out/Debug/bin/run_chrome_public_test_apk -A Feature=Sync
```
## Advanced
- [Creating a component for use in instrumentation tests.](/testing/android/docs/components_for_testing.md)

@ -1,3 +1,339 @@
# Android Test Instructions
# Android Testing in Chromium
See [Android testing in Chromium](/testing/android/docs/README.md)
[TOC]
## Test Types
- **[gtests]**: For writing unit tests in C++. Most tests are cross-platform.
- **Browser Tests**: Built on top of gtest. Used to write integration tests.
- **[Robolectric]**: JUnit tests that run on the host machine by emulating (or
mocking) Android APIs.
- **[Instrumentation Tests]**: JUnit tests that run on devices / emulators.
- **Unit Instrumentation Tests**: Unit tests that do not require
initializing the Content layer. These use [BaseActivityTestRule] or
[BlankUiTestActivityTestCase] based on `BaseActivityTestRule`.
- **Integration Instrumentation Tests**: Instrumentation tests that require
initializing the Content layer to test a certain feature in the end-to-end
flow. These typically use more specialized test rules such as
[ContentShellActivityTestRule] or [ChromeActivityTestRule].
[gtests]: android_gtests.md
[Robolectric]: android_robolectric_tests.md
[Instrumentation Tests]: android_instrumentation_tests.md
[BaseActivityTestRule]: https://source.chromium.org/chromium/chromium/src/+/main:base/test/android/javatests/src/org/chromium/base/test/BaseActivityTestRule.java
[BlankUiTestActivityTestCase]: https://source.chromium.org/chromium/chromium/src/+/main:ui/android/javatests/src/org/chromium/ui/test/util/BlankUiTestActivityTestCase.java
[ContentShellActivityTestRule]: https://source.chromium.org/chromium/chromium/src/+/main:content/shell/android/javatests/src/org/chromium/content_shell_apk/ContentShellActivityTestRule.java
[ChromeActivityTestRule]: https://source.chromium.org/chromium/chromium/src/+/main:chrome/test/android/javatests/src/org/chromium/chrome/test/ChromeActivityTestRule.java
## Device Setup
### Physical Device Setup
#### Root Access
Running tests requires root access, which requires using a userdebug build on
your device.
To use a userdebug build, see
[Running Builds](https://source.android.com/setup/build/running.html). Googlers
can refer to [this page](https://goto.google.com/flashdevice).
If you can't run "adb root", you will get an error when trying to install
the test APKs like "adb: error: failed to copy" and
"remote secure_mkdirs failed: Operation not permitted" (use "adb unroot" to
return adb to normal).
#### ADB Debugging
The adb executable exists within the Android SDK:
```shell
third_party/android_sdk/public/platform-tools/adb
```
In order to allow the ADB to connect to the device, you must enable USB
debugging:
* Developer options are hidden by default. To unhide them:
* Go to "About phone"
* Tap 10 times on "Build number"
* The "Developer options" menu will now be available.
* Check "USB debugging".
* Un-check "Verify apps over USB".
#### Screen
You **must** ensure that the screen stays on while testing: `adb shell svc power
stayon usb` Or do this manually on the device: Settings -> Developer options ->
Stay Awake.
If this option is greyed out, stay awake is probably disabled by policy. In that
case, get another device or log in with a normal, unmanaged account (because the
tests will break in exciting ways if stay awake is off).
#### Disable Verify Apps
You may see a dialog like [this
one](http://www.samsungmobileusa.com/simulators/ATT_GalaxyMega/mobile/screens/06-02_12.jpg),
which states, _Google may regularly check installed apps for potentially harmful
behavior._ This can interfere with the test runner. To disable this dialog, run:
```shell
adb shell settings put global package_verifier_enable 0
```
### Using Emulators
Running tests on emulators is the same as [on device](#Running-Tests). Refer to
[android_emulator.md](/docs/android_emulator.md) for setting up emulators.
## Building Tests
If you're adding a new test file, you'll need to explicitly add it to a gn
target. If you're adding a test to an existing file, you won't need to make gn
changes, but you may be interested in where your test winds up. In either case,
here are some guidelines for where a test belongs:
### C++
C++ test files typically belong in `<top-level directory>_unittests` (e.g.
`base_unittests` for `//base`). There are a few exceptions -- browser tests are
typically their own target (e.g. `content_browsertests` for `//content`, or
`browser_tests` for `//chrome`), and some unit test suites are broken at the
second directory rather than the top-level one.
### Java
Java test files vary a bit more widely than their C++ counterparts:
- Instrumentation test files -- i.e., tests that will run on a device --
typically belong in either `<top-level directory>_javatests` or `<top-level
directory>_test_java`. Regardless, they'll wind up getting packaged into one
of a few test APKs:
- `webview_instrumentation_test_apk` for anything in `//android_webview`
- `content_shell_test_apk` for anything in `//content` or below
- `chrome_public_test_apk` for most things in `//chrome`
- JUnit or Robolectric test files -- i.e., tests that will run on the host --
typically belong in `<top-level directory>_junit_tests` (e.g.
`base_junit_tests` for `//base`), though here again there are cases
(particularly in `//components`) where suites are split at the second
directory rather than the top-level one.
Once you know what to build, just do it like you normally would build anything
else, e.g.: `ninja -C out/Release chrome_public_test_apk`
### Determining Test Target
If you do not know what target a test file belongs to, you can use
`//tools/autotest.py` to figure it out fo you:
```sh
# Builds relevant test target and then runs the test:
tools/autotest.py -C <output directory> TestClassName
```
## Running Tests
All functional tests should be runnable via the wrapper scripts generated at
build time:
```sh
<output directory>/bin/run_<target_name> [options]
```
Note that tests are sharded across all attached devices unless explicitly told
to do otherwise by `-d/--device`.
The commands used by the buildbots are printed in the logs. Look at
https://build.chromium.org/ to duplicate the same test command as a particular
builder.
### Listing Available Tests
Use `--list-tests` to list what tests are available.
```sh
# Prints out all available tests:
<output directory>/bin/run_<target_name> --list-tests
# Prints out all available tests that match a filter:
<output directory>/bin/run_<target_name> --list-tests -f "*MyFilter*"
```
### INSTALL\_FAILED\_CONTAINER\_ERROR or INSTALL\_FAILED\_INSUFFICIENT\_STORAGE
If you see this error when the test runner is attempting to deploy the test
binaries to the AVD emulator, you may need to resize your userdata partition
with the following commands:
```shell
# Resize userdata partition to be 1G
resize2fs android_emulator_sdk/sdk/system-images/android-25/x86/userdata.img 1G
# Set filesystem parameter to continue on errors; Android doesn't like some
# things e2fsprogs does.
tune2fs -e continue android_emulator_sdk/sdk/system-images/android-25/x86/userdata.img
```
### AdbCommandFailedError: failed to stat remote object
There's a known issue (https://crbug.com/1094062) where the unit test binaries can fail on
Android R and later: if you see this error, try rerunning on an Android version
with API level <= 29 (Android <= Q).
## Symbolizing Crashes
Crash stacks are logged and can be viewed using `adb logcat`. To symbolize the
traces, define `CHROMIUM_OUTPUT_DIR=$OUTDIR` where `$OUTDIR` is the argument you
pass to `ninja -C`, and pipe the output through
`third_party/android_platform/development/scripts/stack`. If
`$CHROMIUM_OUTPUT_DIR` is unset, the script will search `out/Debug` and
`out/Release`. For example:
```shell
# If you build with
ninja -C out/Debug chrome_public_test_apk
# You can run:
adb logcat -d | third_party/android_platform/development/scripts/stack
# If you build with
ninja -C out/android chrome_public_test_apk
# You can run:
adb logcat -d | CHROMIUM_OUTPUT_DIR=out/android third_party/android_platform/development/scripts/stack
# or
export CHROMIUM_OUTPUT_DIR=out/android
adb logcat -d | third_party/android_platform/development/scripts/stack
```
## Robolectric Tests
JUnit tests are Java unittests running on the host instead of the target device.
They are faster to run and therefore are recommended over instrumentation tests
when possible.
The JUnits tests are usually following the pattern of *target*\_junit\_tests,
for example, `content_junit_tests` and `chrome_junit_tests`.
When adding a new JUnit test, the associated `BUILD.gn` file must be updated.
For example, adding a test to `chrome_junit_tests` requires to update
`chrome/android/BUILD.gn`.
```shell
# Build the test suite.
ninja -C out/Default chrome_junit_tests
# Run the test suite.
out/Default/bin/run_chrome_junit_tests
# Run a subset of tests. You might need to pass the package name for some tests.
out/Default/bin/run_chrome_junit_tests -f "org.chromium.chrome.browser.media.*"
```
### Debugging
Similar to [debugging apk targets](/docs/android_debugging_instructions.md#debugging-java):
```shell
out/Default/bin/run_chrome_junit_tests --wait-for-java-debugger
out/Default/bin/run_chrome_junit_tests --wait-for-java-debugger # Specify custom port via --debug-socket=9999
```
## Gtests
```shell
# Build a test suite
ninja -C out/Release content_unittests
# Run a test suite
out/Release/bin/run_content_unittests [-vv]
# Run a subset of tests and enable some "please go faster" options:
out/Release/bin/run_content_unittests --fast-local-dev -f "ByteStreamTest.*"
```
## Instrumentation Tests
In order to run instrumentation tests, you must leave your device screen ON and
UNLOCKED. Otherwise, the test will timeout trying to launch an intent.
Optionally you can disable screen lock under Settings -> Security -> Screen Lock
-> None.
Next, you need to build the app, build your tests, and then run your tests
(which will install the APK under test and the test APK automatically).
Examples:
ContentShell tests:
```shell
# Build the tests:
ninja -C out/Release content_shell_test_apk
# Run the test suite:
out/Release/bin/run_content_shell_test_apk [-vv]
# Run a subset of tests and enable some "please go faster" options:
out/Release/bin/run_content_shell_test_apk --fast-local-dev -f "*TestClass*"
```
Android WebView tests:
See [WebView's instructions](/android_webview/docs/test-instructions.md).
In order to run a subset of tests, use -f to filter based on test class/method
or -A/-E to filter using annotations.
More Filtering examples:
```shell
# Run a specific test class
out/Debug/bin/run_content_shell_test_apk -f "AddressDetectionTest.*"
# Run a specific test method
out/Debug/bin/run_content_shell_test_apk -f AddressDetectionTest#testAddressLimits
# Run a subset of tests by size (Smoke, SmallTest, MediumTest, LargeTest,
# EnormousTest)
out/Debug/bin/run_content_shell_test_apk -A Smoke
# Run a subset of tests by annotation, such as filtering by Feature
out/Debug/bin/run_content_shell_test_apk -A Feature=Navigation
```
You might want to add stars `*` to each as a regular expression, e.g.
`*`AddressDetectionTest`*`
### Debugging
Similar to [debugging apk targets](/docs/android_debugging_instructions.md#debugging-java):
```shell
out/Debug/bin/run_content_shell_test_apk --wait-for-java-debugger
```
### Deobfuscating Java Stacktraces
If running with `is_debug=false`, Java stacks from logcat need to be fixed up:
```shell
build/android/stacktrace/java_deobfuscate.py out/Release/apks/ChromePublicTest.apk.mapping < stacktrace.txt
```
Any stacks produced by test runner output will already be deobfuscated.
## Running Blink Web Tests
See [Web Tests](web_tests.md).
## Running Telemetry (Perf) Tests
See [Telemetry](https://chromium.googlesource.com/catapult/+/HEAD/telemetry/README.md).
## Running GPU tests
(e.g. the "Android Debug (Nexus 7)" bot on the chromium.gpu waterfall)
See https://www.chromium.org/developers/testing/gpu-testing for details. Use
`--browser=android-content-shell`. Examine the stdio from the test invocation on
the bots to see arguments to pass to `src/content/test/gpu/run_gpu_test.py`.

@ -15,9 +15,6 @@ for one platform.
* **[gtest]** is Google's C++ test framework,
which helps you write better C++ tests in Chromium.
gtest is test framework for unit tests in Chromium and browser tests are built on top of it.
* **[Junit]** is a unit testing framework
for the Java programming language, and it is used to write
unit tests on Android for Java code in Chromium.
* **Browser Tests** is built on top of gtest, and it is used to write integration tests
and e2e tests in Chromium.
<!-- TODO(leilei) Add link to browser tests --->
@ -27,8 +24,9 @@ for one platform.
in a test renderer (`content_shell`) and comparing the rendered output or
JavaScript output against an expected output file.
Web Tests are required to launch new W3C API support in Chromium.
* **[Instrumentation Tests]** is a test framework specific for Android platform,
it is used to write integration tests or e2e tests for Chromium on Android.
* **[Robolectric]** is build on top of JUnit 4. It emulates Android APIs so
that tests can be run on the host machine instead of on devices / emulators.
* **[Instrumentation Tests]** are JUnit tests that run on devices / emulators.
* **[EarlGrey]** is the integration testing framework used by Chromium for iOS.
* **[Telemetry]** is the performance testing framework used by Chromium.
It allows you to perform arbitrary actions on a set of web pages and
@ -42,10 +40,10 @@ The following table shows which types of test works on which platforms.
| | Linux | Windows | Mac | Android | iOS | CrOS |
|:----------------------------|:--------|:--------|:--------|:--------|:--------|:--------|
| gtest(C++) | &#8730; | &#8730; | &#8730; | &#8730; | &#8730; | &#8730; |
| Junit(Java) | | | | &#8730; | | |
| Browser Tests(C++) | &#8730; | &#8730; | &#8730; | &#8730; | | |
| Web Tests(HTML, JS) | &#8730; | &#8730; | &#8730; | | | |
| Telemetry(Python) | &#8730; | &#8730; | &#8730; | &#8730; | | &#8730; |
| Robolectric(Java) | | | | &#8730; | | |
| Instrumentation Tests(Java) | | | | &#8730; | | |
| EarlGrey | | | | | &#8730; | |
| Fuzzer Tests(C++) | &#8730; | &#8730; | &#8730; | &#8730; | | &#8730; |
@ -184,8 +182,8 @@ Tests are not configured to upload metrics, such as UMA, UKM or crash reports.
[gtest]: https://github.com/google/googletest
[Simple gtests]: https://github.com/google/googletest/blob/main/docs/primer.md#simple-tests
[Junit]: https://developer.android.com/training/testing/junit-rules
[Instrumentation Tests]: https://chromium.googlesource.com/chromium/src/+/main/testing/android/docs/instrumentation.md
[Robolectric]: android_robolectric_tests.md
[Instrumentation Tests]: https://chromium.googlesource.com/chromium/src/+/main/docs/testing/android_instrumentation_tests.md
[EarlGrey]: https://github.com/google/EarlGrey
[Telemetry]: https://chromium.googlesource.com/catapult/+/HEAD/telemetry/README.md
[Fuzzer Tests]: https://chromium.googlesource.com/chromium/src/+/main/testing/libfuzzer/README.md

@ -1 +0,0 @@
*

@ -1,35 +0,0 @@
# Android Testing in Chromium
## Concepts
- **Native Unit Tests**: Normal Chromium unit tests based on the [gtest framework](/testing/android/docs/gtest_implementation.md). Tests for native code.
- **Java Unit Tests**: JUnit tests that run on the host machine using [Robolectric](http://robolectric.org) to emulate Android APIs.
- **Instrumentation Tests**: JUnit tests that run on Android devices (or emulators).
- **Unit Instrumentation Tests**: Instrumentation tests that test an individual feature. They do not require starting up ContentShell (or Chrome browser). These use [BaseActivityTestRule](https://source.chromium.org/chromium/chromium/src/+/main:base/test/android/javatests/src/org/chromium/base/test/BaseActivityTestRule.java) or [BlankUiTestActivityTestCase](https://source.chromium.org/chromium/chromium/src/+/main:ui/android/javatests/src/org/chromium/ui/test/util/BlankUiTestActivityTestCase.java) based on `BaseActivityTestRule`.
- **Integration Instrumentation Tests**: Instrumentation tests that bring up ContentShell (or Chrome browser) to test a certain feature in the end-to-end flow. These typically use more specialized test rules such as [ContentShellActivityTestRule](https://source.chromium.org/chromium/chromium/src/+/main:content/shell/android/javatests/src/org/chromium/content_shell_apk/ContentShellActivityTestRule.java) or [ChromeActivityTestRule](https://source.chromium.org/chromium/chromium/src/+/main:chrome/test/android/javatests/src/org/chromium/chrome/test/ChromeActivityTestRule.java).
## How do I...
- **set up**
- [... set up a device for local testing?](/testing/android/docs/android_test_instructions.md#physical-device-setup)
- [... set up an emulator for local testing?](/docs/android_emulator.md)
- **writing tests**
- [... write an instrumentation test?](/testing/android/docs/instrumentation.md)
- [... write a JUnit or Robolectric test?](/testing/android/docs/junit.md)
- [... write a test that needs to mock native calls?](/third_party/jni_zero/README.md#testing-mockable-natives)
- **running tests**
- [... run a gtest?](/testing/android/docs/android_test_instructions.md#gtests)
- [... run an instrumentation test?](/testing/android/docs/android_test_instructions.md#instrumentation-tests)
- [... run a JUnit test?](/testing/android/docs/android_test_instructions.md#junit-tests)
- [... run the Web tests?](/docs/testing/web_tests.md)
- [... run a telemetry test?](https://chromium.googlesource.com/catapult/+/HEAD/telemetry/README.md)
- **debugging tests**
- [... debug junit tests?](/testing/android/docs/android_test_instructions.md#junit-tests-debugging)
- [... debug flaky tests?](/testing/android/docs/todo.md)
- **miscellaneous**
- [... use code coverage for Java tests?](/build/android/docs/coverage.md)
## How does it work on Android?
- [gtests](/testing/android/docs/gtest_implementation.md)
- [instrumentation tests](https://source.android.com/compatibility/tests/development/instrumentation)
- [junit tests](/testing/android/docs/junit.md)

@ -1,312 +0,0 @@
# Building and Running Android Tests
[TOC]
## Device Setup
### Physical Device Setup
#### Root Access
Running tests requires root access, which requires using a userdebug build on
your device.
To use a userdebug build, see
[Running Builds](https://source.android.com/setup/build/running.html). Googlers
can refer to [this page](https://goto.google.com/flashdevice).
If you can't run "adb root", you will get an error when trying to install
the test APKs like "adb: error: failed to copy" and
"remote secure_mkdirs failed: Operation not permitted" (use "adb unroot" to
return adb to normal).
#### ADB Debugging
The adb executable exists within the Android SDK:
```shell
third_party/android_sdk/public/platform-tools/adb
```
In order to allow the ADB to connect to the device, you must enable USB
debugging:
* Developer options are hidden by default. To unhide them:
* Go to "About phone"
* Tap 10 times on "Build number"
* The "Developer options" menu will now be available.
* Check "USB debugging".
* Un-check "Verify apps over USB".
#### Screen
You **must** ensure that the screen stays on while testing: `adb shell svc power
stayon usb` Or do this manually on the device: Settings -> Developer options ->
Stay Awake.
If this option is greyed out, stay awake is probably disabled by policy. In that
case, get another device or log in with a normal, unmanaged account (because the
tests will break in exciting ways if stay awake is off).
#### Disable Verify Apps
You may see a dialog like [this
one](http://www.samsungmobileusa.com/simulators/ATT_GalaxyMega/mobile/screens/06-02_12.jpg),
which states, _Google may regularly check installed apps for potentially harmful
behavior._ This can interfere with the test runner. To disable this dialog, run:
```shell
adb shell settings put global package_verifier_enable 0
```
### Using Emulators
Running tests on emulators is the same as [on device](#Running-Tests). Refer to
[android_emulator.md](/docs/android_emulator.md) for setting up emulators.
## Building Tests
If you're adding a new test file, you'll need to explicitly add it to a gn
target. If you're adding a test to an existing file, you won't need to make gn
changes, but you may be interested in where your test winds up. In either case,
here are some guidelines for where a test belongs:
### C++
C++ test files typically belong in `<top-level directory>_unittests` (e.g.
`base_unittests` for `//base`). There are a few exceptions -- browser tests are
typically their own target (e.g. `content_browsertests` for `//content`, or
`browser_tests` for `//chrome`), and some unit test suites are broken at the
second directory rather than the top-level one.
### Java
Java test files vary a bit more widely than their C++ counterparts:
- Instrumentation test files -- i.e., tests that will run on a device --
typically belong in either `<top-level directory>_javatests` or `<top-level
directory>_test_java`. Regardless, they'll wind up getting packaged into one
of a few test APKs:
- `webview_instrumentation_test_apk` for anything in `//android_webview`
- `content_shell_test_apk` for anything in `//content` or below
- `chrome_public_test_apk` for most things in `//chrome`
- JUnit or Robolectric test files -- i.e., tests that will run on the host --
typically belong in `<top-level directory>_junit_tests` (e.g.
`base_junit_tests` for `//base`), though here again there are cases
(particularly in `//components`) where suites are split at the second
directory rather than the top-level one.
Once you know what to build, just do it like you normally would build anything
else, e.g.: `ninja -C out/Release chrome_public_test_apk`
### Determining Test Target
If you do not know what target a test file belongs to, you can use
`//tools/autotest.py` to figure it out fo you:
```sh
# Builds relevant test target and then runs the test:
tools/autotest.py -C <output directory> TestClassName
```
## Running Tests
All functional tests should be runnable via the wrapper scripts generated at
build time:
```sh
<output directory>/bin/run_<target_name> [options]
```
Note that tests are sharded across all attached devices unless explicitly told
to do otherwise by `-d/--device`.
The commands used by the buildbots are printed in the logs. Look at
https://build.chromium.org/ to duplicate the same test command as a particular
builder.
### Listing Available Tests
Use `--list-tests` to list what tests are available.
```sh
# Prints out all available tests:
<output directory>/bin/run_<target_name> --list-tests
# Prints out all available tests that match a filter:
<output directory>/bin/run_<target_name> --list-tests -f "*MyFilter*"
```
### INSTALL\_FAILED\_CONTAINER\_ERROR or INSTALL\_FAILED\_INSUFFICIENT\_STORAGE
If you see this error when the test runner is attempting to deploy the test
binaries to the AVD emulator, you may need to resize your userdata partition
with the following commands:
```shell
# Resize userdata partition to be 1G
resize2fs android_emulator_sdk/sdk/system-images/android-25/x86/userdata.img 1G
# Set filesystem parameter to continue on errors; Android doesn't like some
# things e2fsprogs does.
tune2fs -e continue android_emulator_sdk/sdk/system-images/android-25/x86/userdata.img
```
### AdbCommandFailedError: failed to stat remote object
There's a known issue (https://crbug.com/1094062) where the unit test binaries can fail on
Android R and later: if you see this error, try rerunning on an Android version
with API level <= 29 (Android <= Q).
## Symbolizing Crashes
Crash stacks are logged and can be viewed using `adb logcat`. To symbolize the
traces, define `CHROMIUM_OUTPUT_DIR=$OUTDIR` where `$OUTDIR` is the argument you
pass to `ninja -C`, and pipe the output through
`third_party/android_platform/development/scripts/stack`. If
`$CHROMIUM_OUTPUT_DIR` is unset, the script will search `out/Debug` and
`out/Release`. For example:
```shell
# If you build with
ninja -C out/Debug chrome_public_test_apk
# You can run:
adb logcat -d | third_party/android_platform/development/scripts/stack
# If you build with
ninja -C out/android chrome_public_test_apk
# You can run:
adb logcat -d | CHROMIUM_OUTPUT_DIR=out/android third_party/android_platform/development/scripts/stack
# or
export CHROMIUM_OUTPUT_DIR=out/android
adb logcat -d | third_party/android_platform/development/scripts/stack
```
## JUnit tests
JUnit tests are Java unittests running on the host instead of the target device.
They are faster to run and therefore are recommended over instrumentation tests
when possible.
The JUnits tests are usually following the pattern of *target*\_junit\_tests,
for example, `content_junit_tests` and `chrome_junit_tests`.
When adding a new JUnit test, the associated `BUILD.gn` file must be updated.
For example, adding a test to `chrome_junit_tests` requires to update
`chrome/android/BUILD.gn`.
```shell
# Build the test suite.
ninja -C out/Default chrome_junit_tests
# Run the test suite.
out/Default/bin/run_chrome_junit_tests
# Run a subset of tests. You might need to pass the package name for some tests.
out/Default/bin/run_chrome_junit_tests -f "org.chromium.chrome.browser.media.*"
```
### Debugging
Similar to [debugging apk targets](/docs/android_debugging_instructions.md#debugging-java):
```shell
out/Default/bin/run_chrome_junit_tests --wait-for-java-debugger
out/Default/bin/run_chrome_junit_tests --wait-for-java-debugger # Specify custom port via --debug-socket=9999
```
## Gtests
```shell
# Build a test suite
ninja -C out/Release content_unittests
# Run a test suite
out/Release/bin/run_content_unittests [-vv]
# Run a subset of tests and enable some "please go faster" options:
out/Release/bin/run_content_unittests --fast-local-dev -f "ByteStreamTest.*"
```
## Instrumentation Tests
In order to run instrumentation tests, you must leave your device screen ON and
UNLOCKED. Otherwise, the test will timeout trying to launch an intent.
Optionally you can disable screen lock under Settings -> Security -> Screen Lock
-> None.
Next, you need to build the app, build your tests, and then run your tests
(which will install the APK under test and the test APK automatically).
Examples:
ContentShell tests:
```shell
# Build the tests:
ninja -C out/Release content_shell_test_apk
# Run the test suite:
out/Release/bin/run_content_shell_test_apk [-vv]
# Run a subset of tests and enable some "please go faster" options:
out/Release/bin/run_content_shell_test_apk --fast-local-dev -f "*TestClass*"
```
Android WebView tests:
See [WebView's instructions](/android_webview/docs/test-instructions.md).
In order to run a subset of tests, use -f to filter based on test class/method
or -A/-E to filter using annotations.
More Filtering examples:
```shell
# Run a specific test class
out/Debug/bin/run_content_shell_test_apk -f "AddressDetectionTest.*"
# Run a specific test method
out/Debug/bin/run_content_shell_test_apk -f AddressDetectionTest#testAddressLimits
# Run a subset of tests by size (Smoke, SmallTest, MediumTest, LargeTest,
# EnormousTest)
out/Debug/bin/run_content_shell_test_apk -A Smoke
# Run a subset of tests by annotation, such as filtering by Feature
out/Debug/bin/run_content_shell_test_apk -A Feature=Navigation
```
You might want to add stars `*` to each as a regular expression, e.g.
`*`AddressDetectionTest`*`
### Debugging
Similar to [debugging apk targets](/docs/android_debugging_instructions.md#debugging-java):
```shell
out/Debug/bin/run_content_shell_test_apk --wait-for-java-debugger
```
### Deobfuscating Java Stacktraces
If running with `is_debug=false`, Java stacks from logcat need to be fixed up:
```shell
build/android/stacktrace/java_deobfuscate.py out/Release/apks/ChromePublicTest.apk.mapping < stacktrace.txt
```
Any stacks produced by test runner output will already be deobfuscated.
## Running Blink Web Tests
See [Web Tests](web_tests.md).
## Running GPU tests
(e.g. the "Android Debug (Nexus 7)" bot on the chromium.gpu waterfall)
See https://www.chromium.org/developers/testing/gpu-testing for details. Use
`--browser=android-content-shell`. Examine the stdio from the test invocation on
the bots to see arguments to pass to `src/content/test/gpu/run_gpu_test.py`.

@ -1,162 +0,0 @@
[< Instrumentation](/testing/android/docs/instrumentation.md)
# Creating a component for testing
Creating a component, e.g. a [ContentProvider][1] or [Service][2], for use
in testing can be tricky. This is particularly true when:
- app code and test code are in separate APKs
- the test APK instruments the app APK
- the test Service depends on app code.
This doc explains the pitfalls in creating a test component and how to
avoid them.
[TOC]
## What can go wrong with a test component
> **tl;dr:** test components may not be able to access app code when defined in
> the test APK.
Chromium's instrumentation test suites are all currently set up in the manner
described above: app code is in one APK (the _APK under test_), test code is
in another APK (the _test APK_), and auxiliary code, when necessary, is in
one or more other APKs (_support APKs_). Test APKs build against app code
but do not retain it in their .dex files. This reduces the size of test APKs
and avoids potentially conflicting definitions. At instrumentation runtime,
the test code is loaded into the app package's process, and it's consequently
able to access code defined in the app APK's .dex file(s). Test components,
however, run in the test package's process and only have access to code in the
test APK. While test components referencing app code will build without issue,
they will fail to link at runtime and will consequently not be able to be
instantiated.
For example, here's the logcat from an attempt to use a test Service,
`TestPostMessageService`, that extended an app Service, `PostMessageService`.
Note that the runtime link failed because the superclass couldn't be resolved,
and the test Service could not be instantiated as a result.
``` text
Unable to resolve superclass of Lorg/chromium/chrome/browser/customtabs/TestPostMessageService; (184)
Link of class 'Lorg/chromium/chrome/browser/customtabs/TestPostMessageService;' failed
...
FATAL EXCEPTION: main
Process: org.chromium.chrome.tests, PID: 30023
java.lang.RuntimeException:
Unable to instantiate service org.chromium.chrome.browser.customtabs.TestPostMessageService:
java.lang.ClassNotFoundException:
Didn't find class "org.chromium.chrome.browser.customtabs.TestPostMessageService" on path:
DexPathList[
[zip file "/system/framework/android.test.runner.jar",
zip file "/data/app/org.chromium.chrome.tests-1.apk"],
nativeLibraryDirectories=[
/data/app-lib/org.chromium.chrome.tests-1,
/vendor/lib,
/system/lib]]
at android.app.ActivityThread.handleCreateService(ActivityThread.java:2543)
at android.app.ActivityThread.access$1800(ActivityThread.java:135)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1278)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:136)
at android.app.ActivityThread.main(ActivityThread.java:5001)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:785)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:601)
at dalvik.system.NativeStart.main(Native Method)
Caused by: java.lang.ClassNotFoundException:
Didn't find class "org.chromium.chrome.browser.customtabs.TestPostMessageService" on path:
DexPathList[
[zip file "/system/framework/android.test.runner.jar",
zip file "/data/app/org.chromium.chrome.tests-1.apk"],
nativeLibraryDirectories=[
/data/app-lib/org.chromium.chrome.tests-1,
/vendor/lib,
/system/lib]]
at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56)
at java.lang.ClassLoader.loadClass(ClassLoader.java:497)
at java.lang.ClassLoader.loadClass(ClassLoader.java:457)
at android.app.ActivityThread.handleCreateService(ActivityThread.java:2540)
... 10 more
```
## How to implement a test component
There are (at least) two mechanisms for avoiding the failures described above:
using a support APK or using sharedUserIds.
### Use a support APK
Putting the Service in a support APK lets the build system include all necessary
code in the .dex without fear of conflicting definitions, as nothing in the
support APK runs in the same package or process as the test or app code.
To do this:
**Create the component.**
It should either be in an existing directory used by code in the appropriate
support APK or a new directory for such purpose. In particular, it should be
in neither a directory containing app code nor a directory containing test
code.
``` java
package org.chromium.chrome.test;
import org.chromium.chrome.MyAppService;
public class MyTestService extends MyAppService {
...
}
```
**Put the component in a separate gn target.**
This can either be a target upon which the support APK already depends or a new
target.
``` python
android_library("my_test_service") {
sources = [ "src/org/chromium/chrome/test/MyTestService.java" ]
deps = [ ... ]
}
```
The support APK must depend on this target. The target containing your test
code can also depend on this target if you want to refer to the component
class directly, e.g., when binding a Service.
> **NOTE:** Even if your test code directly depends on the service target,
> you won't be able to directly reference the test component instance from
> test code. Checking a test component's internal state will require adding
> APIs specifically for that purpose.
**Define the component in the support APK manifest.**
``` xml
<manifest ...
package="org.chromium.chrome.tests.support" >
<application>
<service android:name="org.chromium.chrome.test.MyTestService"
android:exported="true"
tools:ignore="ExportedService">
...
</service>
...
</application>
</manifest>
```
### Use a sharedUserId
Using a [sharedUserId][3] will allow your component to run in your app's
process.
> Because this requires modifying the app manifest, it is not recommended at
> this time and is intentionally not further documented here.
[1]: https://developer.android.com/reference/android/content/ContentProvider.html
[2]: https://developer.android.com/reference/android/app/Service.html
[3]: https://developer.android.com/guide/topics/manifest/manifest-element.html#uid

@ -1,287 +0,0 @@
# JUnit 4 Migration
As of Android 24 (N), JUnit3 style javatests have been deprecated for the new
JUnit4-based [Android Testing Support Library][1].
We are in the process of changing all instrumentation tests in chromium to
JUnit4 style. This doc explains the differences between JUnit3 and JUnit4
instrumentation tests and how to write or convert them.
[TOC]
## Differences between JUnit3 and JUnit4 instrumentation tests
| | JUnit3 | JUnit4 |
|--------------|-----------------------------------------------------|------------------------------------------|
| Inheritance | Tests extends TestCase or child classes | No inheritance. |
| Test methods | methods named with test prefix | methods annotated with @Test |
| Set up | setUp() method | public method annotated with @Before |
| Tear down | tearDown() method | public method annotated with @After |
| Test runner | declared within test apk AndroidManifest.xml | Must specify `chromium-junit4:"true"` |
| Class runner | N/A | @RunWith(XClassRunner.class) |
| Assertion | Extends from junit.framework.Assert, inherited APIs | Use static methods from org.junit.Assert |
> Please note that during the migration, we support running JUnit3 and JUnit4
> tests in the same apk. This requires two tags, one each for JUnit3 and JUnit4.
> The tag for the JUnit4 runner must specify `chromium-junit4:"true"`
> ([Example][2])
- **Other JUnit4 features**:
- Tests can be annotated to expect an exception, e.g.
`@Test(expected=MyException.class)`. Tests annotated this way will
fail if they do not throw the given exception.
- **Test suite set up**: public static method annotated with `@BeforeClass`
- **Test suite tear down**: public static method annotated with
`@AfterClass`
- **Replacement for JUnit3 test base classes**
- [`TestRule`][3]:
- TestRule is a class to **outsource your test setUp, tearDown, and
utility methods**. Since there are no more interitance and TestBase classes,
one should use TestRule for any API calls provided by its test base classes
previously.
- One test can declare multiple TestRules and the class runner will run all of
them. If the order of the TestRule matters to you, use
[`RuleChain`][8]
- [`ActivityTestRule`][4]
- `ActivityTestRule` is a special `TestRule` provided by Android Testing
Support Library that allows tests to launch an Activity.
([Documentation][4])
## Example Code of JUnit3 test and JUnit4 test
JUnit3:
```java
public class MyTestClass extends MyActivityInstrumentationTestCase2<TestActivity> {
@Override
protected void setUp(){
super.setUp();
setActivityIntent(new Intent());
getActivity();
}
@Override
protected void tearDown() {
specialActionFromSuper();
super.tearDown();
}
public void testA() {
assertEquals(1, 1);
}
}
```
JUnit4:
```java
@RunWith(BaseJUnit4ClassRunner.class);
public class TestClass {
@Rule public ActivityTestRule<TestActivity> mRule = new ActivityTestRule<>(TestActivity.class);
@Before
public void setUp() { //Must be public
mRule.launchActivity(new Intent());
}
@After
public void tearDown() { //Must be public
mRule.specialActionFromActivityTestRule();
}
@Test
public void testA() {
Assert.assertEquals(1, 1);
}
}
```
## Migration process
1. Add required libraries to your target dependencies in BUILD.gn
- JUnit 4 library: `//third_party/junit`
- Android Testing Support Rules:
- `//third_party/androidx:androidx_test_runner_java`
(for `AndroidJUnitRunner`, etc)
- `//third_party/androidx:androidx_test_rules_java`
(for `ActivityTestRule`, etc)
1. Add class runner to your test apk manifest.
([example][2])
- Keep in mind you can have multiple instrumentations in your manifest.
Our test runner will run JUnit4 tests with JUnit4 runner and JUnit3
tests with non-JUnit4 runner.
1. Refactor TestBase class to a TestRule class.
([example CL](https://codereview.chromium.org/2632043002))
- +yolandyan will do this part, however, if you did refactoring yourself,
please add him as a reviewer for your CL and enjoy his eternal appreciation!
1. Use [auto migrate script][5] to or manually convert all JUnit3 tests to
JUnit4 style in a your javatest directory
- we understand it's tedious to just manually write all the annotations,
change modifiers, etc to convert all the javatest, so we created an auto
change script that helps you to convert all the javatests in a certain
directory. Please check its [README page][5]
on instructions.
## Customized TestRule example
TestRule:
```java
public class MyRule implements TestRule {
// 1: Add utility methods...
@Override
public Statement apply(final Statement base, Description desc) {
return new Statement() {
@Override
public void evaluate() {
// 2: Code here runs before @Before method
base.evaluate()
// 3: Code here runs after @After method
}
}
}
}
```
## Command Line Flags
In our Junit3 tests command line flags (set by the CommandLineFlag annotations) were inherited from the
test base classes. As an example, ChromeActivityTestBase is annotated with:
```java
@CommandLineFlags.Add({ChromeSwitches.DISABLE_FIRST_RUN_EXPERIENCE, ...
```
and as a result any test in a class derived from ChromeActivityTestBase will disable the first run experience.
The Junit4 tests classes are not however, derived from test base classes; instead their behavior is defined by
test rules. To support this our Junit4 test runner will examine the command line flag annotations on all rules
referenced with @Rule annotations in the test class. In addition, where one rule is derived from another, the
command line flags propogate through the hierarchy of rules. See, for example, [BottomSheetTestRule][11]
Note:- This has only recently been implemented, so is not yet used in all tests. See [this bug][12]
The CommandLineFlags annonations are more fully documented in the [CommandLineFlags class][13]
## Common Errors
1. Instrumentation tests that rely on test thread to have message handler
will not work. For example error message:
java.lang.RuntimeException: Can't create handler inside thread that has not called Looper.prepare()
or
java.lang.IllegalStateException: The current thread must have a looper!
Please utilize `ActivityTestRule.runOnUiThread(Runnable r)` to refactor
these tests. For more, check this [GitHub issue][6]
1. Use `@UiThreadTest` with caution!!
- Currently, **@UiThreadTest is only effective when UiThreadTestRule or
ActivityTestRule is declared** in the test class.
- Please use **`androidx.test.annotation.UiThreadTest`, NOT
`android.test.UiThreadTest`**.
- When using @UiThreadTest, **it would cause `setUp` and `tearDown` to
run in Ui Thread** as well. Avoid that by calling [`runOnUiThread`][9]
or [`runOnMainSync`][10] with a Runnable.
```java
// Wrong test
public class Test {
@Rule
public ActivityTestRule<MyActivity> mRule = new ActivityTestRule<>(
MyActivity.class);
@Before
public void setUp() {
// Cause failure because this also runs on Ui Thread, while it
// is intended for Instrumentation worker thread
mRule.launchActivity();
}
@UiThreadTest
public void test() {
actionThatNeedsUiThread();
}
}
```
The correct thing to do is
```java
// Correct test
public class Test {
@Rule
public ActivityTestRule<MyActivity> mRule = new ActivityTestRule<>(
MyActivity.class);
@Before
public void setUp() {
mRule.launchActivity();
}
public void test() {
mRule.runOnUiThread(new Runnable() {
@Override
public void run() {
actionThatNeedsUiThread();
}
});
}
}
```
1. `assertEquals(float a, float b)` and `assertEquals(double a, double b)` are
deprecated in JUnit4's Assert class. **Despite only generating a warning at
build time, they fail at runtime.** Please use
`Assert.assertEquals(float a, float b, float delta)`
1. Errorprone expects all public methods starting with `test...` to be
annotated with `@Test`. Failure to meet that expectation will cause
errorprone to fail with something like this:
[JUnit4TestNotRun] Test method will not be run; please add @Test annotation
In particular, you may see this when attempting to disable tests. In that
case, the test should be annotated with both @DisabledTest and @Test.
## Common questions
- Q: Are `@Test` and `@LargeTest/@MediumTest/@SmallTest` annotation
both necessary?
- A: Yes, both are required for now. We plan to refactor this in the
future.
- Q: Isn't the inheritance of the Test classes just migrated to inheritance
of TestRules?
- A: Yes. During the migration, we plan to maintain a 1:1 mapping between
the test base classes and TestRules (e.g. ContentShellTestBase to
ContentShellTestRule in this
[CL](https://codereview.chromium.org/2632043002)).
This allows the auto convert script to replace API calls in any
JUnit3 tests. After the migration, we plan to refactor the TestRules to
be more modular.
If you have any other questions, feel free to report in [this bug][7].
## Links and Crbugs
- [Android Test Support Library documentation][1]
- [Auto change script][5]
- [Crbug for JUnit3 to JUnit4 migration][7]
[1]: https://developer.android.com/topic/libraries/testing-support-library/index.html
[2]: https://cs.chromium.org/chromium/src/android_webview/tools/system_webview_shell/layout_tests/AndroidManifest.xml?l=36
[3]: http://junit.org/junit4/javadoc/4.12/org/junit/rules/TestRule.html
[4]: https://developer.android.com/reference/androidx/test/rule/ActivityTestRule.html
[5]: https://github.com/yoland68/chromium-junit-auto-migrate
[6]: http://github.com/skyisle/android-test-kit/issues/121
[7]: https://bugs.chromium.org/p/chromium/issues/detail?id=640116
[8]: http://junit.org/junit4/javadoc/4.12/org/junit/rules/RuleChain.html
[9]: https://developer.android.com/reference/android/app/Instrumentation.html#runOnMainSync(java.lang.Runnable)
[10]: https://developer.android.com/reference/androidx/test/rule/UiThreadTestRule.html#runOnUiThread(java.lang.Runnable)
[11]: /chrome/test/android/javatests/src/org/chromium/chrome/test/BottomSheetTestRule.java
[12]: https://bugs.chromium.org/p/chromium/issues/detail?id=734553
[13]: /base/test/android/javatests/src/org/chromium/base/test/util/CommandLineFlags.java

@ -1,5 +0,0 @@
# This page hasn't been written yet!
If you'd like more information about what you were looking for, try
contacting one of the people listed in
[//testing/android OWNERS](https://chromium.googlesource.com/chromium/src/+/main/testing/android/OWNERS#2).