Consolidate Android testing docs /testing/android/docs -> /docs/testing
This should hopefully make them more discoverable. Bug: None Change-Id: I65885c50b655a8b867cdc9224bd1be59822dcb14 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4908056 Commit-Queue: Andrew Grieve <agrieve@chromium.org> Owners-Override: Andrew Grieve <agrieve@chromium.org> Reviewed-by: Haiyang Pan <hypan@google.com> Reviewed-by: James Shen <zhiyuans@google.com> Reviewed-by: Henrique Nakashima <hnakashima@chromium.org> Cr-Commit-Position: refs/heads/main@{#1205217}
This commit is contained in:

committed by
Chromium LUCI CQ

parent
58260cf066
commit
b5b3bb3d3e
@ -16,7 +16,7 @@ tests are organized by overall functionality instead). Put new tests into an
|
||||
existing test class or create a new one if a suitable one isn't available.
|
||||
|
||||
#### How to write intrumentation tests?
|
||||
See the [instrumentation test doc](/testing/android/docs/instrumentation.md).
|
||||
See the [instrumentation test doc](/docs/testing/android_instrumentation_tests.md).
|
||||
|
||||
#### How do tests interact with WebView?
|
||||
Tests are written as JUnit4 tests. [AwActivityTestRule] is used to create and
|
||||
@ -49,7 +49,7 @@ Java code.
|
||||
The tests are located in the [junit directory](/android_webview/junit/src/org/chromium/android_webview/robolectric/).
|
||||
|
||||
#### How to write junit tests?
|
||||
See the [JUnit doc](/testing/android/docs/junit.md#junit-tests-with-robolectric).
|
||||
See the [JUnit doc](/docs/testing/android_robolectric_tests.md).
|
||||
|
||||
## Native unittests
|
||||
These are on-device gtests that only exercise android\_webview native code.
|
||||
@ -59,7 +59,7 @@ The tests are located alongside respective source code files under
|
||||
[android\_webview directory](/android_webview/).
|
||||
|
||||
#### How to write gtests?
|
||||
See the [GTest doc](/testing/android/docs/gtest_implementation.md).
|
||||
See the [GTest doc](/docs/testing/android_gtests.md).
|
||||
|
||||
## How to run tests?
|
||||
Running tests is covered in [WebView Test Instructions](/android_webview/docs/test-instructions.md).
|
||||
|
@ -352,7 +352,7 @@ for more on debugging, including how to debug Java code.
|
||||
### Testing
|
||||
|
||||
For information on running tests, see
|
||||
[Android Test Instructions](/testing/android/docs/README.md)
|
||||
[Android Test Instructions](/docs/testing/android_test_instructions.md)
|
||||
|
||||
### Faster Edit/Deploy
|
||||
|
||||
|
@ -44,7 +44,7 @@ After writing a test, you can run it by:
|
||||
Typically, the "relevant gn target" is simply the target containing the
|
||||
other files in the same directory.
|
||||
- Rebuild.
|
||||
- Run the test using the process described [here](/testing/android/docs/todo.md).
|
||||
- Run the test
|
||||
|
||||
## Instrumentation test features
|
||||
|
||||
@ -278,15 +278,10 @@ test cases according to the feature they're testing.
|
||||
|
||||
@Feature doesn't have an inherent function, but it can be used to filter tests
|
||||
via the test runner's
|
||||
[-A/--annotation](/testing/android/docs/todo.md) and
|
||||
[-E/--exclude-annotation](/testing/android/docs/todo.md) flags. For example,
|
||||
this would run only the tests with @Feature annotations containing at least
|
||||
"Sync" in `chrome_public_test_apk`:
|
||||
`-A/--annotation` and `-E/--exclude-annotation` flags. For example, this would
|
||||
run only the tests with @Feature annotations containing at least "Sync" in
|
||||
`chrome_public_test_apk`:
|
||||
|
||||
```bash
|
||||
out/Debug/bin/run_chrome_public_test_apk -A Feature=Sync
|
||||
```
|
||||
|
||||
## Advanced
|
||||
|
||||
- [Creating a component for use in instrumentation tests.](/testing/android/docs/components_for_testing.md)
|
@ -1,3 +1,339 @@
|
||||
# Android Test Instructions
|
||||
# Android Testing in Chromium
|
||||
|
||||
See [Android testing in Chromium](/testing/android/docs/README.md)
|
||||
[TOC]
|
||||
|
||||
## Test Types
|
||||
|
||||
- **[gtests]**: For writing unit tests in C++. Most tests are cross-platform.
|
||||
- **Browser Tests**: Built on top of gtest. Used to write integration tests.
|
||||
- **[Robolectric]**: JUnit tests that run on the host machine by emulating (or
|
||||
mocking) Android APIs.
|
||||
- **[Instrumentation Tests]**: JUnit tests that run on devices / emulators.
|
||||
- **Unit Instrumentation Tests**: Unit tests that do not require
|
||||
initializing the Content layer. These use [BaseActivityTestRule] or
|
||||
[BlankUiTestActivityTestCase] based on `BaseActivityTestRule`.
|
||||
- **Integration Instrumentation Tests**: Instrumentation tests that require
|
||||
initializing the Content layer to test a certain feature in the end-to-end
|
||||
flow. These typically use more specialized test rules such as
|
||||
[ContentShellActivityTestRule] or [ChromeActivityTestRule].
|
||||
|
||||
[gtests]: android_gtests.md
|
||||
[Robolectric]: android_robolectric_tests.md
|
||||
[Instrumentation Tests]: android_instrumentation_tests.md
|
||||
[BaseActivityTestRule]: https://source.chromium.org/chromium/chromium/src/+/main:base/test/android/javatests/src/org/chromium/base/test/BaseActivityTestRule.java
|
||||
[BlankUiTestActivityTestCase]: https://source.chromium.org/chromium/chromium/src/+/main:ui/android/javatests/src/org/chromium/ui/test/util/BlankUiTestActivityTestCase.java
|
||||
[ContentShellActivityTestRule]: https://source.chromium.org/chromium/chromium/src/+/main:content/shell/android/javatests/src/org/chromium/content_shell_apk/ContentShellActivityTestRule.java
|
||||
[ChromeActivityTestRule]: https://source.chromium.org/chromium/chromium/src/+/main:chrome/test/android/javatests/src/org/chromium/chrome/test/ChromeActivityTestRule.java
|
||||
|
||||
## Device Setup
|
||||
|
||||
### Physical Device Setup
|
||||
|
||||
#### Root Access
|
||||
|
||||
Running tests requires root access, which requires using a userdebug build on
|
||||
your device.
|
||||
|
||||
To use a userdebug build, see
|
||||
[Running Builds](https://source.android.com/setup/build/running.html). Googlers
|
||||
can refer to [this page](https://goto.google.com/flashdevice).
|
||||
|
||||
If you can't run "adb root", you will get an error when trying to install
|
||||
the test APKs like "adb: error: failed to copy" and
|
||||
"remote secure_mkdirs failed: Operation not permitted" (use "adb unroot" to
|
||||
return adb to normal).
|
||||
|
||||
#### ADB Debugging
|
||||
|
||||
The adb executable exists within the Android SDK:
|
||||
|
||||
```shell
|
||||
third_party/android_sdk/public/platform-tools/adb
|
||||
```
|
||||
|
||||
In order to allow the ADB to connect to the device, you must enable USB
|
||||
debugging:
|
||||
|
||||
* Developer options are hidden by default. To unhide them:
|
||||
* Go to "About phone"
|
||||
* Tap 10 times on "Build number"
|
||||
* The "Developer options" menu will now be available.
|
||||
* Check "USB debugging".
|
||||
* Un-check "Verify apps over USB".
|
||||
|
||||
#### Screen
|
||||
|
||||
You **must** ensure that the screen stays on while testing: `adb shell svc power
|
||||
stayon usb` Or do this manually on the device: Settings -> Developer options ->
|
||||
Stay Awake.
|
||||
|
||||
If this option is greyed out, stay awake is probably disabled by policy. In that
|
||||
case, get another device or log in with a normal, unmanaged account (because the
|
||||
tests will break in exciting ways if stay awake is off).
|
||||
|
||||
#### Disable Verify Apps
|
||||
|
||||
You may see a dialog like [this
|
||||
one](http://www.samsungmobileusa.com/simulators/ATT_GalaxyMega/mobile/screens/06-02_12.jpg),
|
||||
which states, _Google may regularly check installed apps for potentially harmful
|
||||
behavior._ This can interfere with the test runner. To disable this dialog, run:
|
||||
|
||||
```shell
|
||||
adb shell settings put global package_verifier_enable 0
|
||||
```
|
||||
|
||||
### Using Emulators
|
||||
|
||||
Running tests on emulators is the same as [on device](#Running-Tests). Refer to
|
||||
[android_emulator.md](/docs/android_emulator.md) for setting up emulators.
|
||||
|
||||
## Building Tests
|
||||
|
||||
If you're adding a new test file, you'll need to explicitly add it to a gn
|
||||
target. If you're adding a test to an existing file, you won't need to make gn
|
||||
changes, but you may be interested in where your test winds up. In either case,
|
||||
here are some guidelines for where a test belongs:
|
||||
|
||||
### C++
|
||||
|
||||
C++ test files typically belong in `<top-level directory>_unittests` (e.g.
|
||||
`base_unittests` for `//base`). There are a few exceptions -- browser tests are
|
||||
typically their own target (e.g. `content_browsertests` for `//content`, or
|
||||
`browser_tests` for `//chrome`), and some unit test suites are broken at the
|
||||
second directory rather than the top-level one.
|
||||
|
||||
### Java
|
||||
|
||||
Java test files vary a bit more widely than their C++ counterparts:
|
||||
|
||||
- Instrumentation test files -- i.e., tests that will run on a device --
|
||||
typically belong in either `<top-level directory>_javatests` or `<top-level
|
||||
directory>_test_java`. Regardless, they'll wind up getting packaged into one
|
||||
of a few test APKs:
|
||||
- `webview_instrumentation_test_apk` for anything in `//android_webview`
|
||||
- `content_shell_test_apk` for anything in `//content` or below
|
||||
- `chrome_public_test_apk` for most things in `//chrome`
|
||||
- JUnit or Robolectric test files -- i.e., tests that will run on the host --
|
||||
typically belong in `<top-level directory>_junit_tests` (e.g.
|
||||
`base_junit_tests` for `//base`), though here again there are cases
|
||||
(particularly in `//components`) where suites are split at the second
|
||||
directory rather than the top-level one.
|
||||
|
||||
Once you know what to build, just do it like you normally would build anything
|
||||
else, e.g.: `ninja -C out/Release chrome_public_test_apk`
|
||||
|
||||
### Determining Test Target
|
||||
|
||||
If you do not know what target a test file belongs to, you can use
|
||||
`//tools/autotest.py` to figure it out fo you:
|
||||
|
||||
```sh
|
||||
# Builds relevant test target and then runs the test:
|
||||
tools/autotest.py -C <output directory> TestClassName
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
All functional tests should be runnable via the wrapper scripts generated at
|
||||
build time:
|
||||
|
||||
```sh
|
||||
<output directory>/bin/run_<target_name> [options]
|
||||
```
|
||||
|
||||
Note that tests are sharded across all attached devices unless explicitly told
|
||||
to do otherwise by `-d/--device`.
|
||||
|
||||
The commands used by the buildbots are printed in the logs. Look at
|
||||
https://build.chromium.org/ to duplicate the same test command as a particular
|
||||
builder.
|
||||
|
||||
### Listing Available Tests
|
||||
|
||||
Use `--list-tests` to list what tests are available.
|
||||
|
||||
```sh
|
||||
# Prints out all available tests:
|
||||
<output directory>/bin/run_<target_name> --list-tests
|
||||
|
||||
# Prints out all available tests that match a filter:
|
||||
<output directory>/bin/run_<target_name> --list-tests -f "*MyFilter*"
|
||||
```
|
||||
|
||||
### INSTALL\_FAILED\_CONTAINER\_ERROR or INSTALL\_FAILED\_INSUFFICIENT\_STORAGE
|
||||
|
||||
If you see this error when the test runner is attempting to deploy the test
|
||||
binaries to the AVD emulator, you may need to resize your userdata partition
|
||||
with the following commands:
|
||||
|
||||
```shell
|
||||
# Resize userdata partition to be 1G
|
||||
resize2fs android_emulator_sdk/sdk/system-images/android-25/x86/userdata.img 1G
|
||||
|
||||
# Set filesystem parameter to continue on errors; Android doesn't like some
|
||||
# things e2fsprogs does.
|
||||
tune2fs -e continue android_emulator_sdk/sdk/system-images/android-25/x86/userdata.img
|
||||
```
|
||||
|
||||
### AdbCommandFailedError: failed to stat remote object
|
||||
|
||||
There's a known issue (https://crbug.com/1094062) where the unit test binaries can fail on
|
||||
Android R and later: if you see this error, try rerunning on an Android version
|
||||
with API level <= 29 (Android <= Q).
|
||||
|
||||
## Symbolizing Crashes
|
||||
|
||||
Crash stacks are logged and can be viewed using `adb logcat`. To symbolize the
|
||||
traces, define `CHROMIUM_OUTPUT_DIR=$OUTDIR` where `$OUTDIR` is the argument you
|
||||
pass to `ninja -C`, and pipe the output through
|
||||
`third_party/android_platform/development/scripts/stack`. If
|
||||
`$CHROMIUM_OUTPUT_DIR` is unset, the script will search `out/Debug` and
|
||||
`out/Release`. For example:
|
||||
|
||||
```shell
|
||||
# If you build with
|
||||
ninja -C out/Debug chrome_public_test_apk
|
||||
# You can run:
|
||||
adb logcat -d | third_party/android_platform/development/scripts/stack
|
||||
|
||||
# If you build with
|
||||
ninja -C out/android chrome_public_test_apk
|
||||
# You can run:
|
||||
adb logcat -d | CHROMIUM_OUTPUT_DIR=out/android third_party/android_platform/development/scripts/stack
|
||||
# or
|
||||
export CHROMIUM_OUTPUT_DIR=out/android
|
||||
adb logcat -d | third_party/android_platform/development/scripts/stack
|
||||
```
|
||||
|
||||
## Robolectric Tests
|
||||
|
||||
JUnit tests are Java unittests running on the host instead of the target device.
|
||||
They are faster to run and therefore are recommended over instrumentation tests
|
||||
when possible.
|
||||
|
||||
The JUnits tests are usually following the pattern of *target*\_junit\_tests,
|
||||
for example, `content_junit_tests` and `chrome_junit_tests`.
|
||||
|
||||
When adding a new JUnit test, the associated `BUILD.gn` file must be updated.
|
||||
For example, adding a test to `chrome_junit_tests` requires to update
|
||||
`chrome/android/BUILD.gn`.
|
||||
|
||||
```shell
|
||||
# Build the test suite.
|
||||
ninja -C out/Default chrome_junit_tests
|
||||
|
||||
# Run the test suite.
|
||||
out/Default/bin/run_chrome_junit_tests
|
||||
|
||||
# Run a subset of tests. You might need to pass the package name for some tests.
|
||||
out/Default/bin/run_chrome_junit_tests -f "org.chromium.chrome.browser.media.*"
|
||||
```
|
||||
|
||||
### Debugging
|
||||
|
||||
Similar to [debugging apk targets](/docs/android_debugging_instructions.md#debugging-java):
|
||||
|
||||
```shell
|
||||
out/Default/bin/run_chrome_junit_tests --wait-for-java-debugger
|
||||
out/Default/bin/run_chrome_junit_tests --wait-for-java-debugger # Specify custom port via --debug-socket=9999
|
||||
```
|
||||
|
||||
## Gtests
|
||||
|
||||
```shell
|
||||
# Build a test suite
|
||||
ninja -C out/Release content_unittests
|
||||
|
||||
# Run a test suite
|
||||
out/Release/bin/run_content_unittests [-vv]
|
||||
|
||||
# Run a subset of tests and enable some "please go faster" options:
|
||||
out/Release/bin/run_content_unittests --fast-local-dev -f "ByteStreamTest.*"
|
||||
```
|
||||
|
||||
## Instrumentation Tests
|
||||
|
||||
In order to run instrumentation tests, you must leave your device screen ON and
|
||||
UNLOCKED. Otherwise, the test will timeout trying to launch an intent.
|
||||
Optionally you can disable screen lock under Settings -> Security -> Screen Lock
|
||||
-> None.
|
||||
|
||||
Next, you need to build the app, build your tests, and then run your tests
|
||||
(which will install the APK under test and the test APK automatically).
|
||||
|
||||
Examples:
|
||||
|
||||
ContentShell tests:
|
||||
|
||||
```shell
|
||||
# Build the tests:
|
||||
ninja -C out/Release content_shell_test_apk
|
||||
|
||||
# Run the test suite:
|
||||
out/Release/bin/run_content_shell_test_apk [-vv]
|
||||
|
||||
# Run a subset of tests and enable some "please go faster" options:
|
||||
out/Release/bin/run_content_shell_test_apk --fast-local-dev -f "*TestClass*"
|
||||
```
|
||||
|
||||
Android WebView tests:
|
||||
|
||||
See [WebView's instructions](/android_webview/docs/test-instructions.md).
|
||||
|
||||
In order to run a subset of tests, use -f to filter based on test class/method
|
||||
or -A/-E to filter using annotations.
|
||||
|
||||
More Filtering examples:
|
||||
|
||||
```shell
|
||||
# Run a specific test class
|
||||
out/Debug/bin/run_content_shell_test_apk -f "AddressDetectionTest.*"
|
||||
|
||||
# Run a specific test method
|
||||
out/Debug/bin/run_content_shell_test_apk -f AddressDetectionTest#testAddressLimits
|
||||
|
||||
# Run a subset of tests by size (Smoke, SmallTest, MediumTest, LargeTest,
|
||||
# EnormousTest)
|
||||
out/Debug/bin/run_content_shell_test_apk -A Smoke
|
||||
|
||||
# Run a subset of tests by annotation, such as filtering by Feature
|
||||
out/Debug/bin/run_content_shell_test_apk -A Feature=Navigation
|
||||
```
|
||||
|
||||
You might want to add stars `*` to each as a regular expression, e.g.
|
||||
`*`AddressDetectionTest`*`
|
||||
|
||||
### Debugging
|
||||
|
||||
Similar to [debugging apk targets](/docs/android_debugging_instructions.md#debugging-java):
|
||||
|
||||
```shell
|
||||
out/Debug/bin/run_content_shell_test_apk --wait-for-java-debugger
|
||||
```
|
||||
|
||||
### Deobfuscating Java Stacktraces
|
||||
|
||||
If running with `is_debug=false`, Java stacks from logcat need to be fixed up:
|
||||
|
||||
```shell
|
||||
build/android/stacktrace/java_deobfuscate.py out/Release/apks/ChromePublicTest.apk.mapping < stacktrace.txt
|
||||
```
|
||||
|
||||
Any stacks produced by test runner output will already be deobfuscated.
|
||||
|
||||
|
||||
## Running Blink Web Tests
|
||||
|
||||
See [Web Tests](web_tests.md).
|
||||
|
||||
## Running Telemetry (Perf) Tests
|
||||
|
||||
See [Telemetry](https://chromium.googlesource.com/catapult/+/HEAD/telemetry/README.md).
|
||||
|
||||
## Running GPU tests
|
||||
|
||||
(e.g. the "Android Debug (Nexus 7)" bot on the chromium.gpu waterfall)
|
||||
|
||||
See https://www.chromium.org/developers/testing/gpu-testing for details. Use
|
||||
`--browser=android-content-shell`. Examine the stdio from the test invocation on
|
||||
the bots to see arguments to pass to `src/content/test/gpu/run_gpu_test.py`.
|
||||
|
@ -15,9 +15,6 @@ for one platform.
|
||||
* **[gtest]** is Google's C++ test framework,
|
||||
which helps you write better C++ tests in Chromium.
|
||||
gtest is test framework for unit tests in Chromium and browser tests are built on top of it.
|
||||
* **[Junit]** is a unit testing framework
|
||||
for the Java programming language, and it is used to write
|
||||
unit tests on Android for Java code in Chromium.
|
||||
* **Browser Tests** is built on top of gtest, and it is used to write integration tests
|
||||
and e2e tests in Chromium.
|
||||
<!-- TODO(leilei) Add link to browser tests --->
|
||||
@ -27,8 +24,9 @@ for one platform.
|
||||
in a test renderer (`content_shell`) and comparing the rendered output or
|
||||
JavaScript output against an expected output file.
|
||||
Web Tests are required to launch new W3C API support in Chromium.
|
||||
* **[Instrumentation Tests]** is a test framework specific for Android platform,
|
||||
it is used to write integration tests or e2e tests for Chromium on Android.
|
||||
* **[Robolectric]** is build on top of JUnit 4. It emulates Android APIs so
|
||||
that tests can be run on the host machine instead of on devices / emulators.
|
||||
* **[Instrumentation Tests]** are JUnit tests that run on devices / emulators.
|
||||
* **[EarlGrey]** is the integration testing framework used by Chromium for iOS.
|
||||
* **[Telemetry]** is the performance testing framework used by Chromium.
|
||||
It allows you to perform arbitrary actions on a set of web pages and
|
||||
@ -42,10 +40,10 @@ The following table shows which types of test works on which platforms.
|
||||
| | Linux | Windows | Mac | Android | iOS | CrOS |
|
||||
|:----------------------------|:--------|:--------|:--------|:--------|:--------|:--------|
|
||||
| gtest(C++) | √ | √ | √ | √ | √ | √ |
|
||||
| Junit(Java) | | | | √ | | |
|
||||
| Browser Tests(C++) | √ | √ | √ | √ | | |
|
||||
| Web Tests(HTML, JS) | √ | √ | √ | | | |
|
||||
| Telemetry(Python) | √ | √ | √ | √ | | √ |
|
||||
| Robolectric(Java) | | | | √ | | |
|
||||
| Instrumentation Tests(Java) | | | | √ | | |
|
||||
| EarlGrey | | | | | √ | |
|
||||
| Fuzzer Tests(C++) | √ | √ | √ | √ | | √ |
|
||||
@ -184,8 +182,8 @@ Tests are not configured to upload metrics, such as UMA, UKM or crash reports.
|
||||
|
||||
[gtest]: https://github.com/google/googletest
|
||||
[Simple gtests]: https://github.com/google/googletest/blob/main/docs/primer.md#simple-tests
|
||||
[Junit]: https://developer.android.com/training/testing/junit-rules
|
||||
[Instrumentation Tests]: https://chromium.googlesource.com/chromium/src/+/main/testing/android/docs/instrumentation.md
|
||||
[Robolectric]: android_robolectric_tests.md
|
||||
[Instrumentation Tests]: https://chromium.googlesource.com/chromium/src/+/main/docs/testing/android_instrumentation_tests.md
|
||||
[EarlGrey]: https://github.com/google/EarlGrey
|
||||
[Telemetry]: https://chromium.googlesource.com/catapult/+/HEAD/telemetry/README.md
|
||||
[Fuzzer Tests]: https://chromium.googlesource.com/chromium/src/+/main/testing/libfuzzer/README.md
|
||||
|
@ -1 +0,0 @@
|
||||
*
|
@ -1,35 +0,0 @@
|
||||
# Android Testing in Chromium
|
||||
|
||||
## Concepts
|
||||
- **Native Unit Tests**: Normal Chromium unit tests based on the [gtest framework](/testing/android/docs/gtest_implementation.md). Tests for native code.
|
||||
- **Java Unit Tests**: JUnit tests that run on the host machine using [Robolectric](http://robolectric.org) to emulate Android APIs.
|
||||
- **Instrumentation Tests**: JUnit tests that run on Android devices (or emulators).
|
||||
- **Unit Instrumentation Tests**: Instrumentation tests that test an individual feature. They do not require starting up ContentShell (or Chrome browser). These use [BaseActivityTestRule](https://source.chromium.org/chromium/chromium/src/+/main:base/test/android/javatests/src/org/chromium/base/test/BaseActivityTestRule.java) or [BlankUiTestActivityTestCase](https://source.chromium.org/chromium/chromium/src/+/main:ui/android/javatests/src/org/chromium/ui/test/util/BlankUiTestActivityTestCase.java) based on `BaseActivityTestRule`.
|
||||
- **Integration Instrumentation Tests**: Instrumentation tests that bring up ContentShell (or Chrome browser) to test a certain feature in the end-to-end flow. These typically use more specialized test rules such as [ContentShellActivityTestRule](https://source.chromium.org/chromium/chromium/src/+/main:content/shell/android/javatests/src/org/chromium/content_shell_apk/ContentShellActivityTestRule.java) or [ChromeActivityTestRule](https://source.chromium.org/chromium/chromium/src/+/main:chrome/test/android/javatests/src/org/chromium/chrome/test/ChromeActivityTestRule.java).
|
||||
|
||||
## How do I...
|
||||
|
||||
- **set up**
|
||||
- [... set up a device for local testing?](/testing/android/docs/android_test_instructions.md#physical-device-setup)
|
||||
- [... set up an emulator for local testing?](/docs/android_emulator.md)
|
||||
- **writing tests**
|
||||
- [... write an instrumentation test?](/testing/android/docs/instrumentation.md)
|
||||
- [... write a JUnit or Robolectric test?](/testing/android/docs/junit.md)
|
||||
- [... write a test that needs to mock native calls?](/third_party/jni_zero/README.md#testing-mockable-natives)
|
||||
- **running tests**
|
||||
- [... run a gtest?](/testing/android/docs/android_test_instructions.md#gtests)
|
||||
- [... run an instrumentation test?](/testing/android/docs/android_test_instructions.md#instrumentation-tests)
|
||||
- [... run a JUnit test?](/testing/android/docs/android_test_instructions.md#junit-tests)
|
||||
- [... run the Web tests?](/docs/testing/web_tests.md)
|
||||
- [... run a telemetry test?](https://chromium.googlesource.com/catapult/+/HEAD/telemetry/README.md)
|
||||
- **debugging tests**
|
||||
- [... debug junit tests?](/testing/android/docs/android_test_instructions.md#junit-tests-debugging)
|
||||
- [... debug flaky tests?](/testing/android/docs/todo.md)
|
||||
- **miscellaneous**
|
||||
- [... use code coverage for Java tests?](/build/android/docs/coverage.md)
|
||||
|
||||
## How does it work on Android?
|
||||
|
||||
- [gtests](/testing/android/docs/gtest_implementation.md)
|
||||
- [instrumentation tests](https://source.android.com/compatibility/tests/development/instrumentation)
|
||||
- [junit tests](/testing/android/docs/junit.md)
|
@ -1,312 +0,0 @@
|
||||
# Building and Running Android Tests
|
||||
|
||||
[TOC]
|
||||
|
||||
## Device Setup
|
||||
|
||||
### Physical Device Setup
|
||||
|
||||
#### Root Access
|
||||
|
||||
Running tests requires root access, which requires using a userdebug build on
|
||||
your device.
|
||||
|
||||
To use a userdebug build, see
|
||||
[Running Builds](https://source.android.com/setup/build/running.html). Googlers
|
||||
can refer to [this page](https://goto.google.com/flashdevice).
|
||||
|
||||
If you can't run "adb root", you will get an error when trying to install
|
||||
the test APKs like "adb: error: failed to copy" and
|
||||
"remote secure_mkdirs failed: Operation not permitted" (use "adb unroot" to
|
||||
return adb to normal).
|
||||
|
||||
#### ADB Debugging
|
||||
|
||||
The adb executable exists within the Android SDK:
|
||||
|
||||
```shell
|
||||
third_party/android_sdk/public/platform-tools/adb
|
||||
```
|
||||
|
||||
In order to allow the ADB to connect to the device, you must enable USB
|
||||
debugging:
|
||||
|
||||
* Developer options are hidden by default. To unhide them:
|
||||
* Go to "About phone"
|
||||
* Tap 10 times on "Build number"
|
||||
* The "Developer options" menu will now be available.
|
||||
* Check "USB debugging".
|
||||
* Un-check "Verify apps over USB".
|
||||
|
||||
#### Screen
|
||||
|
||||
You **must** ensure that the screen stays on while testing: `adb shell svc power
|
||||
stayon usb` Or do this manually on the device: Settings -> Developer options ->
|
||||
Stay Awake.
|
||||
|
||||
If this option is greyed out, stay awake is probably disabled by policy. In that
|
||||
case, get another device or log in with a normal, unmanaged account (because the
|
||||
tests will break in exciting ways if stay awake is off).
|
||||
|
||||
#### Disable Verify Apps
|
||||
|
||||
You may see a dialog like [this
|
||||
one](http://www.samsungmobileusa.com/simulators/ATT_GalaxyMega/mobile/screens/06-02_12.jpg),
|
||||
which states, _Google may regularly check installed apps for potentially harmful
|
||||
behavior._ This can interfere with the test runner. To disable this dialog, run:
|
||||
|
||||
```shell
|
||||
adb shell settings put global package_verifier_enable 0
|
||||
```
|
||||
|
||||
### Using Emulators
|
||||
|
||||
Running tests on emulators is the same as [on device](#Running-Tests). Refer to
|
||||
[android_emulator.md](/docs/android_emulator.md) for setting up emulators.
|
||||
|
||||
## Building Tests
|
||||
|
||||
If you're adding a new test file, you'll need to explicitly add it to a gn
|
||||
target. If you're adding a test to an existing file, you won't need to make gn
|
||||
changes, but you may be interested in where your test winds up. In either case,
|
||||
here are some guidelines for where a test belongs:
|
||||
|
||||
### C++
|
||||
|
||||
C++ test files typically belong in `<top-level directory>_unittests` (e.g.
|
||||
`base_unittests` for `//base`). There are a few exceptions -- browser tests are
|
||||
typically their own target (e.g. `content_browsertests` for `//content`, or
|
||||
`browser_tests` for `//chrome`), and some unit test suites are broken at the
|
||||
second directory rather than the top-level one.
|
||||
|
||||
### Java
|
||||
|
||||
Java test files vary a bit more widely than their C++ counterparts:
|
||||
|
||||
- Instrumentation test files -- i.e., tests that will run on a device --
|
||||
typically belong in either `<top-level directory>_javatests` or `<top-level
|
||||
directory>_test_java`. Regardless, they'll wind up getting packaged into one
|
||||
of a few test APKs:
|
||||
- `webview_instrumentation_test_apk` for anything in `//android_webview`
|
||||
- `content_shell_test_apk` for anything in `//content` or below
|
||||
- `chrome_public_test_apk` for most things in `//chrome`
|
||||
- JUnit or Robolectric test files -- i.e., tests that will run on the host --
|
||||
typically belong in `<top-level directory>_junit_tests` (e.g.
|
||||
`base_junit_tests` for `//base`), though here again there are cases
|
||||
(particularly in `//components`) where suites are split at the second
|
||||
directory rather than the top-level one.
|
||||
|
||||
Once you know what to build, just do it like you normally would build anything
|
||||
else, e.g.: `ninja -C out/Release chrome_public_test_apk`
|
||||
|
||||
### Determining Test Target
|
||||
|
||||
If you do not know what target a test file belongs to, you can use
|
||||
`//tools/autotest.py` to figure it out fo you:
|
||||
|
||||
```sh
|
||||
# Builds relevant test target and then runs the test:
|
||||
tools/autotest.py -C <output directory> TestClassName
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
All functional tests should be runnable via the wrapper scripts generated at
|
||||
build time:
|
||||
|
||||
```sh
|
||||
<output directory>/bin/run_<target_name> [options]
|
||||
```
|
||||
|
||||
Note that tests are sharded across all attached devices unless explicitly told
|
||||
to do otherwise by `-d/--device`.
|
||||
|
||||
The commands used by the buildbots are printed in the logs. Look at
|
||||
https://build.chromium.org/ to duplicate the same test command as a particular
|
||||
builder.
|
||||
|
||||
### Listing Available Tests
|
||||
|
||||
Use `--list-tests` to list what tests are available.
|
||||
|
||||
```sh
|
||||
# Prints out all available tests:
|
||||
<output directory>/bin/run_<target_name> --list-tests
|
||||
|
||||
# Prints out all available tests that match a filter:
|
||||
<output directory>/bin/run_<target_name> --list-tests -f "*MyFilter*"
|
||||
```
|
||||
|
||||
### INSTALL\_FAILED\_CONTAINER\_ERROR or INSTALL\_FAILED\_INSUFFICIENT\_STORAGE
|
||||
|
||||
If you see this error when the test runner is attempting to deploy the test
|
||||
binaries to the AVD emulator, you may need to resize your userdata partition
|
||||
with the following commands:
|
||||
|
||||
```shell
|
||||
# Resize userdata partition to be 1G
|
||||
resize2fs android_emulator_sdk/sdk/system-images/android-25/x86/userdata.img 1G
|
||||
|
||||
# Set filesystem parameter to continue on errors; Android doesn't like some
|
||||
# things e2fsprogs does.
|
||||
tune2fs -e continue android_emulator_sdk/sdk/system-images/android-25/x86/userdata.img
|
||||
```
|
||||
|
||||
### AdbCommandFailedError: failed to stat remote object
|
||||
|
||||
There's a known issue (https://crbug.com/1094062) where the unit test binaries can fail on
|
||||
Android R and later: if you see this error, try rerunning on an Android version
|
||||
with API level <= 29 (Android <= Q).
|
||||
|
||||
## Symbolizing Crashes
|
||||
|
||||
Crash stacks are logged and can be viewed using `adb logcat`. To symbolize the
|
||||
traces, define `CHROMIUM_OUTPUT_DIR=$OUTDIR` where `$OUTDIR` is the argument you
|
||||
pass to `ninja -C`, and pipe the output through
|
||||
`third_party/android_platform/development/scripts/stack`. If
|
||||
`$CHROMIUM_OUTPUT_DIR` is unset, the script will search `out/Debug` and
|
||||
`out/Release`. For example:
|
||||
|
||||
```shell
|
||||
# If you build with
|
||||
ninja -C out/Debug chrome_public_test_apk
|
||||
# You can run:
|
||||
adb logcat -d | third_party/android_platform/development/scripts/stack
|
||||
|
||||
# If you build with
|
||||
ninja -C out/android chrome_public_test_apk
|
||||
# You can run:
|
||||
adb logcat -d | CHROMIUM_OUTPUT_DIR=out/android third_party/android_platform/development/scripts/stack
|
||||
# or
|
||||
export CHROMIUM_OUTPUT_DIR=out/android
|
||||
adb logcat -d | third_party/android_platform/development/scripts/stack
|
||||
```
|
||||
|
||||
## JUnit tests
|
||||
|
||||
JUnit tests are Java unittests running on the host instead of the target device.
|
||||
They are faster to run and therefore are recommended over instrumentation tests
|
||||
when possible.
|
||||
|
||||
The JUnits tests are usually following the pattern of *target*\_junit\_tests,
|
||||
for example, `content_junit_tests` and `chrome_junit_tests`.
|
||||
|
||||
When adding a new JUnit test, the associated `BUILD.gn` file must be updated.
|
||||
For example, adding a test to `chrome_junit_tests` requires to update
|
||||
`chrome/android/BUILD.gn`.
|
||||
|
||||
```shell
|
||||
# Build the test suite.
|
||||
ninja -C out/Default chrome_junit_tests
|
||||
|
||||
# Run the test suite.
|
||||
out/Default/bin/run_chrome_junit_tests
|
||||
|
||||
# Run a subset of tests. You might need to pass the package name for some tests.
|
||||
out/Default/bin/run_chrome_junit_tests -f "org.chromium.chrome.browser.media.*"
|
||||
```
|
||||
|
||||
### Debugging
|
||||
|
||||
Similar to [debugging apk targets](/docs/android_debugging_instructions.md#debugging-java):
|
||||
|
||||
```shell
|
||||
out/Default/bin/run_chrome_junit_tests --wait-for-java-debugger
|
||||
out/Default/bin/run_chrome_junit_tests --wait-for-java-debugger # Specify custom port via --debug-socket=9999
|
||||
```
|
||||
|
||||
## Gtests
|
||||
|
||||
```shell
|
||||
# Build a test suite
|
||||
ninja -C out/Release content_unittests
|
||||
|
||||
# Run a test suite
|
||||
out/Release/bin/run_content_unittests [-vv]
|
||||
|
||||
# Run a subset of tests and enable some "please go faster" options:
|
||||
out/Release/bin/run_content_unittests --fast-local-dev -f "ByteStreamTest.*"
|
||||
```
|
||||
|
||||
## Instrumentation Tests
|
||||
|
||||
In order to run instrumentation tests, you must leave your device screen ON and
|
||||
UNLOCKED. Otherwise, the test will timeout trying to launch an intent.
|
||||
Optionally you can disable screen lock under Settings -> Security -> Screen Lock
|
||||
-> None.
|
||||
|
||||
Next, you need to build the app, build your tests, and then run your tests
|
||||
(which will install the APK under test and the test APK automatically).
|
||||
|
||||
Examples:
|
||||
|
||||
ContentShell tests:
|
||||
|
||||
```shell
|
||||
# Build the tests:
|
||||
ninja -C out/Release content_shell_test_apk
|
||||
|
||||
# Run the test suite:
|
||||
out/Release/bin/run_content_shell_test_apk [-vv]
|
||||
|
||||
# Run a subset of tests and enable some "please go faster" options:
|
||||
out/Release/bin/run_content_shell_test_apk --fast-local-dev -f "*TestClass*"
|
||||
```
|
||||
|
||||
Android WebView tests:
|
||||
|
||||
See [WebView's instructions](/android_webview/docs/test-instructions.md).
|
||||
|
||||
In order to run a subset of tests, use -f to filter based on test class/method
|
||||
or -A/-E to filter using annotations.
|
||||
|
||||
More Filtering examples:
|
||||
|
||||
```shell
|
||||
# Run a specific test class
|
||||
out/Debug/bin/run_content_shell_test_apk -f "AddressDetectionTest.*"
|
||||
|
||||
# Run a specific test method
|
||||
out/Debug/bin/run_content_shell_test_apk -f AddressDetectionTest#testAddressLimits
|
||||
|
||||
# Run a subset of tests by size (Smoke, SmallTest, MediumTest, LargeTest,
|
||||
# EnormousTest)
|
||||
out/Debug/bin/run_content_shell_test_apk -A Smoke
|
||||
|
||||
# Run a subset of tests by annotation, such as filtering by Feature
|
||||
out/Debug/bin/run_content_shell_test_apk -A Feature=Navigation
|
||||
```
|
||||
|
||||
You might want to add stars `*` to each as a regular expression, e.g.
|
||||
`*`AddressDetectionTest`*`
|
||||
|
||||
### Debugging
|
||||
|
||||
Similar to [debugging apk targets](/docs/android_debugging_instructions.md#debugging-java):
|
||||
|
||||
```shell
|
||||
out/Debug/bin/run_content_shell_test_apk --wait-for-java-debugger
|
||||
```
|
||||
|
||||
### Deobfuscating Java Stacktraces
|
||||
|
||||
If running with `is_debug=false`, Java stacks from logcat need to be fixed up:
|
||||
|
||||
```shell
|
||||
build/android/stacktrace/java_deobfuscate.py out/Release/apks/ChromePublicTest.apk.mapping < stacktrace.txt
|
||||
```
|
||||
|
||||
Any stacks produced by test runner output will already be deobfuscated.
|
||||
|
||||
|
||||
## Running Blink Web Tests
|
||||
|
||||
See [Web Tests](web_tests.md).
|
||||
|
||||
## Running GPU tests
|
||||
|
||||
(e.g. the "Android Debug (Nexus 7)" bot on the chromium.gpu waterfall)
|
||||
|
||||
See https://www.chromium.org/developers/testing/gpu-testing for details. Use
|
||||
`--browser=android-content-shell`. Examine the stdio from the test invocation on
|
||||
the bots to see arguments to pass to `src/content/test/gpu/run_gpu_test.py`.
|
@ -1,162 +0,0 @@
|
||||
[< Instrumentation](/testing/android/docs/instrumentation.md)
|
||||
|
||||
# Creating a component for testing
|
||||
|
||||
Creating a component, e.g. a [ContentProvider][1] or [Service][2], for use
|
||||
in testing can be tricky. This is particularly true when:
|
||||
- app code and test code are in separate APKs
|
||||
- the test APK instruments the app APK
|
||||
- the test Service depends on app code.
|
||||
|
||||
This doc explains the pitfalls in creating a test component and how to
|
||||
avoid them.
|
||||
|
||||
[TOC]
|
||||
|
||||
## What can go wrong with a test component
|
||||
|
||||
> **tl;dr:** test components may not be able to access app code when defined in
|
||||
> the test APK.
|
||||
|
||||
Chromium's instrumentation test suites are all currently set up in the manner
|
||||
described above: app code is in one APK (the _APK under test_), test code is
|
||||
in another APK (the _test APK_), and auxiliary code, when necessary, is in
|
||||
one or more other APKs (_support APKs_). Test APKs build against app code
|
||||
but do not retain it in their .dex files. This reduces the size of test APKs
|
||||
and avoids potentially conflicting definitions. At instrumentation runtime,
|
||||
the test code is loaded into the app package's process, and it's consequently
|
||||
able to access code defined in the app APK's .dex file(s). Test components,
|
||||
however, run in the test package's process and only have access to code in the
|
||||
test APK. While test components referencing app code will build without issue,
|
||||
they will fail to link at runtime and will consequently not be able to be
|
||||
instantiated.
|
||||
|
||||
For example, here's the logcat from an attempt to use a test Service,
|
||||
`TestPostMessageService`, that extended an app Service, `PostMessageService`.
|
||||
Note that the runtime link failed because the superclass couldn't be resolved,
|
||||
and the test Service could not be instantiated as a result.
|
||||
|
||||
``` text
|
||||
Unable to resolve superclass of Lorg/chromium/chrome/browser/customtabs/TestPostMessageService; (184)
|
||||
Link of class 'Lorg/chromium/chrome/browser/customtabs/TestPostMessageService;' failed
|
||||
...
|
||||
FATAL EXCEPTION: main
|
||||
Process: org.chromium.chrome.tests, PID: 30023
|
||||
java.lang.RuntimeException:
|
||||
Unable to instantiate service org.chromium.chrome.browser.customtabs.TestPostMessageService:
|
||||
java.lang.ClassNotFoundException:
|
||||
Didn't find class "org.chromium.chrome.browser.customtabs.TestPostMessageService" on path:
|
||||
DexPathList[
|
||||
[zip file "/system/framework/android.test.runner.jar",
|
||||
zip file "/data/app/org.chromium.chrome.tests-1.apk"],
|
||||
nativeLibraryDirectories=[
|
||||
/data/app-lib/org.chromium.chrome.tests-1,
|
||||
/vendor/lib,
|
||||
/system/lib]]
|
||||
at android.app.ActivityThread.handleCreateService(ActivityThread.java:2543)
|
||||
at android.app.ActivityThread.access$1800(ActivityThread.java:135)
|
||||
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1278)
|
||||
at android.os.Handler.dispatchMessage(Handler.java:102)
|
||||
at android.os.Looper.loop(Looper.java:136)
|
||||
at android.app.ActivityThread.main(ActivityThread.java:5001)
|
||||
at java.lang.reflect.Method.invokeNative(Native Method)
|
||||
at java.lang.reflect.Method.invoke(Method.java:515)
|
||||
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:785)
|
||||
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:601)
|
||||
at dalvik.system.NativeStart.main(Native Method)
|
||||
Caused by: java.lang.ClassNotFoundException:
|
||||
Didn't find class "org.chromium.chrome.browser.customtabs.TestPostMessageService" on path:
|
||||
DexPathList[
|
||||
[zip file "/system/framework/android.test.runner.jar",
|
||||
zip file "/data/app/org.chromium.chrome.tests-1.apk"],
|
||||
nativeLibraryDirectories=[
|
||||
/data/app-lib/org.chromium.chrome.tests-1,
|
||||
/vendor/lib,
|
||||
/system/lib]]
|
||||
at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56)
|
||||
at java.lang.ClassLoader.loadClass(ClassLoader.java:497)
|
||||
at java.lang.ClassLoader.loadClass(ClassLoader.java:457)
|
||||
at android.app.ActivityThread.handleCreateService(ActivityThread.java:2540)
|
||||
... 10 more
|
||||
```
|
||||
|
||||
## How to implement a test component
|
||||
|
||||
There are (at least) two mechanisms for avoiding the failures described above:
|
||||
using a support APK or using sharedUserIds.
|
||||
|
||||
### Use a support APK
|
||||
|
||||
Putting the Service in a support APK lets the build system include all necessary
|
||||
code in the .dex without fear of conflicting definitions, as nothing in the
|
||||
support APK runs in the same package or process as the test or app code.
|
||||
|
||||
To do this:
|
||||
|
||||
**Create the component.**
|
||||
|
||||
It should either be in an existing directory used by code in the appropriate
|
||||
support APK or a new directory for such purpose. In particular, it should be
|
||||
in neither a directory containing app code nor a directory containing test
|
||||
code.
|
||||
|
||||
``` java
|
||||
package org.chromium.chrome.test;
|
||||
|
||||
import org.chromium.chrome.MyAppService;
|
||||
|
||||
public class MyTestService extends MyAppService {
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
**Put the component in a separate gn target.**
|
||||
|
||||
This can either be a target upon which the support APK already depends or a new
|
||||
target.
|
||||
|
||||
``` python
|
||||
android_library("my_test_service") {
|
||||
sources = [ "src/org/chromium/chrome/test/MyTestService.java" ]
|
||||
deps = [ ... ]
|
||||
}
|
||||
```
|
||||
|
||||
The support APK must depend on this target. The target containing your test
|
||||
code can also depend on this target if you want to refer to the component
|
||||
class directly, e.g., when binding a Service.
|
||||
|
||||
> **NOTE:** Even if your test code directly depends on the service target,
|
||||
> you won't be able to directly reference the test component instance from
|
||||
> test code. Checking a test component's internal state will require adding
|
||||
> APIs specifically for that purpose.
|
||||
|
||||
**Define the component in the support APK manifest.**
|
||||
|
||||
``` xml
|
||||
<manifest ...
|
||||
package="org.chromium.chrome.tests.support" >
|
||||
|
||||
<application>
|
||||
<service android:name="org.chromium.chrome.test.MyTestService"
|
||||
android:exported="true"
|
||||
tools:ignore="ExportedService">
|
||||
...
|
||||
</service>
|
||||
|
||||
...
|
||||
</application>
|
||||
</manifest>
|
||||
```
|
||||
|
||||
### Use a sharedUserId
|
||||
|
||||
Using a [sharedUserId][3] will allow your component to run in your app's
|
||||
process.
|
||||
|
||||
> Because this requires modifying the app manifest, it is not recommended at
|
||||
> this time and is intentionally not further documented here.
|
||||
|
||||
[1]: https://developer.android.com/reference/android/content/ContentProvider.html
|
||||
[2]: https://developer.android.com/reference/android/app/Service.html
|
||||
[3]: https://developer.android.com/guide/topics/manifest/manifest-element.html#uid
|
@ -1,287 +0,0 @@
|
||||
# JUnit 4 Migration
|
||||
|
||||
As of Android 24 (N), JUnit3 style javatests have been deprecated for the new
|
||||
JUnit4-based [Android Testing Support Library][1].
|
||||
We are in the process of changing all instrumentation tests in chromium to
|
||||
JUnit4 style. This doc explains the differences between JUnit3 and JUnit4
|
||||
instrumentation tests and how to write or convert them.
|
||||
|
||||
[TOC]
|
||||
|
||||
## Differences between JUnit3 and JUnit4 instrumentation tests
|
||||
|
||||
| | JUnit3 | JUnit4 |
|
||||
|--------------|-----------------------------------------------------|------------------------------------------|
|
||||
| Inheritance | Tests extends TestCase or child classes | No inheritance. |
|
||||
| Test methods | methods named with test prefix | methods annotated with @Test |
|
||||
| Set up | setUp() method | public method annotated with @Before |
|
||||
| Tear down | tearDown() method | public method annotated with @After |
|
||||
| Test runner | declared within test apk AndroidManifest.xml | Must specify `chromium-junit4:"true"` |
|
||||
| Class runner | N/A | @RunWith(XClassRunner.class) |
|
||||
| Assertion | Extends from junit.framework.Assert, inherited APIs | Use static methods from org.junit.Assert |
|
||||
|
||||
> Please note that during the migration, we support running JUnit3 and JUnit4
|
||||
> tests in the same apk. This requires two tags, one each for JUnit3 and JUnit4.
|
||||
> The tag for the JUnit4 runner must specify `chromium-junit4:"true"`
|
||||
> ([Example][2])
|
||||
|
||||
- **Other JUnit4 features**:
|
||||
- Tests can be annotated to expect an exception, e.g.
|
||||
`@Test(expected=MyException.class)`. Tests annotated this way will
|
||||
fail if they do not throw the given exception.
|
||||
- **Test suite set up**: public static method annotated with `@BeforeClass`
|
||||
- **Test suite tear down**: public static method annotated with
|
||||
`@AfterClass`
|
||||
- **Replacement for JUnit3 test base classes**
|
||||
- [`TestRule`][3]:
|
||||
- TestRule is a class to **outsource your test setUp, tearDown, and
|
||||
utility methods**. Since there are no more interitance and TestBase classes,
|
||||
one should use TestRule for any API calls provided by its test base classes
|
||||
previously.
|
||||
- One test can declare multiple TestRules and the class runner will run all of
|
||||
them. If the order of the TestRule matters to you, use
|
||||
[`RuleChain`][8]
|
||||
|
||||
- [`ActivityTestRule`][4]
|
||||
- `ActivityTestRule` is a special `TestRule` provided by Android Testing
|
||||
Support Library that allows tests to launch an Activity.
|
||||
([Documentation][4])
|
||||
|
||||
## Example Code of JUnit3 test and JUnit4 test
|
||||
|
||||
JUnit3:
|
||||
|
||||
```java
|
||||
public class MyTestClass extends MyActivityInstrumentationTestCase2<TestActivity> {
|
||||
@Override
|
||||
protected void setUp(){
|
||||
super.setUp();
|
||||
setActivityIntent(new Intent());
|
||||
getActivity();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void tearDown() {
|
||||
specialActionFromSuper();
|
||||
super.tearDown();
|
||||
}
|
||||
|
||||
public void testA() {
|
||||
assertEquals(1, 1);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
JUnit4:
|
||||
|
||||
```java
|
||||
@RunWith(BaseJUnit4ClassRunner.class);
|
||||
public class TestClass {
|
||||
@Rule public ActivityTestRule<TestActivity> mRule = new ActivityTestRule<>(TestActivity.class);
|
||||
|
||||
@Before
|
||||
public void setUp() { //Must be public
|
||||
mRule.launchActivity(new Intent());
|
||||
}
|
||||
|
||||
@After
|
||||
public void tearDown() { //Must be public
|
||||
mRule.specialActionFromActivityTestRule();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testA() {
|
||||
Assert.assertEquals(1, 1);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Migration process
|
||||
|
||||
1. Add required libraries to your target dependencies in BUILD.gn
|
||||
- JUnit 4 library: `//third_party/junit`
|
||||
- Android Testing Support Rules:
|
||||
- `//third_party/androidx:androidx_test_runner_java`
|
||||
(for `AndroidJUnitRunner`, etc)
|
||||
- `//third_party/androidx:androidx_test_rules_java`
|
||||
(for `ActivityTestRule`, etc)
|
||||
1. Add class runner to your test apk manifest.
|
||||
([example][2])
|
||||
- Keep in mind you can have multiple instrumentations in your manifest.
|
||||
Our test runner will run JUnit4 tests with JUnit4 runner and JUnit3
|
||||
tests with non-JUnit4 runner.
|
||||
1. Refactor TestBase class to a TestRule class.
|
||||
([example CL](https://codereview.chromium.org/2632043002))
|
||||
- +yolandyan will do this part, however, if you did refactoring yourself,
|
||||
please add him as a reviewer for your CL and enjoy his eternal appreciation!
|
||||
1. Use [auto migrate script][5] to or manually convert all JUnit3 tests to
|
||||
JUnit4 style in a your javatest directory
|
||||
- we understand it's tedious to just manually write all the annotations,
|
||||
change modifiers, etc to convert all the javatest, so we created an auto
|
||||
change script that helps you to convert all the javatests in a certain
|
||||
directory. Please check its [README page][5]
|
||||
on instructions.
|
||||
|
||||
|
||||
## Customized TestRule example
|
||||
|
||||
TestRule:
|
||||
|
||||
```java
|
||||
public class MyRule implements TestRule {
|
||||
// 1: Add utility methods...
|
||||
|
||||
@Override
|
||||
public Statement apply(final Statement base, Description desc) {
|
||||
return new Statement() {
|
||||
@Override
|
||||
public void evaluate() {
|
||||
// 2: Code here runs before @Before method
|
||||
base.evaluate()
|
||||
// 3: Code here runs after @After method
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Command Line Flags
|
||||
|
||||
In our Junit3 tests command line flags (set by the CommandLineFlag annotations) were inherited from the
|
||||
test base classes. As an example, ChromeActivityTestBase is annotated with:
|
||||
|
||||
```java
|
||||
@CommandLineFlags.Add({ChromeSwitches.DISABLE_FIRST_RUN_EXPERIENCE, ...
|
||||
```
|
||||
|
||||
and as a result any test in a class derived from ChromeActivityTestBase will disable the first run experience.
|
||||
|
||||
The Junit4 tests classes are not however, derived from test base classes; instead their behavior is defined by
|
||||
test rules. To support this our Junit4 test runner will examine the command line flag annotations on all rules
|
||||
referenced with @Rule annotations in the test class. In addition, where one rule is derived from another, the
|
||||
command line flags propogate through the hierarchy of rules. See, for example, [BottomSheetTestRule][11]
|
||||
|
||||
Note:- This has only recently been implemented, so is not yet used in all tests. See [this bug][12]
|
||||
|
||||
The CommandLineFlags annonations are more fully documented in the [CommandLineFlags class][13]
|
||||
|
||||
## Common Errors
|
||||
|
||||
1. Instrumentation tests that rely on test thread to have message handler
|
||||
will not work. For example error message:
|
||||
|
||||
java.lang.RuntimeException: Can't create handler inside thread that has not called Looper.prepare()
|
||||
|
||||
or
|
||||
|
||||
java.lang.IllegalStateException: The current thread must have a looper!
|
||||
|
||||
Please utilize `ActivityTestRule.runOnUiThread(Runnable r)` to refactor
|
||||
these tests. For more, check this [GitHub issue][6]
|
||||
|
||||
1. Use `@UiThreadTest` with caution!!
|
||||
- Currently, **@UiThreadTest is only effective when UiThreadTestRule or
|
||||
ActivityTestRule is declared** in the test class.
|
||||
- Please use **`androidx.test.annotation.UiThreadTest`, NOT
|
||||
`android.test.UiThreadTest`**.
|
||||
- When using @UiThreadTest, **it would cause `setUp` and `tearDown` to
|
||||
run in Ui Thread** as well. Avoid that by calling [`runOnUiThread`][9]
|
||||
or [`runOnMainSync`][10] with a Runnable.
|
||||
|
||||
```java
|
||||
// Wrong test
|
||||
public class Test {
|
||||
@Rule
|
||||
public ActivityTestRule<MyActivity> mRule = new ActivityTestRule<>(
|
||||
MyActivity.class);
|
||||
|
||||
@Before
|
||||
public void setUp() {
|
||||
// Cause failure because this also runs on Ui Thread, while it
|
||||
// is intended for Instrumentation worker thread
|
||||
mRule.launchActivity();
|
||||
}
|
||||
|
||||
@UiThreadTest
|
||||
public void test() {
|
||||
actionThatNeedsUiThread();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The correct thing to do is
|
||||
|
||||
```java
|
||||
// Correct test
|
||||
public class Test {
|
||||
@Rule
|
||||
public ActivityTestRule<MyActivity> mRule = new ActivityTestRule<>(
|
||||
MyActivity.class);
|
||||
|
||||
@Before
|
||||
public void setUp() {
|
||||
mRule.launchActivity();
|
||||
}
|
||||
|
||||
public void test() {
|
||||
mRule.runOnUiThread(new Runnable() {
|
||||
@Override
|
||||
public void run() {
|
||||
actionThatNeedsUiThread();
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
1. `assertEquals(float a, float b)` and `assertEquals(double a, double b)` are
|
||||
deprecated in JUnit4's Assert class. **Despite only generating a warning at
|
||||
build time, they fail at runtime.** Please use
|
||||
`Assert.assertEquals(float a, float b, float delta)`
|
||||
|
||||
1. Errorprone expects all public methods starting with `test...` to be
|
||||
annotated with `@Test`. Failure to meet that expectation will cause
|
||||
errorprone to fail with something like this:
|
||||
|
||||
[JUnit4TestNotRun] Test method will not be run; please add @Test annotation
|
||||
|
||||
In particular, you may see this when attempting to disable tests. In that
|
||||
case, the test should be annotated with both @DisabledTest and @Test.
|
||||
|
||||
## Common questions
|
||||
|
||||
- Q: Are `@Test` and `@LargeTest/@MediumTest/@SmallTest` annotation
|
||||
both necessary?
|
||||
- A: Yes, both are required for now. We plan to refactor this in the
|
||||
future.
|
||||
- Q: Isn't the inheritance of the Test classes just migrated to inheritance
|
||||
of TestRules?
|
||||
- A: Yes. During the migration, we plan to maintain a 1:1 mapping between
|
||||
the test base classes and TestRules (e.g. ContentShellTestBase to
|
||||
ContentShellTestRule in this
|
||||
[CL](https://codereview.chromium.org/2632043002)).
|
||||
This allows the auto convert script to replace API calls in any
|
||||
JUnit3 tests. After the migration, we plan to refactor the TestRules to
|
||||
be more modular.
|
||||
|
||||
If you have any other questions, feel free to report in [this bug][7].
|
||||
|
||||
## Links and Crbugs
|
||||
|
||||
- [Android Test Support Library documentation][1]
|
||||
- [Auto change script][5]
|
||||
- [Crbug for JUnit3 to JUnit4 migration][7]
|
||||
|
||||
[1]: https://developer.android.com/topic/libraries/testing-support-library/index.html
|
||||
[2]: https://cs.chromium.org/chromium/src/android_webview/tools/system_webview_shell/layout_tests/AndroidManifest.xml?l=36
|
||||
[3]: http://junit.org/junit4/javadoc/4.12/org/junit/rules/TestRule.html
|
||||
[4]: https://developer.android.com/reference/androidx/test/rule/ActivityTestRule.html
|
||||
[5]: https://github.com/yoland68/chromium-junit-auto-migrate
|
||||
[6]: http://github.com/skyisle/android-test-kit/issues/121
|
||||
[7]: https://bugs.chromium.org/p/chromium/issues/detail?id=640116
|
||||
[8]: http://junit.org/junit4/javadoc/4.12/org/junit/rules/RuleChain.html
|
||||
[9]: https://developer.android.com/reference/android/app/Instrumentation.html#runOnMainSync(java.lang.Runnable)
|
||||
[10]: https://developer.android.com/reference/androidx/test/rule/UiThreadTestRule.html#runOnUiThread(java.lang.Runnable)
|
||||
[11]: /chrome/test/android/javatests/src/org/chromium/chrome/test/BottomSheetTestRule.java
|
||||
[12]: https://bugs.chromium.org/p/chromium/issues/detail?id=734553
|
||||
[13]: /base/test/android/javatests/src/org/chromium/base/test/util/CommandLineFlags.java
|
@ -1,5 +0,0 @@
|
||||
# This page hasn't been written yet!
|
||||
|
||||
If you'd like more information about what you were looking for, try
|
||||
contacting one of the people listed in
|
||||
[//testing/android OWNERS](https://chromium.googlesource.com/chromium/src/+/main/testing/android/OWNERS#2).
|
Reference in New Issue
Block a user