mozilla-central/remote/doc/marionette/Testing.md

Source code

Revision control

Copy as Markdown

Other Tools

# Testing
We verify and test Marionette in a couple of different ways, using
a combination of unit tests and functional tests. There are three
distinct components that we test:
* the Marionette **server**, using a combination of xpcshell
unit tests and functional tests written in Python spread across
Marionette- and WPT tests;
* the Python **client** is tested with the same body of functional
Marionette tests;
* and the **harness** that backs the Marionette, or `Mn` job on
try, tests is verified using separate mock-styled unit tests.
All these tests can be run by using [mach].
## xpcshell unit tests
Marionette has a set of [xpcshell] unit tests located in
_remote/marionette/test/xpcshell. These can be run this way:
```shell
% ./mach test remote/marionette/test/unit
```
Because tests are run in parallel and xpcshell itself is quite
chatty, it can sometimes be useful to run the tests sequentially:
```shell
% ./mach test --sequential remote/marionette/test/xpcshell/test_error.js
```
These unit tests run as part of the `X` jobs on Treeherder.
## Marionette functional tests
We also have a set of [functional tests] that make use of the Marionette
Python client. These start a Firefox process and tests the Marionette
protocol input and output, and will appear as `Mn` on Treeherder.
The following command will run all tests locally:
```shell
% ./mach marionette-test
```
But you can also run individual tests:
```shell
% ./mach marionette-test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py
```
In case you want to run the tests with another binary like [Firefox Nightly]:
```shell
% ./mach marionette-test --binary /path/to/nightly/firefox TEST
```
When working on Marionette it is often useful to surface the stdout
from Gecko, which can be achieved using the `--gecko-log` option.
See [Debugging](Debugging.md) for usage instructions, but the gist is that
you can redirect all Gecko output to stdout:
```shell
% ./mach marionette-test --gecko-log - TEST
```
Our functional integration tests pop up Firefox windows sporadically,
and a helpful tip is to suppress the window can be to use Firefox’
headless mode:
```shell
% ./mach marionette-test -z TEST
```
`-z` is an alias for the `--headless` flag and equivalent to setting
the `MOZ_HEADLESS` output variable. In addition to `MOZ_HEADLESS`
there is also `MOZ_HEADLESS_WIDTH` and `MOZ_HEADLESS_HEIGHT` for
controlling the dimensions of the no-op virtual display. This is
similar to using Xvfb(1) which you may know from the X windowing system,
but has the additional benefit of also working on macOS and Windows.
[functional tests]: PythonTests.md
[Firefox Nightly]: https://nightly.mozilla.org/
### Android
Prerequisites:
which means you have the AVD you need.
When running tests on Fennec, you can have Marionette runner take care of
starting Fennec and an emulator, as shown below.
```shell
% ./mach marionette-test --emulator --app fennec
--avd-home /path/to/.mozbuild/android-device/avd
--emulator-binary /path/to/.mozbuild/android-sdk/emulator/emulator
--avd=mozemulator-x86
```
For Fennec tests, if the appropriate `emulator` command is in your `PATH`, you may omit the `--emulator-binary` argument. See `./mach marionette-test -h`
for additional options.
Alternately, you can start an emulator yourself and have the Marionette runner
start Fennec for you:
```shell
% ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828
```
To connect to an already-running Fennec in an emulator or on a device,
you will need to have it started with the `-marionette` command line argument,
or by setting the environment variable `MOZ_MARIONETTE=1` for the process.
Make sure port 2828 is forwarded:
```shell
% adb forward tcp:2828 tcp:2828
```
If Fennec is already started:
```shell
% ./mach marionette-test --app='fennec' --address=127.0.0.1:2828
```
If Fennec is not already started on the emulator/device, add the `--emulator`
option. Marionette Test Runner will take care of forwarding the port and
starting Fennec with the correct prefs. (You may need to run
`adb forward --remove-all` to allow the runner to start.)
```shell
% ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828 --startup-timeout=300
```
If you need to troubleshoot the Marionette connection, the most basic check is
to start Fennec with `-marionette` or the environment variable `MOZ_MARIONETTE=1`,
make sure that the port 2828 is forwarded, and then see if you get any response from
Marionette when you connect manually:
```shell
% telnet 127.0.0.1:2828
```
You should see output like `{"applicationType":"gecko","marionetteProtocol":3}`
[geckodriver]: /testing/geckodriver/index.rst
## WPT functional tests
Marionette is also indirectly tested through [geckodriver] with WPT
(`Wd` on Treeherder). To run them:
```shell
% ./mach wpt testing/web-platform/tests/webdriver
```
WPT tests conformance to the [WebDriver] standard and uses
[geckodriver]. Together with the Marionette remote protocol in
Gecko, they make up Mozilla’s WebDriver implementation.
This command supports a `--webdriver-arg='-vv'` argument that
enables more detailed logging, as well as `--jsdebugger` for opening
the Browser Toolbox.
A particularly useful trick is to combine this with the headless
mode for Firefox:
```shell
% ./mach wpt --webdriver-arg='-vv' --headless testing/web-platform/tests/webdriver
```
## Harness tests
The Marionette harness Python package has a set of mock-styled unit
tests that uses the [pytest] framework. The following command will
run all tests:
```shell
% ./mach python-test testing/marionette
```
To run a specific test specify the full path to the module:
```shell
% ./mach python-test testing/marionette/harness/marionette_harness/tests/harness_unit/test_serve.py
```
## One-click loaners
Additionally, for debugging hard-to-reproduce test failures in CI,
one-click loaners from [Taskcluster](Taskcluster.md) can be particularly useful.
## Out-of-tree testing
All the above examples show tests running _in-tree_, with a local
checkout of _central_ and a local build of Firefox. It is also
possibly to run the Marionette tests _without_ a local build and
with a downloaded test archive from [Taskcluster](Taskcluster.md)
If you want to run tests from a downloaded test archive, you will
need to download the `target.common.tests.tar.gz` artifact attached to
Treeherder [build jobs] `B` for your system. Extract the archive
and set up the Python Marionette client and harness by executing
the following command in a virtual environment:
```shell
% pip install -r config/marionette_requirements.txt
```
The tests can then be found under
_marionette/tests/testing/marionette/harness/marionette_harness/tests_
and can be executed with the command `marionette`. It supports
the same options as described above for `mach`.