Revision control
Copy as Markdown
# Guide to Testing a Rust Component
This document gives a high-level overview of how we test components in `application-services`.
It will be useful to you if you're adding a new component, or working on increasing the test
coverage of an existing component.
If you are only interested in running the existing test suite, please consult the
[contributor docs](../contributing.md) and the [tests.py](https://github.com/mozilla/application-services/blob/main/automation/tests.py) script.
## Unit and Functional Tests
### Rust code
Since the core implementations of our components live in rust, so does the core of our testing strategy.
Each rust component should be accompanied by a suite of unit tests, following the [guidelines for writing
Some additional tips:
* Where possible, it's better use use the Rust typesystem to make bugs impossible than to write
tests to assert that they don't occur in practice. But given that the ultimate consumers of our
code are not in Rust, that's sometimes not possible. The best idiomatic Rust API for a feature
is not necessarily the best API for consuming it over an FFI boundary.
* Rust's builtin assertion macros are sparse; we use the [more_asserts](https://crates.io/crates/more_asserts)
for some additional helpers.
* Rust's strict typing can make test mocks difficult. If there's something you need to mock out in tests,
The Rust tests for a component should be runnable via `cargo test`.
### FFI Layer code
We are currently using [`uniffi`](converting-a-component-to-uniffi.md) to generate most ((and soon all!) of our FFI code and thus the FFI code itself does not need to be extensively tested.
### Kotlin code
The Kotlin wrapper code for a component should have its own test suite, which should follow the general guidelines for
[testing Android code in Mozilla projects](https://github.com/mozilla-mobile/shared-docs/blob/main/android/testing.md#jvm-testing).
In practice that means we use
[JUnit](https://github.com/mozilla-mobile/shared-docs/blob/main/android/testing.md#junit-testing-framework)
as the test framework and
[Robolectric](https://github.com/mozilla-mobile/shared-docs/blob/main/android/testing.md#robolectric-android-api-shadows)
to provide implementations of Android-specific APIs.
The Kotlin tests for a component should be runnable via `./gradlew <component>:test`.
The tests at this layer are designed to ensure that the API binding code is working as intended,
and should not repeat tests for functionality that is already well tested at the Rust level.
But given that the Kotlin bindings involve a non-trivial amount of hand-written boilerplate code,
it's important to exercise that code throughly.
One complication with running Kotlin tests is that the code needs to run on your local development machine,
but the Kotlin code's native dependencies are typically compiled and packaged for Android devices. The
tests need to ensure that an appropriate version of JNA and of the compiled Rust code is available in
their library search path at runtime. Our `build.gradle` files contain a collection of hackery that ensures
this, which should be copied into any new components.
The majority of our Kotlin bindings are autogenerated using [`uniffi`](converting-a-component-to-uniffi.md) and do not need extensive testing.
### Swift code
The Swift wrapper code for a component should have its own test suite, using Apple's
Due to the way that all rust components need to be compiled together into a single ["megazord"](../design/megazords.md)
framework, this entire repository is a single Xcode project. The Swift tests for each component
thus need to live under `megazords/ios-rust/MozillaTestServicesTests/` rather than in the directory
for the corresponding component. (XXX TODO: is this true? it would be nice to find a way to avoid having
them live separately because it makes them easy to overlook).
The tests at this layer are designed to ensure that the API binding code is working as intended,
and should not repeat tests for functionality that is already well tested at the Rust level.
But given that the Swift bindings involve a non-trivial amount of hand-written boilerplate code,
it's important to exercise that code thoroughly.
The majority of our Swift bindings are autogenerated using [`uniffi`](converting-a-component-to-uniffi.md) and do not need extensive testing.
## Integration tests
### End-to-end Sync Tests
#### ⚠️ Those tests were disabled because of how flakey the stage server was. See [#3909](https://github.com/mozilla/application-services/issues/3909) ⚠️
The [`testing/sync-test`](https://github.com/mozilla/application-services/tree/main/testing/sync-test) directory contains a test harness for running sync-related
Rust components against a live Firefox Sync infrastructure, so that we can verifying the functionality
end-to-end.
Each component that implements a sync engine should have a corresponding suite of tests in this directory.
* XXX TODO: places doesn't.
* XXX TODO: send-tab doesn't (not technically a sync engine, but still, it's related)
* XXX TODO: sync-manager doesn't
### Android Components Test Suite
It's important that changes in `application-services` are tested against upstream consumer code in the
a manual process involving:
* Configuring your local checkout of android-components to [use your local application-services
build](./locally-published-components-in-fenix.md).
* Running the android-components test suite via `./gradle test`.
* Manually building and running the android-components sample apps to verify that they're still working.
Ideally some or all of this would be automated and run in CI, but we have not yet invested in such automation.
## Test Coverage
We currently have code coverage reporting on Github using [codecov](https://github.com/mozilla/application-services/blob/main/codecov.yml). However, our code coverage does not tell us how much more coverage is caused by our consumers' tests.
## Ideas for Improvement
* ASan, Memsan, and maybe other sanitizer checks, especially around the points where we cross FFI boundaries.
* We could consider making a mocking backend for viaduct, which would also be mockable from Kotlin/Swift.
* Add more end-to-end integration tests!
* Live device tests, e.g. actual Fenixes running in an emulator and syncing to each other.
* Run consumer integration tests in CI against main.