converting-a-component-to-uniffi.md

# Converting an existing Component to use UniFFI

When we started building the components in this repo, exposing Rust code to

Kotlin and Swift was a manual process and each component had its own

hand-written FFI layer and foreign-language bindings.

As we've gained more experience with building components in this way, we've

started to automate bindings generation and capture best practices in a

tool called [UniFFI](https://mozilla.github.io/uniffi-rs/), which is the

currently recommended approach when [adding a new component from scratch](

./adding-a-new-component.md).

We expect that existing components will gradually be ported over to use

UniFFI, and this document is a guide to doing that port.

## First, get familiar with UniFFI

First, make sure you've perused the [UniFFI guide](https://mozilla.github.io/uniffi-rs/)

to understand the overall architecture of a UniFFI component, and take a look

at the [guide to adding a new component](./adding-a-new-component.md) to understand

how such components fit in to this repo. The aim of porting will be to have a component

that looks like it was added by the process described therein.

## Next, get familiar with the target component

Pre-UniFFI components typically consist of four main parts:

* A Rust crate implementing the core functionality of the component

* A separate Rust crate that exposes the core functionality over a C-style FFI.

* An Android package that imports the C-style FFI into idiomatic Kotlin.

* A Swift module that imports the C-style FFI into idiomatic Swift.

The code for these parts will be laid out something like this:

* `components/<component_name>/`

    * `Cargo.toml`

    * `src/`

        * Rust code for the core functionality of the component goes here.

    * `ffi/`

        * `Cargo.toml`

        * `src/`

            * Rust code specifically for exposing the C-style FFI goes here.

    * `android/`

        * `build.gradle`

        * `src/`

          * `main/`

              * `AndroidManifest.xml`

              * `java/mozilla/appservices/<component_name>/`

                  * `Lib<ComponentName>FFI.kt` (low-level bindings to the C-style FFI)

                  * Higher-level hand-written Kotlin that wraps the FFI.

    * `ios/`

        * `<component_name>/`

          * `Rust<ComponentName>API.h` (low-level bindings to the C-style FFI)

          * Higher-level hand-written Swift that wraps the FFI.

The goal here is to replace much of the hand-written wrapper layers with autogenerated

code:

* The `./ffi/` crate will disappear entirely, its work is automated by UniFFI

    * If you still need some hand-written `pub extern "C"` functions, perhaps to

      implement features not currently supported by UniFFI, then they should move

      into `lib.rs` of the main component crate.

* The low-level `Lib<ComponentName>FFI.kt` file will disappear entirely, as will some of the

  code that converts it back into nice high-level Kotlin classes and interfaces.

    * Some of the hand-written Kotlin code may remain, if it provides functionality that

      cannot be implemented in Rust.

* The low-level `Rust<ComponentName>API.h` file will disappear entirely, as will some of the

  code that converts it back into nice high-level Swift classes and interfaces.

    * Some of the hand-written Swift code may remain, if it provides functionality that

      cannot be implemented in Rust.

You'll aim to end up with a simplified file structure that looks like this:

* `components/<component_name>/`

    * `Cargo.toml`

    * `uniffi.toml`

    * `src/`

        * `<component_name>.udl` (abstract interface definition)

        * Rust code here.

    * `android/`

        * `build.gradle`

        * `src/`

          * `main/`

              * `AndroidManifest.xml`

              * `java/mozilla/appservices/<component_name>/`

                  * Optional hand-written Kotlin code here.

    * `ios/`

        * `<component_name>/`

          * Optional hand-written Swift code here.

## Write a first draft of the `.udl` file for the component's interface

Make sure you've got the `uniffi-bindgen` command available; `cargo install uniffi_bindgen` will

ensure you have the latest version.

Create `./src/<component_name>.udl` and try to describe the intended interface for the component

using [UniFFI's interface definition language](https://mozilla.github.io/uniffi-rs/udl_file_spec.html).

You'll probably need to reverse-engineer it a little bit from the existing hand-written Kotlin and/or

Swift code.

Don't spend too much time on trying to match every minute detail of the existing hand-written API.

There are likely to be small differences between how UniFFI likes to do things and how the hand-written

APIs were structured, and it's in everyone's best long-term interests to just push ahead and update

consumers to accommodate any breaking API changes, rathern than e.g. trying to convince UniFFI to

capitalize enum variant names in the same style that the hand-written code was using.

To check whether the `.udl` file is syntactically valid, you can use `uniffi-bindgen` to generate

the Rust FFI scaffolding like so:

```

uniffi-bindgen scaffolding ./src/<component_name>.udl

```

If this succeeds, it will generate a file `./src/<component_name>.uniffi.rs` with a bunch of

thorny auto-generated Rust code. If it fails, it will likely fail with an inscrutable error message.

Unfortunately the error reporting in UniFFI is currently a known pain point, and it can take a

bit of trial-and-error to identify what part of the file is causing the issue. Sorry :-(

The aim at this point is to ensure that the intended interface of the component can be expressed

in terms that UniFFI understands. Most cases should be supported, but you may find some aspect of

the existing component that is hard to express in UniFFI, perhaps even uncovering new functionality

that needs to be added to UniFFI itself!

The `.udl` file is definitely a first draft at this point. It is normal and expected to need

to iterate on this file as you port over the underlying Rust code.

## Restructure the Rust code to introduce UniFFI

You will now restructure the existing Rust crate so that its public API surface

and overall "shape" match what you defined in the `.udl` file.

Start by deleting the `./ffi` sub-crate, because you're going to use UniFFI to generate

all of that code. You'll also need to remove it from the workspace in the top-level

`Cargo.toml` file, as well as change the crates under `/megazords` to import the core

Rust crate for the component rather than importing the FFI sub-crate.

Add UniFFI to the crate's dependencies and configure its `build.rs` script to invoke the

UniFFI scaffolding generator, as described in ["adding a new component"](adding-a-new-component.md).

Now, edit `./lib.rs` so that it matches the interface defined in the `.udl` file as closely

as possible. If the `.udl` has an `interface Example` then `lib.rs` should contain a

`pub struct Example`, if the `.udl` contains an `enum ExampleItem` then `lib.rs` should

contain a `pub enum ExampleItem`, and so-on.

The details of this step will depend heavily on the specific crate, but some tips include:

* You may find it useful to move all of the existing code into a sub-module named `internal`,

  and then make a brand new `lib.rs` that imports or re-defines just the pieces it needs

  in order to implement the interface from the `.udl` file. The `fxa-client` crate is an

  example of a case where this worked out well, though of course your mileage may vary.

* If the existing crate contains a file named like `<component_name>_msg_types.proto`, then

  it was using Protocol Buffers to serialize data to pass over the FFI. The message types

  defined in the `.proto` file will need to be converted into `dictionary` or `enum` definitions

  in your `.udl` file. See the section below for more details.

As noted above, don't be afraid to accept some API churn during the conversion process.

We're willing to accept some breaking API changes as the cost of getting bindings generated

for free, as long as the core functionality and mental model of the component remain intact.

At this point, in theory the crate should be buildable with UniFFI, although it's likely

to require some iteration to get it all working! Run `cargo check` to check for any

compilation errors without having to do a full build.

### Removing Protobuf Messages

Passing rich structured data over the FFI is the most complex part of our hand-written bindings,

and was previously done by [serializing data via Protocol Buffers](

https://hacks.mozilla.org/2019/04/crossing-the-rust-ffi-frontier-with-protocol-buffers/).

This is something that UniFFI tries to make as simple as possible.

Start by locating the `<component_name>_msg_types.proto` file for the component. This file defines

the structured messages that can be passed over the FFI, and you should see that they correspond

to various types of structured data that the component wants to receive from, or return to,

the foreign-language code.

Find the places in your `.udl` interface that correspond to these message types and make sure

that you've got a similarly-shaped `dictionary` or `enum` for each one. You should find that

representing this structured data in UDL is simpler than protobuf in many cases - for example

many of our `.protobuf` files need to use a separate `ExampleStructs` message in order to

pass a list of `ExampleStruct` messages over the FFI, but in UniFFI this is represented

directly as `sequence<ExampleStruct>`.

Find the places in the Rust code that are using these message types to return structured data.

In simple cases, you may be able to directly replace uses of `msg_types::ExampleStruct` with

the corresponding `crate::ExampleStruct` from your public API.

For more complex cases, you may find it helpful to define an `Into` mapping between the

UniFFI dictionary/enum in the crate's public interface, and a more complex struct designed

for internal use.

As noted above, don't be afraid to accept some API churn during this conversion process.

Once you have replaced all uses of the `msg_types` structs in the Rust code:

* Delete `./src/<component_name>_msg_types.proto`.

* Delete `./src/mozilla.appservices.<component_name>.protobuf.rs`, which is generated from the `.proto` file.

* Remote `prost` and `prost-derive` from the crate's dependencies.

* Delete the crate from the list in `/tools/protobuf_files.toml`.

If you happen to find that you've deleted the last crate from the list in `protobuf_files.toml`,

congratulations! You've successfully removed protocol buffers from this repo entirely, and should

file a bug to track the complete removal of protobuf from our tooling and dependency chain.

## Document the Public API in the Rust code

Write consumer-facing documentation on the public API in `lib.rs` using Rust's standard

[rustdoc](https://doc.rust-lang.org/rustdoc/how-to-write-documentation.html) conventions

and tools. The `fxa-client` crate may serve as a good example.

You can view the generated documentation by running:

```

cargo doc --no-deps --open

```

In future, we intend to automatically extract documentation from the Rust code

and make it easily available to consumers of the generated bindings.

(In fact there is some work-in-progress code in [uniffi-rs#416](https://github.com/mozilla/uniffi-rs/pull/416)

that can read docs from the Rust code and write them back into the `.udl` file, which you're

welcome to try out if you're feeling adventurous. But it's just a very hacky prototype.)

## Set up the Kotlin wrapper

It's easiest to start by removing all of the hand-written Kotlin code under `android/src/main/java`

and then restoring parts of it later if necessary. Leave the `AndroidManifest.xml` file and any tests

in place.

Delete the `android/build.gradle` file and then follow the instructions for [adding Kotlin bindings

for a new component](adding-a-new-component.md#the-kotlin-bindings) to create a new `build.gradle`

file and a corresponding `uniffi.toml`.

This should be all that's required to set up UniFFI to build the Kotlin bindings. Try building

the Android package to confirm:

* `./gradlew <component_name>:assembleDebug`

The UniFFI-generated Kotlin code will be under `./android/build/generated/source/uniffi/` and

may be useful for debugging.

If there are existing Kotlin tests for the component, the next step is to get those passing:

* `./gradlew <component_name>:test`

As noted above, it is normal and expected for the autogenerated bindings to be subtly different

from the previous hand-written ones. For example, UniFFI insists on using SHOUTY_SNAKE_CASE

variant names in Kotlin enums while the hand-written code may have used CamelCase. Some components

also have small naming differences between the Rust code and the hand-written Kotlin bindings,

which UniFFI will not allow.

If the component had functionality in its Kotlin layer that was not part of the Rust API,

then you'll need to add some hand-written Kotlin code under `android/src/main/java` to

implement it. The `fxa-client` component may be a good example here: its Rust layer exposes

a `FirefoxAccount` struct that the Kotlin code wraps into a `PersistedFirefoxAccount` class,

adding the ability to set a persistence callback.

Finally, you will need to try out the new bindings with a consuming app. For Kotlin code you should

[make a local build of android-components and Fenix](locally-published-components-in-fenix.md),

updating them to accomodate any changes in the component's public API.

## Set up the Swift wrapper

It's easiest to start by removing all of the hand-written Swift code under `./ios` and then

restoring parts of it later if necessary.

Edit `/megazords/ios-rust/MozillaTestServices.h` to remove any references to `Rust<ComponentName>API.h`,

replacing them with the UniFFI-generated header file name `<component_name>FFI.h`.

Open `/megazords/ios-rust/MozillaTestServices.xcodeproj` in Xcode and follow the instructions for

[adding Swift bindings for a new component](adding-a-new-component.md#the-swift-bindings) to

configure Xcode to build your UniFFI-generated bindings.

While you are in the Xcode Project Navigator, you should also delete any references to

`Rust<ComponentName>API.h` or to the old hand-written Swift wrappers. (They should be highlighted

in red in the Project Navigator, because the files will be missing from disk after you

deleted them above).

This should be all that's required to set up UniFFI to build the Swift bindings. Try building

the project in Xcode to confirm.

The UniFFI-generated Swift code will be under `ios/Generated` and may be useful for debugging.

If there are existing Swift tests for the component, the next step is to get those passing:

* `./automation/run_ios_tests.sh`

* (or run them from the Xcode GUI)

As noted above, it is normal and expected for the autogenerated bindings to be subtly different

from the previous hand-written ones. Many existing components have small naming differences

between the Rust code and the hand-written Swift bindings, which UniFFI will not allow.

If the component had functionality in its Swift layer that was not part of the Rust API,

then you'll need to add some hand-written Swift code under `./ios/<ComponentName>` to

implement it. The `fxa-client` component may be a good example here: its Rust layer exposes

a `FirefoxAccount` struct that the Swift code wraps into a `PersistedFirefoxAccount` class,

adding the ability to set a persistence callback.

You will need to add any such file to the "Compile Sources" list in Xcode, in the same way

that you added the `.udl` file.

Finally, you will need to try out the new bindings with a consuming app. For Swift code you should make a local build of Firefox iOS, you can do that by following the steps in [this document](./locally-published-components-in-firefox-ios.md)

Revision control

Copy as Markdown