tokio/tokio/docs/ffi.md

RFC: An FFI API for Tokio

Summary

This RFC proposes the creation of an initially unstable FFI API for the Tokio crate. In the interest of seeing implementation, usage, and feedback faster, the initial scope of this effort will be narrow but will grow over time and evolve as we better explore the solution space. This is expected to remain unstable for some time as it is a relatively new and unproven problem space, and the API is expected to potentially evolve considerably over time.

Motivation

While "Rewrite it in Rust" is a common trope, for many projects this is not realistic. Many projects are too large to rewrite entirely or non-incrementally. Thus, when adopted, Rust may often have to live alongside C or C++ in a codebase for a significant amount of time, potentially permanently.

Unfortunately, Tokio does not currently expose an FFI API which other languages can interact with. This creates a significant barrier to the use of Tokio and the adoption of Rust within these systems.

However, if we were to expose an FFI API, existing C/C++ event loops capable of doing so could be bridged to Tokio, significantly easing this process.

Long term, it may even be desirable to allow the reverse - Tokio could act as an interface into other runtimes, however that is much dicier and will be left out of the scope of this proposal.

Implementation Design

Within the Tokio crate, an ffi-experimental subdirectory will be added containing three subdirectories, public-include, containing header files specifying the public interface of the crate, examples, containing examples of usage which should be tested for compilation in CI, and tests, containing source code for a library consisting of test cases which we can build, link to, and invoke when calling Tokio's integration tests behind a special --cfg flag. A Makefile with rules for invoking tests and running examples should also be added. Within the Tokio crate, an ffi_experimental module should be added containing extern bindings for exposed APIs.

It is assumed initially that users will be vendoring this code and managing builds themselves, so any build system logic we have should be focused on our own testing and usage.

Header files and extern "C" bindings should be written and designed manually and not autogenerated using tooling. Header files will be targeted at C17.

Process and Lifecycle

This API should be implemented in several phases. The API should be evolved and design changes and decisions should be made as we go along. The API should be kept clearly marked as unstable, although we should also avoid and very clearly document breaking changes where we make them, and ideally phase these changes by preferring new functions over changing existing ones.

The initial phase should be the implementation of core features: runtime construction and operation, as well as task spawning, waking, and execution, timers, signals, and IO via AsyncFd. In this phase, the goal is to ship the minimum viable product capable of being useful and gather feedback on it to feed into subsequent phases and amend plans as needed.

After the core components have been shipped, we can begin exploring additional APIs which may be useful to expose, and can continue iterating on existing components. New APIs may include exposing additional modules within Tokio, or even useful bridge-layer abstractions, like coroutine support.

This RFC avoids discussions on the precise API designs themselves in order to avoid and defer bike-shedding and to establish early consensus about the usefulness of this initiative.

Drawbacks

This may be a significant undertaking, and it may take time and iteration to get this right.

Rationale and alternatives

We could insist that users maintain their own bridges, but that would potentially limit the scope of what can be bridged, impose a significant burden on users wishing to do so, and lead to wasted, duplicative effort.

Prior art

There is relatively limited prior art here - we are treading on a significant amount of new ground. As a result, this RFC heavily emphasizes the evolvability of any design.

Unresolved questions

All details of the specific APIs themselves are left unresolved here, and should be handled individually by API.

Also left unresolved through this RFC are questions of modifications to Tokio itself needed may be needed. For example, do we need to expose cooperatively abortable tasks as a construct, or support dynamic linking?

Future possibilities

As this initiative evolves and progresses, we can add feedback for more APIs within Tokio, add support for C++ coroutines, and potentially even support allowing Tokio to act as an interface to other runtimes, allowing Tokio to be used as a runtime for runtimes like Folly, rather than just a backend.