Major feature: Integrate with capnp-rpc (Cap'n Proto RPC layer).

[ygoldfeld]

Consider the high-level task of transmitting structured data between two local processes with zero-copy (via SHM). We like Cap'n Proto as the structured-data representation in this context. (gRPC is another popular choice. Integrating with it might be a similar major-feature ticket.)

Roughly speaking there are three components involved in this:

1. Low-level transmission of binary blobs (+native handles/FDs if desired) over IPC primitives (Unix domain sockets, MQs; so far that's what we have but could be others, and our concepts formally support whatever).
2. Placing native data into SHM and sharing it between processes, to enable zero-copy.
3. Representing of capnp-structured data conveniently and exchanging messages using a convenient protocol (at least with things like request-response but potentially more advanced concepts such as programmatic interfaces aka RPC, remote procedure call).

For our purposes here we can consider 1 and 2 to be one layer -- call it transport-layer. Call 3 to be protocol-layer.

As it stands, Flow-IPC provides all of this. In particular, the protocol-layer is provided in the form of ipc::transport::struc::Channel (+ Msg_in, Msg_out). These are the stars of ipc_transport_structured repo. The other stuff is ipc::transport (in ipc_core) and ipc::shm (general SHM concepts) + specific SHM-providers that slot into it (SHM-classic and SHM-jemalloc, in various shm::classic and shm::arena_lend::jemalloc sub-namespaces). The session stuff in ipc_session glues it all together, when one wants to initially connect two processes in a conversation (session).

But ipc_transport_structured (Channel, Msg_in, Msg_out) are the main point. The nice thing about it is it's fairly simple to slot into any event loop (at least relatively speaking), with the async-I/O and sync_io patterns both provided, depending on what suits one's event loop style best. It does provide 2-3 basic protocol features:

• Organizing of an overall process-to-process protocol into channels/messages.
• Mux/demux by message type. (E.g., register a handler for GET requests and a separate one for POST requests.)
• Request/response (optional to use). (E.g., send a GET request message, expect a response to that specific message.) (It's optional, in that one can also send notification messages, meaning no response is expected.)

The less-nice things about it are:

• It's a new, custom protocol. A given user may or may not want to use such a thing. They might be used to capnp-rpc, or gRPC, or HTTP, or....
• It's not all that advanced. The above 3 things are fine, but it lacks RPC-type semantics, where one expresses things as interfaces and procedure calls.

It's pretty useful, we've found. However, so is capnp-rpc. It has full-on RPC semantics and has promise pipelining. It's quite clever! And, if you've got a capnp install, you can use capnp-rpc. Integrating it into an existing event loop is not trivial, and the provided all-in-one starting point that is easy to use (called EzRpc) makes major assumptions and probably cannot be used directly in more complex production applications. Nevertheless, it is certainly doable and worth it. (In conversation with Mr. Kenton Varda, capnp's creator, he opined that while most capnp users limit themselves to the serialization layer, to him the RPC layer is where the real power of the product lies.)

TL;DR: We should integrate the capnp-rpc protocol goodies with our transport-layer. In short, one can then use capnp-rpc and gain the zero-copy goodness of Flow-IPC. (Kenton mentioned combining capnp-rpc with a SHM layer was on his to-do list, but now that Flow-IPC is a thing, ideally they'd just use that. For my part, I can attest that properly implementing SHM support in this -- or any, really -- context is no mean feat. Flow-IPC however has done the hard work on that already.)

Thus, this ticket is about providing an alternative to ipc::transport::struc::Channel (et al) as the structured-data protocol layer, in the form of capnp-rpc specifically.

Details omitted for now. I've gotten some tips from Kenton on the likeliest integration points in capnp, and we have some institutional experience at Akamai w/r/t non-zero-copy capnp-rpc integration into event loops, beyond EzRpc.

Difficulty

While certainly non-trivial, it seems clear to me that all the hard parts are available and merely need judicious gluing together. It might start as taking EzRpc and then modifying it to work with our transport-layer instead of its current direct use of stream sockets (locally, Unix domain stream sockets). This process should make reasonably clear a decent API, as the main EzRpc classes gets blown up into the appropriate APIs for flexibly setting this up.

Priority

This is the next major project. I would like to get it done by the end of Q1-2025.