waPC & Protobufs

- 8 minutes read - 1684 words

I’m hacking around with a solution that combines WASM and Google Trillian.

Ultimately, I’d like to be able to ship WASM (binaries) to a Trillian personality and then invoke (exported) functions on them. Some this was borne from the interesting exploration of Krustlet and its application of wascc.

I’m still booting into WASM but it’s a very interesting technology that has most interesting potential outside the browser. Some folks have been trailblazing the technology and I have been reading Kevin Hoffman’s medium and wascc (nee waxosuit) work. From this, I stumbled upon Kevin’s waPC and I’m using waPC in this prototyping as a way to exchange data between clients and servers running WASM binaries.

Protobufs

It seems (!?) to me that protobufs could be a convenient way to pass function data around in my solution. Trillian is a gRPC-based platform. Communication between Trillian personalities and Trillian’s runtime is via gRPC. Often, Trillian personalities extend gRPC to their own clients too. I plan to use gRPC in this way in this proposed solution.

So, if a Trillian personality (client) ships WASM to a Trillian server using gRPC (and protobufs), it’s natural to also use protobufs (but not gRPC) to invoke the WASM functions on the host. However, WASM binaries run in their host’s runtime process; there’s no network stack between the two and thus no need for the R(emote) in RPC (and thus gRPC). Kevin’s waPC provides a mechanism for transferring bytes between a host runtime and a guest WASM binary (and vice versa).

So, this project demonstrates taking arbitrary protobuf messages encoded as Any which are envelopes containing bytes (and a unique descriptor of the type), and shipping these across the WASM host:guest boundary using waPC

Example

I’m a rust noob; feedback is always welcome!

Here’s the prototypical example:

extern crate wapc_guest as guest;

use guest::prelude::*;

wapc_handler!(handle_wapc);

pub fn handle_wapc(operation: &str, msg: &[u8]) -> CallResult {
    match operation {
        "sample:mirror" => mirror(msg),
        _ => Err("bad dispatch".into()),
    }
}

pub fn mirror(msg: &[u8]) -> CallResult {
    // Calls into Host
    host_call("something", "Host", "Call", b"[mirror] Entered")?;
    // Implement Guest function
    let s = match str::from_utf8(msg) {
        Ok(v) => v.chars().rev().collect::<String>(),
        Err(e) => panic!("Invalid UTF-8: {}", e),
    };
    Ok(s.into_bytes())
}

NOTE IIUC the operation (sample:mirror) is simply a string key, i.e. there’s no real structure to it but the host’s call must match one of the function’s supported by the guest.

The match in handle_wapc maps an arbitrary set of incoming string keys with functions.

The implementation of mirror is only slightly confusing with the addition of the host_call but this function calls back to the host with a unique ID for the guest and an operation (string) and payload that could also be pattern-matched on the host for routing to specific functions.

Protobuf-waPC

All that’s left to do is permit the host and guest to exchange protobuf messages this way. As mentioned above, waPC requires bytes for exchange and so we must transform arbitrary protobuf messages down to bytes.

In my prototype, I’m using a very simple protobuf service with 2 methods (add,sub) and sharing SimplexRequest requests (parameters) and SimplexResponse response (results):

syntax = "proto3";

package examples;

option go_package = "github.com/[[PROJECT]]/protos";

service SimplexService {
    rpc add(SimplexRequest) returns (SimplexResponse) {};
    rpc sub(SimplexRequest) returns (SimplexResponse) {};
}
message SimplexRequest {
    int32 a = 1;
    int32 b = 2;
}
message SimplexResponse {
    int32 result = 1;
}

I’m using Stepancheg’s protobuf crate and this permits compiling protobufs to Rust sources using a build.rs:

fn main() {
    match protoc_rust::Codegen::new()
        .out_dir("src/protos")
        .include("../protos")
        .inputs(&["../protos/simplex.proto"])
        .run()
    {
        Ok(_) => println!("Successfully compiled protobufs"),
        Err(e) => println!("{:?}", e),
    }
}

I’m then, of course, using the generated structs in both the host (main.rs) and WASM binary (lib.rs) projects.

Rust WASM Guest

Let’s start with the WASM guest.

We simply add our upcoming functions (add and sub) to the handler:

pub fn handle_wapc(operation: &str, msg: &[u8]) -> CallResult {
    match operation {
        "e:mirror" => example::mirror(msg),
        "e:add" => example::add(msg),
        "e:sub" => example::sub(msg),
        _ => Err("bad dispatch".into()),
    }
}

These functions have a signature: &[u8] -> wapc_guest::prelude::CallResult (borrowed slice of bytes to ). We’re going to implement our WASM functions as if service (stubs) were generated for us by protobuf, i.e.:

fn add(rqst: SimplexRequest) -> SimplexResponse {}
fn sub(rqst: SimplexRequest) -> SimplexResponse {}

You’ll recall from the above that there’s a protobuf (well-known-type) Any that can be used to envelope arbitrary message types, let’s use that:

fn add(msg: &[u8]) -> CallResult {
    // From bytes --> Any
    let rqst = Any {
        type_url: "type.googleapis.com/examples.SimplexRequest".to_string(),
        value: msg.iter().cloned().collect(),
        ..Default::default()
    };
    // Call the RPC
    let resp = x::add(rqst)?;
    // Any --> bytes
    let b = resp.get_value();
    let v = b.iter().cloned().collect();
    Ok(v)
}

NOTE the module x is primarily to keep similarly named functions distinct by namespace.

This get us from message bytes to an Any that includes the bytes (not as a vector) and a type. This type is prefixed by type.googleapis.com and then uses the protobuf package name (examples) and the message (SimpleRequest) to uniquely reference it. This could be anything as long as the receiving end is able to match the same type (string) to the appropriate protobuf message.

Then, we need implementations for e.g. Any-> Any so, for add:

fn add(any: Any) -> Result<Any, ProtobufError> {
    // From Any --> SimplexRequest
    let rqst = match any.unpack::<SimplexRequest>() {
        Ok(v) => match v {
            Some(m) => m,
            None => panic!("protobuf type mismatch"),
            },
        Err(e) => panic!("protobuf message parse failure: {}", e),
    };
    // Call the RPC
    let resp: SimplexResponse = server::add(rqst);
    // From SimplexResponse --> Any
    Any::pack(&resp)
}

Which gets us from Any to e.g. SimplexRequest that we can use to invoke our protobuf implementation of add and then we return SimplexResponse as an Any.

NOTE pack uses the message’s descriptor to set the type of Any. In this case (for SimplexResponse) it will be type.googleapis.com/examples.SimplexResponse.

We compile the code with:

cargo clean && \
cargo build --target wasm32-unknown-unknown

And, all being well, this generates:

ls -l target/wasm32-unknown-unknown/debug/
total 16452
build
deps
examples
incremental
simplex.d
simplex.wasm

NOTE We’re only interested in simple.wasm.

Rust WASM Host

The host is essentially everything above in reverse ;-)

We want to make calls of the form:

let rqst = SimplexRequest {
    a: 39,
    b: 3,
    ..Default::default()
};
let resp = add(rqst);
assert_eq!(resp.get_result(), 42);
println!("{:?}", resp);

As with the waPC example, the host loads the compiled WASM binary file and creates a WapcHost from the WASM and a callback (to handle the guest’s host_calls). In this prototype, the host makes simple calls into the guest’s functions, e.g.:

let resp = host.call(op: "e:add", payload: v)?;

But the payload v needs to be a slice of bytes, so:

fn add(rqst: SimplexRequest) -> SimplexResponse {
    let any: Any = Any::pack(message: &rqst).expect("unable to pack message");
    // Use the Any's value as the message in the call
    // NB Discard the Any's type here
    let v: &[u8] = any.get_value();
    // Make the call
    let resp: Vec<u8> = host.call(op: "e:add", payload: v)?;
    // Create another Any from the response
    // NB Statically assign the Any's type here
    let any: Any = Any {
        type_url: "type.googleapis.com/examples.SimplexResponse".to_string(),
        value: resp,
        ..Default::default()
    };
    // From Any --> SimpleResponse (panic otherwise)
    match any.unpack::<SimplexResponse>() {
        Ok(v: Option<SimpleResponse>) => match v {
            Some(m: SimpleResponse) => m,
            None => panic!("protobuf message type mismatch"),
        },
        Err(e: ProtobufError) => panic!("protobuf message parse mismatch: {}", e),
    }
}

To avoid having to close over host and to better abstract the functionality:

struct Stub {
    host: WapcHost,
}
impl Stub {
    fn _add(self: &mut Self, any: Any) -> Any {
        // NB Discard the incoming Any's type here
        let rqst: &[u8] = any.get_value();
        // Make the call
        let resp: Vec<u8> = self.host.call("e:add", rqst).unwrap();
        // Create another Any from the response
        // NB Statically assign the Any's type here
        Any {
            type_url: "type.googleapis.com/examples.SimplexResponse".to_string(),
            value: resp,
            ..Default::default()
        }
    }
    pub fn add(self: &mut Self, rqst: &SimplexRequest) -> SimplexResponse {
        let rqst: Any = Any::pack(rqst).expect("unable to pack message");
        let resp = self._add(rqst);
        // From Any --> SimpleResponse (panic otherwise)
        match resp.unpack::<SimplexResponse>() {
            Ok(v) => match v {
                Some(m) => m,
                None => panic!("protobuf message type mismatch"),
            },
            Err(e) => panic!("protobuf message parse mismatch: {}", e),
        }
    }
}

Running this:

cargo run

Gets us:

cargo run
   Compiling simplex v0.1.0 (/home/.../host)
    Finished dev [unoptimized + debuginfo] target(s) in 5.61s
     Running `target/debug/simplex`
result: 42

Conclusion

Out-of-the-box, WASM currently permits only numbers to be used as function parameters and a single number value to be returned. Using, waPC, it’s possible to invoke functions using bytes and receive results as bytes. Using protobufs and the Any well-known type, these lower-level bytes can be typed. Even though the example described above is both trivial and easily implemented without protobufs and waPC (because simple math functions are possible using WASM’s current parameter|result constraints), the example should (!) support arbitrary protobuf messages.

Backgrounder

WASM-related technologies as I understand them. Much has been written about WASM and I won’t attempt to replicate that or explain WASM here. Suffice to say that the Krustlet team makes a compelling argument for the utility of WASM in its blog post announcing Krustlet when it writes “A comparison of Linux Containers and WebAssembly”.

Assuming we’re running WASM outside of the browser, we need a host|runtime environment for it. There are several being developed (wasmtime, lucet, wascc). An immediate problem is how WASM functions may access host functionality (e.g. networking) and a solution is an interface called WASI that all of the aforementioned runtimes support (it’s like *nix’s POSIX) and ensures that not only does WASM run unchanged between runtimes but that a host runtime implements functionality expected of a WASI runtime. wascc goes further and provides a mechanism for pluggable ‘capabilities’ (e.g. HTTP Server Provider, logging, redis etc.) that provide higher-level functionality to WASM binaries. When I find the time, I’d like to write a Prometheus Provider.

Lastly, moving data into and from WASM binaries is currently limited (to numerical data and shared memory) see link. There’s a proposal for WASM Inteface Types which appears to provide a more powerful (typed?) mechanism. There’s also waPC which provides a low-level ‘bytes’ format that I’m using in this project.