waPC and MsgPack (Rust|Golang)
- 6 minutes read - 1226 wordsAs my reader will know (Hey Mom!), I’ve been noodling around with WASM and waPC. I’ve been exploring ways to pass structured messages across the host:guest boundary.
Protobufs was my first choice. @KevinHoffman created waPC and waSCC and he explained to me and that wSCC uses Message Pack.
It’s slightly surprising to me (still) that technologies like this exist with everyone else seemingly using them and I’ve not heard of them. I don’t expect to know everything but I’m surprised I’ve not stumbled upon msgpack until now.
Anyway…
I rewrote the Rust implementation of my waPC guest proof-of-concept for basic complex number math to use msgpack and then both a Rust and Golang waPC host implementations.
Rust waPC guest
I’m still way back up the learning path on Rust. Feedback is always welcome.
Cargo.toml:
[dependencies]
num-complex = { version = ">=0.0.0", features = ["serde"] }
rmp = ">0.0.0"
rmp-serde = ">0.0.0"
serde = { version = ">0.0.0", features = ["derive"] }
wapc-guest = ">=0.0.0"
NOTE Is it bad form to always grab the latest crate versions?
Following the waPC ‘convention’, lib.rs:
mod complex;
#[macro_use]
extern crate serde_derive;
use wapc_guest::prelude::*;
wapc_handler!(handle_wapc);
pub fn handle_wapc(operation: &str, msg: &[u8]) -> CallResult {
match operation {
"add" => complex::export::add(msg),
"sub" => complex::export::sub(msg),
"mul" => complex::export::mul(msg),
"div" => complex::export::div(msg),
e => Err((String::from("dispatch error: ") + &String::from(e)).into()),
}
}
And then: export.rs, e.g.:
use rmp_serde::{decode, encode};
pub fn add(msg: &[u8]) -> CallResult {
let rqst: ComplexRequest = decode::from_slice(msg).expect("complexrequest");
let c: Complex<f32> = rqst.a + rqst.b;
let resp = encode::to_vec(&c).expect("complex");
Ok(resp)
}
NOTE
rmp-serdeis a crate that facilitates serializing|deserializing (encoding|decoding) using the highly regardedserde
I continue to struggle understanding Rust documentation and it took me some time to realize that the rmp-serde examples were easier done using encode and decode.
What about ComplexRequest?
use serde::{Deserialize, Serialize};
#[derive(Deserialize, Serialize)]
struct ComplexRequest {
a: Complex<f32>,
b: Complex<f32>,
}
impl fmt::Display for ComplexRequest {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(
f,
"ComplexRequest: [a: {}, b: {}]",
self.a.to_string(),
self.b.to_string(),
)
}
}
Compile this with e.g. cargo build --target=wasm32-unknown-unknown --release to yield:
ls -la target/wasm32-unknown-unknown/release
total 1644
4096 Aug 7 11:04 .
4096 Aug 7 10:57 ..
4096 Aug 7 11:04 build
0 Aug 7 10:24 .cargo-lock
332 Aug 7 10:24 complex.d
1648624 Aug 7 11:04 complex.wasm
4096 Aug 7 11:04 deps
4096 Aug 7 10:24 examples
4096 Aug 7 11:04 .fingerprint
4096 Aug 7 10:24 incremental
Golang waPC host
I tested this using a Rust waPC host but, to be concise, I’m going to describe the Golang implementation here. The Golang implementation is more interesting because it requires some tweaking to the Golang structs to generate the expect (!?) msgpack messages.
There were a couple of Golang msgpack SDKs. I chose tinylib msgp. It’s good. It leverages Golang’s code generation which I’d not encountered until now. In this case, code generation provides type-specific implementations of the msgpack Golang interfaces (marshaling, encoding) for user-defined types.
Here’s an example.
This was my first (and incorrect) attempt to define ComplexRequest in Golang:
//go:generate msgp
type ComplexRequest1 struct {
A complex64 `msg:"a,complex"`
B complex64 `msg:"b,complex"`
}
NOTE the annotation
//go:generate msgpdefines files (containing types) that will have code generated by msgp.
NOTE the SDK supports Go’s
complex64,complex128andtime.Timetypes but there’s a caveat. Read on. The above shows how to referencecomplex*.
You’ll need msgp so go get github.com/tinylib/msgp and then run go generate. This will generate two files: main_gen.go and main_gen_test.go (how good is that!?). main_*.go because, in my case, the package is main.
main_gen.go will include DecodeMsg, EncodeMsg, MarshalMsg, UnmarshalMsg, Msgsize for each type in the file (unless otherwise restricted).
You can then:
rqst := ComplexRequest1{
A: complex(float32(39), 3),
B: complex(float32(39), 3),
}
msg, _ := rqst.MarshalMsg(nil)
log.Printf("Encoded: %x\n", msg)
log.Printf("Encoded: %+v\n", msg)
buf := new(bytes.Buffer)
msgp.CopyToJSON(buf, bytes.NewReader(msg))
log.Printf("JSON: %s", buf.String())
This will yield:
Encoded: 82a161d703421c000040400000a162d703421c000040400000
Encoded: [130 161 97 215 3 66 28 0 0 64 64 0 0 161 98 215 3 66 28 0 0 64 64 0 0]
JSON: {"a":{"type:"3,"data":"QhwAAEBAAAA="},"b":{"type:"3,"data":"QhwAAEBAAAA="}}
NOTE as you can see, the
CopyToJSONfunction is useful (for debugging).
You will note that the JSON has a curious encoding for both complex numbers:
{
"type": 3,
"data": "QhwAAEBAAAA="
}
This is because the library is using an in-built Extension for complex64 associated with type 3.
If we take the value of "data", base64 decode it and output it as decimal (printf "QhwAAEBAAAA=" | base64 --decode | od -t u1), we get: 66 280 0 0 64 64 0 0 (see below)
Having also written a Rust version of the waPC host, I know that the Rust-Rust implementation ships the following msgpack message for the above-valued ComplexRequest:
Encoded: [146 146 202 66 28 0 0 202 64 64 0 0 146 202 66 28 0 0 202 64 64 0 0]
NOTE
146(a preamble?) then146then202then66 28 00then202then64 64 0 0. Clearly this matches the expected encoding.
Plugging this in to the Golang code as the value for msg, we get:
JSON: [[39,3],[39,3]]
The above is encoded as arrays (or tuples).
It just so happens that the Golang msgpack SDK supports generating tuples too. But, we need to do this, not only for ComplexRequest but for the complex numbers it represents too.
This could be achieved by tweaking the above struct and providing a new Extension implementation for (an alias of?) complex64 but, for simplicity, I went with the following:
//msgp:tuple ComplexRequest2
type ComplexRequest struct {
A Complex
B Complex
}
//msgp:tuple Complex
type Complex struct {
Re float32 `msg:"re"`
Im float32 `msg:"im"`
}
func NewComplex(c complex64) Complex {
return Complex{
Re: real(c),
Im: imag(c),
}
}
The annotations //msgp:tuple [[Type]] tell msgp to encode the structs as tuple (array) types. After re-running go generate and retrying the code, we will now get:
Encoded: 9292ca421c0000ca4040000092ca421c0000ca40400000
Encoded: [146 146 202 66 28 0 0 202 64 64 0 0 146 202 66 28 0 0 202 64 64 0 0]
JSON: [[39,3],[39,3]]
Great! That matches what the Rust waPC guest implementation expects.
Now we must simply ship this msgpack’d message to it:
code, _ := ioutil.ReadFile("path/to/complex.wasm")
module, _ := wapc.New(consoleLog, code, callback)
defer module.Close()
instance, _ := module.Instantiate()
defer instance.Close()
ctx := context.Background()
result, _ := instance.Invoke(ctx, "add", msg)
log.Printf("[main] Result: %+v\n", result)
NOTE the value of
operationin theInvokemethod matches one of the operation strings ("add","sub","mul","div") defined in thehandle_wapcfunction in the Rustlib.rsabove.
Running this yields:
[callback] binding: binding operation: add payload: ComplexRequest: [a: 39+3i, b: 39+3i]
[callback] binding: binding operation: add payload: 78+6i
Result: [146 202 66 156 0 0 202 64 192 0 0]
NOTE the above includes some host callback lines that I did not show in the Rust guest.
There’s a useful online tool: https://kawanet.github.io/msgpack-lite/
Plugging the Result from the above into the MessagePack panel on the right hand side of the tool (Array) and interspersing , between each value, the JSON is shown to be:
[
78,
6
]
Which, of course, corresponds to a Complex value resulting from 39+3i + 39+3i = 78+6i
All that work to do 2 adds :-)
Let’s just unmarshal result:
var resp Complex
_, _ = resp.UnmarshalMsg(result)
log.Printf("[main] Response: %+v\n", resp)
yields:
Response: {Re:78 Im:6}
A quick tour of msgp using Rust, Golang, WASM and waPC.