FauxRPC using gRPCurl, Golang and rust
- 12 minutes read - 2402 wordsRead FauxRPC + Testcontainers on Hacker News and was intrigued. I spent a little more time “evaluating” this than I’d planned because I’m forcing myself to use rust as much as possible and my ignorance (see below) caused me some challenges.
The technology is interesting and works well. The experience helped me explore Testcontainers too which I’d heard about but not explored until this week.
For my future self:
What | What? |
---|---|
FauxRPC | A general-purpose tool (built using Buf’s Connect) that includes registry and stub (gRPC) services that can be (programmatically) configured (using a Protobuf descriptor) and stubs (example method responses) to help test gRPC implementations. |
Testcontainers | Write code (e.g. rust) to create and interact (test)services (running in [Docker] containers). |
Connect | (More than a) gRPC implementation used by FauxRPC |
gRPCurl | A command-line gRPC tool. |
I started following along with FauxRPC’s Testcontainers example but, being unfamiliar with Connect, I wasn’t familiar with its Eliza service. The service is available on demo.connectrpc.com:443
and is described using eliza.proto
as part of examples-go
. Had I realized this sooner, I would have used this example rather than the Health Checking protocol.
FileDescriptorSet
The gRPC Health Checking Protocol is my go-to service and the one I used instead.
We’re going to need a Protobuf Descriptor for the Health Checking protocol:
protoc \
--descriptor_set_out=health.bin \
health.proto
NOTE The Descriptor generated by default by
protoc
differs from the one generated by rust tonic. The latter includesSourceCodeInfo
by default. The difference is not material to what follows.
To use gRPCurl with FauxRPC registry service, we’re going to need to create an AddDescriptorRequest
and to format the message as JSON. It’s straightforward to do this using any of the Protocol Buffer languages.
We need to use protoc
to compile the registry_service.proto
:
PROTOS="/path/to/protos" # Including registry/v1/registry_service.proto
protoc \
--proto-path=${PROTOS} \
--python_out=${PWD} \
--pyi_out=${PWD} \
${PROTOS}/registry/v1/registry_server.proto
This will generate registry/v1/registry_service_pb2.py
which includes AddDescriptorsRequest
.
Here’s the Python code using protobuf
:
from google.protobuf import json_format
from google.protobuf import descriptor_pb2
from registry.v1 import registry_service_pb2
with open("health.bin", "rb") as f:
d = descriptor_pb2.FileDescriptorSet()
with open("descriptor.json", "w") as f:
r = registry_service_pb2.AddDescriptorsRequest(
descriptors=fds
)
j = json_format.MessageToJson(r)
f.write(j)
The result is:
{
"descriptors": {
"file": [
{
"name": "health.proto",
"package": "grpc.health.v1",
"messageType": [
{
"name": "HealthCheckRequest",
"field": [
{
"name": "service",
"number": 1,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "service"
}
]
},
{
"name": "HealthCheckResponse",
"field": [
{
"name": "status",
"number": 1,
"label": "LABEL_OPTIONAL",
"type": "TYPE_ENUM",
"typeName": ".grpc.health.v1.HealthCheckResponse.ServingStatus",
"jsonName": "status"
}
],
"enumType": [
{
"name": "ServingStatus",
"value": [
{
"name": "UNKNOWN",
"number": 0
},
{
"name": "SERVING",
"number": 1
},
{
"name": "NOT_SERVING",
"number": 2
},
{
"name": "SERVICE_UNKNOWN",
"number": 3
}
]
}
]
}
],
"service": [
{
"name": "Health",
"method": [
{
"name": "Check",
"inputType": ".grpc.health.v1.HealthCheckRequest",
"outputType": ".grpc.health.v1.HealthCheckResponse"
},
{
"name": "Watch",
"inputType": ".grpc.health.v1.HealthCheckRequest",
"outputType": ".grpc.health.v1.HealthCheckResponse",
"serverStreaming": true
}
]
}
],
"options": {
"javaPackage": "io.grpc.health.v1",
"javaOuterClassname": "HealthProto",
"javaMultipleFiles": true,
"goPackage": "google.golang.org/grpc/health/grpc_health_v1",
"csharpNamespace": "Grpc.Health.V1"
},
"syntax": "proto3"
}
]
}
}
gRPCurl
and buf curl
Even though I began by writing a Golang test for the Health Checking service under FauxRPC running in a Testcontainers (container!?), I’m going to reorder the work to show a more logical evolution. So, let’s start by installing the FauxRPC (Golang) binary and the Buf CLI:
go install github.com/sudorandom/fauxrpc/cmd/fauxrpc@latest
go install github.com/bufbuild/buf/cmd/buf@latest
NOTE You can
fauxrpc run --schema=health.proto
to register the Health Checking Protocol directly:fauxrpc run --schema=health.proto
And then interact with the service:
grpcurl \ -plaintext \ -proto health.proto \ localhost:6660 grpc.health.v1.Health/Check
Returning:
{ "status": "NOT_SERVING" }
For consistency with what follows, we’ll:
PORT="6660"
fauxrpc run --empty --addr=":${PORT}"
NOTE If you’d prefer you can run the FauxRPC container (
docker.io/sudorandom/fauxrpc
)
FauxRPC (dev (none) @ unknown; go1.23.2) - 0 services loaded
Listening on http://:6660
OpenAPI documentation: http://:6660/fauxrpc/openapi.html
Example Commands:
$ buf curl --http2-prior-knowledge http://:6660 --list-methods
$ buf curl --http2-prior-knowledge http://:6660/[METHOD_NAME]
Server started.
And, per the instruction, you can:
buf curl \
--http2-prior-knowledge \
http://:6660 \
--list-methods
Which should be empty (per the --empty
flag).
FauxRPC includes a registry service (registry_service.proto
) and a stubs service (stubs_service.proto
).
I’m going to use gRPCurl to interact with FauxRPC(’s gRPC services).
We will use the registry service to register the Health Checking Protocol using its descriptor (health.bin
):
NOTE We saw this implicitly with the Python code but
registry_service.proto
imports Google’sdescriptor.proto
. The easiest way to avoid recursive dependencies is to clone Google’sgoogleapis
cat descriptor.json \
| grpcurl \
-plaintext \
--import-path=protos \
--proto=protos/registry/v1/registry_service.proto \
-d @ \
localhost:6660 \
registry.v1.RegistryService.AddDescriptors
{}
NOTE
AddDescriptors
returnsAddDescriptorsResponse
which is{}
.
But, we can rerun the buf curl
command to see that we know have the Health Checking Protocol’s Check
and Watch
methods in the registry:
buf curl --http2-prior-knowledge http://:6660 --list-methods
grpc.health.v1.Health/Check
grpc.health.v1.Health/Watch
Equivalently, we can use gRPCurl:
grpcurl -plaintext localhost:6660 describe grpc.health.v1.Health
grpc.health.v1.Health is a service:
service Health {
rpc Check ( .grpc.health.v1.HealthCheckRequest ) returns ( .grpc.health.v1.HealthCheckResponse );
rpc Watch ( .grpc.health.v1.HealthCheckRequest ) returns ( stream .grpc.health.v1.HealthCheckResponse );
}
And, these can be invoked:
grpcurl -plaintext localhost:6660 grpc.health.v1.Health/Check
{
"status": "NOT_SERVING"
}
Hang on, why are we getting NOT_SERVING
from a service that is only a mock without any implementation? The reason is that FauxRPC is generating random values for the value of the enum used to represent status
. If we repeat the command, we should get random results.
Using FauxRPC’s Stubs service, we can define a specific result: 2 corresponds to NOT_SERVING
D='{
"stubs":[
{
"ref":{
"id":"11",
"target":"grpc.health.v1.Health/Check"
},
"json":"{\"status\":2}"
}
]
}'
grpcurl \
-plaintext \
--import-path=protos \
-proto=protos/stubs/v1/stubs_service.proto \
-d "${D}" \
localhost:6660 \
stubs.v1.StubsService/AddStubs
{
"stubs": [
{
"ref": {
"id": "11",
"target": "grpc.health.v1.HealthCheckResponse"
},
"json": "{\"status\":2}"
}
]
}
Now, when we invoke the Check
method, we’ll get NOT_SERVING
as the result.
Golang
I wrote several Golang tests and implementations in order to help me build the rust clients.
The first was the result of following the FauxRPC tutorial to test running fauxrpctestcontainers
. So, although it’s duplicative, it uses the Health Checking protocol instead of Eliza and includes some of the (possibly obvious steps) omitted from the tutorial.
But first, an explanation. testcontainers
provides an interface for running arbitrary containers (often servers|services) to facilitate testing. Developers can also provide implementations that wrap some of this functionality to provide a more streamlined solution (see Modules). There is a fauxrpctestcontainers
Golang package that provides an implementation for FauxRPC and makes it easier to use.
So here are two solutions in Golang. The first using the FauxRPC binary (e.g. fauxrpc run --empty --addr=":${PORT}"
or using the container image) which makes the gRPC calls AddDescriptors
and AddStubs
explained previously:
package main
import (
"context"
"fmt"
"log/slog"
"os"
"testing"
healthpb "{MODULE}/protos/grpc/health/v1"
registrypb "{MODULE}/protos/registry/v1"
stubspb "{MODULE}/protos/stubs/v1"
"github.com/google/uuid"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/descriptorpb"
)
const (
health string = "/path/to/health.bin"
)
const (
endpoint string = ":6660"
)
func TestMain(t *testing.T) {
opts := []grpc.DialOption{
grpc.WithTransportCredentials(insecure.NewCredentials()),
}
conn, err := grpc.NewClient(endpoint, opts...)
if err != nil {
t.Error(err)
}
defer conn.Close()
ctx := context.Background()
{
clientRegistryService := registrypb.NewRegistryServiceClient(conn)
b, err := os.ReadFile(health)
if err != nil {
t.Error(err)
}
fds := &descriptorpb.FileDescriptorSet{}
if err := proto.Unmarshal(b, fds); err != nil {
t.Error(err)
}
rqst := ®istrypb.AddDescriptorsRequest{
Descriptors: fds,
}
resp, err := clientRegistryService.AddDescriptors(ctx, rqst)
if err != nil {
t.Error(err)
}
slog.Info("Response", "resp", resp)
}
{
clientStubsService := stubspb.NewStubsServiceClient(conn)
id := uuid.New().String()
method := "grpc.health.v1.Health/Check"
ref := &stubspb.StubRef{
Id: id,
Target: method,
}
b, err := proto.Marshal(&healthpb.HealthCheckResponse{
Status: healthpb.HealthCheckResponse_SERVING,
})
if err != nil {
t.Error(err)
}
content := stubspb.Stub_Proto{
Proto: b,
}
rqst := &stubspb.AddStubsRequest{
Stubs: []*stubspb.Stub{
{
Ref: ref,
Content: &content,
},
},
}
x, err := proto.Marshal(rqst)
if err != nil {
t.Error(err)
}
slog.Info("Request", "x", fmt.Sprintf("%x", x))
resp, err := clientStubsService.AddStubs(ctx, rqst)
if err != nil {
t.Error(err)
}
slog.Info("Response", "resp", resp)
}
{
clientHealthService := healthpb.NewHealthClient(conn)
rqst := &healthpb.HealthCheckRequest{
Service: "",
}
resp, err := clientHealthService.Check(ctx, rqst)
if err != nil {
t.Error(err)
}
slog.Info("Response", "resp", resp)
}
}
You’ll need to replace {MODULE}
with your go mod init ${MODULE}
value and, you’ll want to compile the protocol buffers sources that you can find in the FauxRPC repo proto
folder:
PROTOS="/path/to/protos"
protoc \
--proto_path=${PROTOS} \
--go_out=${PWD} \
--go_opt=Mregistry/v1/registry_service.proto=protos/registry/v1 \
--go_opt=Mstubs/v1/stubs_service.proto=protos/stubs/v1 \
--go_opt=Mstubs/v1/stubs.proto=protos/stubs/v1 \
--go-grpc_out=${PWD} \
--go-grpc_opt=Mregistry/v1/registry_service.proto=protos/registry/v1 \
--go-grpc_opt=Mstubs/v1/stubs_service.proto=protos/stubs/v1 \
--go-grpc_opt=Mstubs/v1/stubs.proto=protos/stubs/v1 \
${PROTOS}/registry/v1/registry_service.proto \
${PROTOS}/stubs/v1/stubs_service.proto \
${PROTOS}/stubs/v1/stubs.proto
The second approach uses fauxrpctestcontainers
and is more straightforward. It will run (and terminate on success) a docker.io/sudorandom/fauxrpc
container for you so you don’t need to run fauxrpc
manually:
package main
import (
"context"
"os"
"strings"
"testing"
fauxrpctestcontainers "github.com/sudorandom/fauxrpc/testcontainers"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/health/grpc_health_v1"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/reflect/protodesc"
"google.golang.org/protobuf/reflect/protoreflect"
"google.golang.org/protobuf/types/descriptorpb"
)
const (
health string = "/path/to/health.bin"
)
const (
img string = "docker.io/sudorandom/fauxrpc:v0.0.23"
method string = "grpc.health.v1.Health/Check"
status = grpc_health_v1.HealthCheckResponse_SERVING
)
func getHealthFileDescriptor(name string) (protoreflect.FileDescriptor, error) {
data, err := os.ReadFile(name)
if err != nil {
return nil, err
}
fds := &descriptorpb.FileDescriptorSet{}
if err := proto.Unmarshal(data, fds); err != nil {
return nil, err
}
fd, err := protodesc.NewFile(fds.File[0], nil)
return fd, err
}
func TestMain(t *testing.T) {
ctx := context.Background()
container, err := fauxrpctestcontainers.Run(ctx, img)
if err != nil {
t.Error(err)
}
t.Cleanup(func() {
container.Terminate(ctx)
})
fd, err := getHealthFileDescriptor(health)
if err != nil {
t.Error(err)
}
container.MustAddFileDescriptor(ctx, fd)
// Define method's response
want := status
container.MustAddStub(ctx, method, &grpc_health_v1.HealthCheckResponse{
Status: want,
})
// BaseURL includes "http://" which is incompatible with grpc.NewClient
baseURL := container.MustBaseURL(ctx)
baseURL = strings.TrimPrefix(baseURL, "http://")
conn, err := grpc.NewClient(
baseURL,
grpc.WithTransportCredentials(insecure.NewCredentials()),
)
if err != nil {
t.Error(err)
}
defer conn.Close()
healthClient := grpc_health_v1.NewHealthClient(conn)
rqst := &grpc_health_v1.HealthCheckRequest{}
resp, err := healthClient.Check(ctx, rqst)
if err != nil {
t.Error(err)
}
got := resp.Status
if got != want {
t.Errorf("got: %s, want: %s", got, want)
}
}
rust
These dependencies
and build-dependencies
are required by the code that follows:
[dependencies]
prost = "0.13.3"
prost-types = "0.13.3"
redis = "0.27.4"
testcontainers = "0.23.1"
tokio = "1.40.0"
tonic = { version = "0.12.3", features=["tls","tls-roots"] }
uuid = { version = "1.11.0", features=["v4"] }
[build-dependencies]
tonic-build = "0.12.3"
Rust’s tonic
comes with a companion tonic-build
that enables us to write a build.rs
to compile protobufs:
build.rs
:
use std::{env, path::PathBuf};
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Required by prost-reflection to create descriptors.bin in correct OUT_DIR
let out_dir = PathBuf::from(env::var("OUT_DIR").expect("unable to determine `OUT_DIR`"));
// `file_descriptor_set_path` appends `.bin` to the file that's created
tonic_build::configure()
.build_client(true)
.build_server(false)
.file_descriptor_set_path(out_dir.join("health.bin"))
.compile_protos(&["/path/to/protos/grpc/health/v1/health.proto"], &["/path/to/protos"])?;
tonic_build::configure()
.build_client(true)
.build_server(false)
.file_descriptor_set_path(out_dir.join("registry_service.bin"))
.compile_protos(
&["/path/to/protos/registry/v1/registry_service.proto"],
&["/path/to/protos"],
)?;
let googleapis = "/path/to/googleapis";
tonic_build::configure()
.build_client(true)
.build_server(false)
.file_descriptor_set_path(out_dir.join("stubs_service.bin"))
.compile_protos(
&["/path/to/protos/stubs/v1/stubs_service.proto"],
&["/path/to/protos", googleapis],
)?;
Ok(())
}
There are 3 variants of the rust implementation. The first also assumes fauxrpc --empty --addr=":${PORT}
or that you’re running the FauxRPC container:
static FILE_DESCRIPTOR_SET: &[u8] = include_file_descriptor_set!("health");
#[tokio::test]
async fn test_grpc() -> Result<(), Box<dyn std::error::Error + 'static>> {
let host = "localhost";
let port = 6660;
let url = format!("http://{host}:{port}");
let uri: tonic::transport::Uri = url
.parse()
.expect("invalid gRPC endpoint");
let endpoint = Channel::builder(uri);
let channel = endpoint.connect()
.await
.expect("unable to connect");
let file_descriptor_set =
FileDescriptorSet::decode(FILE_DESCRIPTOR_SET)
.expect("invalid file descriptor set");
let mut registry_service_client =
RegistryServiceClient::new(channel.clone());
let rqst = tonic::Request::new(AddDescriptorsRequest {
descriptors: Some(file_descriptor_set),
});
let resp = registry_service_client
.add_descriptors(rqst)
.await
.expect("invalid response");
println!("{:?}", resp);
let mut stubs_service_client =
StubsServiceClient::new(channel.clone());
let id = Uuid::new_v4().as_hyphenated().to_string();
let method = "grpc.health.v1.Health/Check".to_string();
let r#ref = Some(stubs::v1::StubRef {
id: id,
target: method,
});
let b = HealthCheckResponse { status: 1 }.encode_to_vec();
let content = Some(stubs::v1::stub::Content::Proto(b));
let rqst = tonic::Request::new(AddStubsRequest {
stubs: vec![stubs::v1::Stub {
r#ref: r#ref,
content: content,
}],
});
let resp = stubs_service_client
.add_stubs(rqst)
.await
.expect("invalid response");
println!("{:?}", resp);
let mut health_client = HealthClient::new(channel.clone());
let rqst = tonic::Request::new(HealthCheckRequest {
service: "".to_owned(),
});
let resp = health_client.check(rqst).await?;
let msg = resp.into_inner();
assert_eq!(msg.status, 1);
Ok(())
}
Replacing the static references to a local fauxrpc --empty --addr=":${PORT}
(or container):
let host = "localhost";
let port = 6660;
We can create a testcontainer
:
let image = "docker.io/sudorandom/fauxrpc";
let tag = "v0.0.23";
let port: u16 = 6660;
let container = GenericImage::new(image, tag)
.with_exposed_port(port.tcp())
.with_wait_for(WaitFor::message_on_stdout("Server started."))
.with_cmd(vec![
"run",
"--log-level=debug",
"--empty",
format!("--addr=:{port}").as_str(),
])
.start()
.await?;
let host = container.get_host().await?;
let port = container.get_host_port_ipv4(port).await?;
I’m a rust noob but I’ve begun to implement something similar to fauxrpctestcontainers
in rust and using testcontainers-modules
as a guide.
It is very much a work-in-progress:
mod.rs
:
use testcontainers::{
core::{ContainerPort, WaitFor},
Image, ImageExt,
};
use crate::{
registry::v1::{
registry_service_client::RegistryServiceClient, AddDescriptorsRequest,
AddDescriptorsResponse,
},
stubs::v1::{
stub::Content, stubs_service_client::StubsServiceClient, AddStubsRequest, AddStubsResponse,
Stub, StubRef,
},
};
use tonic::{transport::Channel, Status};
use uuid::Uuid;
const DEFAULT_IMAGE_NAME: &str = "docker.io/sudorandom/fauxrpc";
const DEFAULT_IMAGE_TAG: &str = "v0.0.23";
const FAUXRPC_PORT: ContainerPort = ContainerPort::Tcp(6660);
#[derive(Debug, Default, Clone)]
pub struct FauxRPC {}
impl FauxRPC {
pub async fn add_file_descriptor(
&self,
channel: Channel,
file_descriptor_set: prost_types::FileDescriptorSet,
) -> Result<AddDescriptorsResponse, Status> {
let mut client: RegistryServiceClient<Channel> =
RegistryServiceClient::new(channel);
let rqst = tonic::Request::new(AddDescriptorsRequest {
descriptors: Some(file_descriptor_set),
});
let resp = client.add_descriptors(rqst).await;
match resp {
Ok(r) => Ok(r.into_inner()),
Err(e) => Err(e),
}
}
pub async fn add_stub(
&self,
channel: Channel,
bytes: Vec<u8>,
) -> Result<AddStubsResponse, Status> {
let mut client: StubsServiceClient<Channel> =
StubsServiceClient::new(channel);
let id = Uuid::new_v4().as_hyphenated().to_string();
let method = "grpc.health.v1.Health/Check".to_string();
let rqst = tonic::Request::new(AddStubsRequest {
stubs: vec![Stub {
r#ref: Some(StubRef {
id: id,
target: method,
}),
content: Some(Content::Proto(bytes)),
}],
});
let resp = client.add_stubs(rqst).await;
match resp {
Ok(r) => Ok(r.into_inner()),
Err(e) => Err(e),
}
}
}
impl Image for FauxRPC {
fn name(&self) -> &str {
DEFAULT_IMAGE_NAME
}
fn tag(&self) -> &str {
DEFAULT_IMAGE_TAG
}
fn ready_conditions(&self) -> Vec<WaitFor> {
vec![WaitFor::message_on_stdout("Server started")]
}
fn expose_ports(&self) -> &[ContainerPort] {
&[FAUXRPC_PORT]
}
}
// impl ImageExt<I> for FauxRPC {}
#[cfg(test)]
mod tests {
use crate::{
fauxrpctestcontainers::FauxRPC,
testcontainers::runners::AsyncRunner, ImageExt,
};
#[tokio::test]
async fn test_fauxrpc() -> Result<(), Box<dyn std::error::Error + 'static>> {
let container = FauxRPC::default()
// Would be good for this to be the default
// But it requires impl ImageExt which has a bunch of functions to be implemented
.with_cmd(vec![
"run",
"--empty",
"--addr=:6660",
])
.start()
.await?;
let host = container.get_host().await?;
let port = container.get_host_port_ipv4(6660).await?;
let url = format!("{host}:{port}");
// assert_eq!(url, "127.0.0.1:6660");
Ok(())
}
}
And:
main.rs
:
#[tokio::test]
async fn test_fauxrpc() -> Result<(), Box<dyn std::error::Error + 'static>> {
let port: u16 = 6660;
let container = FauxRPC::default()
.with_cmd(vec![
"run",
"--empty",
format!("--addr=:{port}").as_str(),
])
.start()
.await?;
let host = container.get_host().await?;
let port = container.get_host_port_ipv4(port).await?;
let url = format!("http://{host}:{port}");
let uri: tonic::transport::Uri = url
.parse()
.expect("invalid gRPC endpoint");
let endpoint = Channel::builder(uri);
let channel = endpoint.connect()
.await
.expect("unable to connect");
let file_descriptor_set =
FileDescriptorSet::decode(FILE_DESCRIPTOR_SET)
.expect("invalid file descriptor set");
let _r = container
.image()
.add_file_descriptor(channel.clone(), file_descriptor_set)
.await;
println!("{:?}", _r);
let b = HealthCheckResponse {
status: health_check_response::ServingStatus::Serving as i32,
}
.encode_to_vec();
let _r = container.image().add_stub(channel.clone(), b).await;
println!("{:?}", _r);
let mut health_client = HealthClient::new(channel.clone());
let rqst = tonic::Request::new(HealthCheckRequest {
service: "".to_owned(),
});
let resp = health_client.check(rqst).await?;
let msg = resp.into_inner();
assert_eq!(
msg.status,
health_check_response::ServingStatus::Serving as i32
);
Ok(())
}
That’s all!