PyPi Transparency Client (Rust)

- 3 minutes read - 578 words

I’ve finally being able to hack my way through to a working Rust gRPC client (for PyPi Transparency).

It’s not very good: poorly structured, hacky etc. but it serves the purpose of giving me a foothold into Rust development so that I can evolve it as I learn the language and its practices.

There are several Rust crates (SDK) for gRPC. There’s no sanctioned SDK for Rust on grpc.io.

I chose stepancheg’s grpc-rust because it’s a pure Rust implementation (not built atop the C implementation).

I hacked from the library’s examples to something that works.

Here’s the project structure:

.
├── Cargo.lock
├── Cargo.toml
├── protos
│   ├── package.proto
│   └── pypi_transparency.proto
├── README.md
├── src
│   ├── main.rs
│   ├── package.rs
│   ├── pypi_transparency_grpc.rs
│   └── pypi_transparency.rs
└── target
    ├── debug
    └── rls

NB I’d like to structure it so that the proto files are kept separately and the protos directory becomes a module containing the generated files. Currently the generated sources are alongside main.rs in ./src

Cargo.toml:

[package]
name = "pypi-transparency-client-rust"
version = "0.0.1"
authors = ["Daz Wilkin <...>"]
edition = "2018"

[dependencies]
grpc            = "0.6.1"
protobuf        = "2.8.1"
futures         = "0.1.29"
futures-cpupool = "0.1.8"
protoc-rust-grpc = "0.6.1"

I simplified the service’s protos, removing the Google API annotations and the Go and Java options.

I’m aware that I should be able to have a distinct build process (build.rs) but, currently, I’m swapping the protoc generation in when needed:

protoc_rust_grpc::run(protoc_rust_grpc::Args {
    out_dir: "src",
    includes: &["./"],
    input: &["protos/pypi_transparency.proto", "protos/package.proto"],
    rust_protobuf: true,
    ..Default::default()
}).expect("protoc-rust-grpc");

Then, simply:

extern crate futures;
extern crate futures_cpupool;
extern crate grpc;
extern crate protobuf;

extern crate protoc_rust_grpc;

pub mod package;
pub mod pypi_transparency;
pub mod pypi_transparency_grpc;

use grpc::ClientConf;
use grpc::ClientStubExt;

use package::*;
use pypi_transparency::*;
use pypi_transparency_grpc::*;

fn main() {
    let host = "localhost";
    let port = 50051;

    let client = PyPiTransparencyClient::new_plain(host, port, ClientConf::new()).expect("client");

    let mut p = Package::new();
    p.set_name("grpcio".to_string());
    p.set_version("1.23.0".to_string());
    p.set_filename("grpcio-1.23.0-cp37-cp37m-manylinux1_x86_64.whl".to_string());
    p.set_url("https://files.pythonhosted.org/packages/e5/27/1f908ebb99c8d48a5ba4eb9d7997f5633b920d98fe712f67aaa0663f1307/grpcio-1.23.0-cp37-cp37m-manylinux1_x86_64.whl".to_string());

    // Create a Vector to hold the SHA-256 bytes but not actually representing any
    let v: Vec<u8> = Vec::new();
    p.set_sha256(v);

    let mut rqst = AddPackageRequest::new();

    // 'p' is moved (not borrowed) by 'add_package'
    rqst.set_package(p);
    let resp = client.add_package(grpc::RequestOptions::new(), rqst);
    println!("{:?}", resp.wait());
}

Improvements include:

  • Structuring the package into modules
  • Understanding the client’s creation (PyPiTransparencyClient::new_plain(...)) vs alternatives
  • Understanding (why the generated functions don’t borrow) Rust better
  • Understanding (how to decompose the resp) Rust better
  • Computing the SHA-256 and BLAKE2 hashes; the latter helps determine the package’s URL on PyPi

Still, I can build the above and run it:

cargo build && cargo run
   Compiling pypi-transparency-client-rust v0.0.1 (/home/dazwilkin/Projects/pypi-transparency-client-rust)
    Finished dev [unoptimized + debuginfo] target(s) in 2.00s
    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
     Running `target/debug/pypi-transparency-client-rust`
Ok((Metadata { entries: [MetadataEntry { key: MetadataKey { name: "content-type" }, value: b"application/grpc" }] }, ok: true, Metadata { entries: [] }))

And I can see that this includes the desired ok: true response so that’s good. As expected, the server sees the request:

pypi-transparency-server    | 2019/09/30 18:38:02 [server:AddPackage] Started
pypi-transparency-server    | 2019/09/30 18:38:02 [RepositoryClient:Open] Proxy configured
pypi-transparency-server    | 2019/09/30 18:38:02 [File:Open] Package: grpcio [1.23.0]
pypi-transparency-server    | 2019/09/30 18:38:02 [file:Filename] /server-cache/grpcio-1.23.0-cp37-cp37m-manylinux1_x86_64.whl
pypi-transparency-server    | 2019/09/30 18:38:02 [RepositoryClient:Open] Proxy contains package
pypi-transparency-server    | 2019/09/30 18:38:02 [hash:SHA256] Warning: calculation exhausts reader
pypi-transparency-server    | 2019/09/30 18:38:02 [server:add] Started
pypi-transparency-server    | 2019/09/30 18:38:02 [server:add] Already Exists
pypi-transparency-server    | 2019/09/30 18:38:02 [server:GetPackage] Started
pypi-transparency-server    | 2019/09/30 18:38:02 [server:get] Started
pypi-transparency-server    | 2019/09/30 18:38:02 [server:get] Leaf hash: 7288e362b47f3448ce770b0b319380a94a9f59023b33c03f19ad5e1bfdcd9f26
pypi-transparency-server    | 2019/09/30 18:38:02 [server:get] ok:true

Clearly (a lot) more work for me to do but… success! :-)