Trillian Map Mode

September 8, 2020 - 4 minutes read - 808 words

Chatting with one of Google’s Trillian team, I was encouraged to explore Trillian’s Map Mode. The following is the result of some spelunking through this unfamiliar cave. I can’t provide any guarantee that this usage is correct or sufficient.

Here’s the repo: https://github.com/DazWilkin/go-trillian-map

I’ve written about Trillian Log Mode elsewhere.

I uncovered use of Trillian Map Mode through Trillian’s integration tests. I’m unclear on the distinction between TrillianMapClient and TrillianMapWriteClient but the latter served most of my needs.

Trillian includes sample deployment examples for Docker Compose and Kubernetes. This repo includes both.

First, we need a Map.

GRPC="50051"
MAPID=$(\
go run github.com/google/trillian/cmd/createtree \
--admin_server=:${GRPC} \
--tree_type=MAP \
--hash_strategy=CONIKS_SHA512_256) && echo ${MAPID}

NOTE we must override defaults values for --tree_type and --hash_strategy to create a Map

The example is trivial but demonstrates writing and getting data from the Map.

One (of the) aspect(s) that I don’t grok with Maps is the concept of revisions. I suspect (and need to determine whether) I can try getting leaves from any revision (see Issue #2)

The example uses a trivial map of key-value pairs that are used to create Leaves for the Map.

var Examples = map[string]string{
	"Freddie":  "Border Collie",
	"Noisette": "Australian Shepherd",
	"Artie":    "Weimeraner",
	"Louie":    "Formosan Mountain Dog",
	"Luna":     "Yorkiepoo",
	"Bertie":   "Cockapoo",
	"Mr Magoo": "Scottish Terrier",
}

The Add section batches the leaves into an []*trillian.MapLeaf that is added in a single WriteLeaves request (using a single revision).

MapLeaf comprise an Index and a LeafValue. Both are []byte and Index is a hash value in the Trillian uses. In the following, key and value correspond to entries in the above map:

hasher := sha256.New()
hasher.Write([]byte(key))
index := hasher.Sum(nil)

leaf := &trillian.MapLeaf{
	Index:     index,
	LeafValue: []byte(value),
}

An evolution of the code that would mainly be for intellectual curiosity, would be to replace the simple string values with a protobuf or MessagePack marshaled value (see Issues #3)

Then to write the leaves, simply:

func (c *Client) Add(ctx context.Context, leaves []*trillian.MapLeaf, revision int64) error {
    log.Print("[Client:Add] Entered")
	rqst := &trillian.WriteMapLeavesRequest{
		MapId:          c.mapID,
		Leaves:         leaves,
		ExpectRevision: revision,
	}
	resp, err := c.client.WriteLeaves(ctx, rqst)
	if err != nil {
		return err
	}

	log.Printf("[Client:Add] %+v", resp)
	return nil
}

The Get section gets each leaf in turn for the revision. As before, we hash the key value to create the index and then create the request with a singular index. We iterate over the leaves in the response and print them:

indexes := [][]byte{
	index,
}
leaves, err := client.Get(ctx, indexes, rev)
if err != nil {
	log.Fatal(err)
}
for i, leaf := range leaves {
	log.Printf("[main:get] Leaf #%02d: %s", i, toString(leaf))
}

The leaf retrieval uses Trillian’s GetMapLeavesByRevisionRequest:

func (c *Client) Get(ctx context.Context, indexes [][]byte, revision int64) ([]*trillian.MapLeaf, error) {
	log.Print("[Client:Get] Entered")
	rqst := &trillian.GetMapLeavesByRevisionRequest{
		MapId:    c.mapID,
		Index:    indexes,
		Revision: revision,
	}
	resp, err := c.client.GetLeavesByRevision(ctx, rqst)
	if err != nil {
		return nil, err
	}

	log.Printf("[Client:Get] %+v", resp)
	return resp.GetLeaves(), nil
}

Running the server against the Kubernetes deployment involves determining the code’s (Kubernetes) Service endpoint:

ENDPOINT=$(\
  kubectl get service/map-server \
  --namespace=trillian \
  --output=jsonpath="{.spec.clusterIP}:{.spec.ports[?(@.name==\"grpc\")].port}"\
) && echo ${ENDPOINT}

Then invoking the binary:

go run ./cmd/server \
--tmap_endpoint=${ENDPOINT} \
--tmap_id=${MAPID} \
--tmap_rev=1

Yielding:

[main:add] Bertie
[main:add] Bertie: 2eed8f9879977565d3037e6363b505ff2ca35a2227083de91d198ebfcde76b3d
[main:add] Leaf #00: Index: 2eed8f9879977565d3037e6363b505ff2ca35a2227083de91d198ebfcde76b3d, Value: Cockapoo
[main:add] Mr Magoo
[main:add] Mr Magoo: cec7fa35beed1c12b44f1bab965fc47ebe8c590510a7eb774d10b440f68c8355
[main:add] Leaf #01: Index: cec7fa35beed1c12b44f1bab965fc47ebe8c590510a7eb774d10b440f68c8355, Value: Scottish Terrier
[main:add] Freddie
[main:add] Freddie: b146ae2a1fb1d772cbe9c16fd9ae30eb11ebef7c948829167bb04fae541aa5f5
[main:add] Leaf #02: Index: b146ae2a1fb1d772cbe9c16fd9ae30eb11ebef7c948829167bb04fae541aa5f5, Value: Border Collie
[main:add] Noisette
[main:add] Noisette: 12bb631807ea32865a0611e72a65e2b33ec9ad8b9600e9db4d29837481e45b38
[main:add] Leaf #03: Index: 12bb631807ea32865a0611e72a65e2b33ec9ad8b9600e9db4d29837481e45b38, Value: Australian Shepherd
[main:add] Artie
[main:add] Artie: d69c866405c582529c4c74fe474575a62c0db4fa8e5321f62fb197cbb7bc8fe7
[main:add] Leaf #04: Index: d69c866405c582529c4c74fe474575a62c0db4fa8e5321f62fb197cbb7bc8fe7, Value: Weimeraner
[main:add] Louie
[main:add] Louie: 7435d5755367c07fd592e3210fe0db424a9278b255faab470c96324809d1a773
[main:add] Leaf #05: Index: 7435d5755367c07fd592e3210fe0db424a9278b255faab470c96324809d1a773, Value: Formosan Mountain Dog
[main:add] Luna
[main:add] Luna: 9d77a24d0f4c91a2e968ca607e49dd3b15d5489f66a5bc81fa72803b32443419
[main:add] Leaf #06: Index: 9d77a24d0f4c91a2e968ca607e49dd3b15d5489f66a5bc81fa72803b32443419, Value: Yorkiepoo
[main:add] Add'ing
[Client:Add] Entered
[Client:Add] revision:7
[main:get] Freddie
[main:get] Freddie: b146ae2a1fb1d772cbe9c16fd9ae30eb11ebef7c948829167bb04fae541aa5f5
[main:get] Get'ing Freddie
[Client:Get] Entered
[Client:Get] leaves:{index:"\xb1F\xae*\x1f\xb1\xd7r\xcb\xe9\xc1oٮ0\xeb\x11\xeb\xef|\x94\x88)\x16{\xb0O\xaeT\x1a\xa5\xf5"  leaf_value:"Border Collie"}
[main:get] Leaf #00: Index: b146ae2a1fb1d772cbe9c16fd9ae30eb11ebef7c948829167bb04fae541aa5f5, Value: Border Collie
[main:get] Sleeping 1 second
[main:get] Noisette
[main:get] Noisette: 12bb631807ea32865a0611e72a65e2b33ec9ad8b9600e9db4d29837481e45b38
[main:get] Get'ing Noisette
[Client:Get] Entered
[Client:Get] leaves:{index:"\x12\xbbc\x18\x07\xea2\x86Z\x06\x11\xe7*e\xe2\xb3>ɭ\x8b\x96\x00\xe9\xdbM)\x83t\x81\xe4[8"  leaf_value:"Australian Shepherd"}
[main:get] Leaf #00: Index: 12bb631807ea32865a0611e72a65e2b33ec9ad8b9600e9db4d29837481e45b38, Value: Australian Shepherd
[main:get] Sleeping 1 second
[main:get] Artie
[main:get] Artie: d69c866405c582529c4c74fe474575a62c0db4fa8e5321f62fb197cbb7bc8fe7
[main:get] Get'ing Artie
[Client:Get] Entered
[Client:Get] leaves:{index:"֜\x86d\x05łR\x9cLt\xfeGEu\xa6,\r\xb4\xfa\x8eS!\xf6/\xb1\x97˷\xbc\x8f\xe7"  leaf_value:"Weimeraner"}
[main:get] Leaf #00: Index: d69c866405c582529c4c74fe474575a62c0db4fa8e5321f62fb197cbb7bc8fe7, Value: Weimeraner
[main:get] Sleeping 1 second
[main:get] Louie
[main:get] Louie: 7435d5755367c07fd592e3210fe0db424a9278b255faab470c96324809d1a773
[main:get] Get'ing Louie
[Client:Get] Entered
[Client:Get] leaves:{index:"t5\xd5uSg\xc0Ւ\xe3!\x0f\xe0\xdbBJ\x92x\xb2U\xfa\xabG\x0c\x962H\tѧs"  leaf_value:"Formosan Mountain Dog"}
[main:get] Leaf #00: Index: 7435d5755367c07fd592e3210fe0db424a9278b255faab470c96324809d1a773, Value: Formosan Mountain Dog
[main:get] Sleeping 1 second
[main:get] Luna
[main:get] Luna: 9d77a24d0f4c91a2e968ca607e49dd3b15d5489f66a5bc81fa72803b32443419
[main:get] Get'ing Luna
[Client:Get] Entered
[Client:Get] leaves:{index:"\x9dw\xa2M\x0fL\x91\xa2\xe9h\xca`~I\xdd;\x15\xd5H\x9ff\xa5\xbc\x81\xfar\x80;2D4\x19"  leaf_value:"Yorkiepoo"}
[main:get] Leaf #00: Index: 9d77a24d0f4c91a2e968ca607e49dd3b15d5489f66a5bc81fa72803b32443419, Value: Yorkiepoo
[main:get] Sleeping 1 second
[main:get] Bertie
[main:get] Bertie: 2eed8f9879977565d3037e6363b505ff2ca35a2227083de91d198ebfcde76b3d
[main:get] Get'ing Bertie
[Client:Get] Entered
[Client:Get] leaves:{index:".폘y\x97ue\xd3\x03~cc\xb5\x05\xff,\xa3Z\"'\x08=\xe9\x1d\x19\x8e\xbf\xcd\xe7k="  leaf_value:"Cockapoo"}
[main:get] Leaf #00: Index: 2eed8f9879977565d3037e6363b505ff2ca35a2227083de91d198ebfcde76b3d, Value: Cockapoo
[main:get] Sleeping 1 second
[main:get] Mr Magoo
[main:get] Mr Magoo: cec7fa35beed1c12b44f1bab965fc47ebe8c590510a7eb774d10b440f68c8355
[main:get] Get'ing Mr Magoo
[Client:Get] Entered
[Client:Get] leaves:{index:"\xce\xc7\xfa5\xbe\xed\x1c\x12\xb4O\x1b\xab\x96_\xc4~\xbe\x8cY\x05\x10\xa7\xebwM\x10\xb4@\xf6\x8c\x83U"  leaf_value:"Scottish Terrier"}
[main:get] Leaf #00: Index: cec7fa35beed1c12b44f1bab965fc47ebe8c590510a7eb774d10b440f68c8355, Value: Scottish Terrier
[main:get] Sleeping 1 second

That’s all (for now)!