Firestore Golang Timestamps & Merging

May 6, 2021 - 6 minutes read - 1206 words

I’m using Google’s Golang SDK for Firestore. The experience is excellent and I’m quickly becoming a fan of Firestore. However, as a Golang Firestore developer, I’m feeling less loved and some of the concepts in the database were causing me a conundrum.

I’m still not entirely certain that I have Timestamps nailed but… I learned an important lesson on the auto-creation of Timestamps in documents and how to retain these values.

Here’s a simple Golang struct with firestore tags:

type Dog struct {
    Name    string    `firestore:"name"`
    Age     uint8     `firestore:"age"`
    Created time.Time `firestore:"created,serverTimestamp"`
    Updated time.Time `firestore:"updated,omitempty"`
}

NOTE It’s good practice to include description tags too but these are omitted here for succinctness

This approach provides some Golang syntactic sugar for working with Firestore documents. Conceptually, I think (!?), it’s good to consider Firestore documents as JSON documents. This model also helps explain some of what follows.

The tag serverTimestamp indicates that the Golang field (Firestore document property) should be replaced by the service’s timestamp when the document is created.

Let’s test that:

freddie := &Dog{
    Name: "Freddie",
    Age:  2,
}
log.Printf("%+v", freddie)

ref := client.Collection("dogs").Doc("freddie")

if _, err := ref.Set(ctx, freddie); err != nil {
    log.Fatal(err)
}

snap, err := ref.Get(ctx)
if err != nil {
    log.Fatal(err)
}

d := &Dog{}
if err := snap.DataTo(d); err != nil {
    log.Fatal(err)
}

log.Printf("%+v", d)

NOTE DocumentRefs do not consume network bandwidth but it pays to not unnecessarily recompute them.

This yields:

{
    Name:Freddie
    Age:2
    Created:0001-01-01 00:00:00 +0000 UTC
    Updated:0001-01-01 00:00:00 +0000 UTC
}
{
    Name:Freddie
    Age:2
    Created:2021-05-06 16:33:02.768 +0000 UTC
    Updated:0001-01-01 00:00:00 +0000 UTC
}

NOTE pretty-printed to aid comprehension

NOTE The Firestore document does not contain an Updated property but the Golang struct defaults this value to Golang’s zero time (see IsZero)

But, great, the Created field is set.

Let’s update the retrieved (!) record, Set then Get it again:

freddie.Name = "Frederik Jack"
freddie.Updated = time.Now()

if _, err := ref.Set(ctx, freddie); err != nil {
    log.Fatal(err)
}

snap, err := ref.Get(ctx)
if err != nil {
    log.Fatal(err)
}

d := &Dog{}
if err := snap.DataTo(d); err != nil {
    log.Fatal(err)
}

log.Printf("%+v", d)

Yields:

{
    Name:Frederik Jack
    Age:2
    Created:2021-05-06 16:33:02.768 +0000 UTC
    Updated:2021-05-06 16:37:39.571753 +0000 UTC
}

All good… The Name and Updated fields were updated.

BUT, this probably isn’t what we’d customarily do. In the above code, we retrieve the document, mutate it and then persist it.

We’d probably do this in practice:

freddie := &Dog{
    Name: "Freddie",
    Age:  2,
}
log.Printf("%+v", freddie)

ref := client.Collection("dogs").Doc("freddie")

if _, err := ref.Set(ctx, freddie); err != nil {
    log.Fatal(err)
}

freddie.Name = "Frederik Jack"
freddie.Updated = time.Now()

if _, err := ref.Set(ctx, freddie); err != nil {
    log.Fatal(err)
}

snap, err := ref.Get(ctx)
if err != nil {
    log.Fatal(err)
}

d := &Dog{}
if err := snap.DataTo(d); err != nil {
    log.Fatal(err)
}

log.Printf("%+v", d)

The difference here is that we treat our local object as the source of truth, create it, persist it, change it and then persist it. We’re not retrieving the document after each Set because we assume (incorrectly) that it is the source of truth. It isn’t!

Case in point, our local copy doesn’t include a value for Created timestamp.

{
    Name:Frederik
    Age:2
    Created:2021-05-06 10:01:48
}
{
    Name:Frederik Jack
    Age:2
    Created:2021-05-06 10:03:31
    Updated:2021-05-06 10:03:30
}

There are a bunch of (subtle) changes here:

Frederik to Frederik Jack, good… expected
Updated is created and set , good… expected
Created changes from 10:01:48 to 10:03:31
Updated is earlier than Created

Why is this?

Returning to the idea that Firestore documents are JSON documents, each time we Set, we’re rewriting the JSON document (provided to Set) to Firestore. When the updated freddie Golang struct is persisted a second time:

Name is included (and has changed)
Age is included (and hasn’t changed)
Created is included but is unset|nil (IsZero)
Updated is included and probably includes a non-nil timestamp

The first important detail is that, using Set as above, the document that’s written to Firestore only includes fields that are included in the Golang struct.

The second important detail is that, because Created is nil (still) when it’s written to Firestore, the service applies serverTimestamp again (!) and effectively updates the Created timestamp. We don’t want it to do this!

NOTE Updated is earlier than Created because we set Updated to time.Now() before persisting the record when the server updates the Created timestamp.

What’s the solution?

If I understand this correctly, the solution is that, unless we’re always Seting every field and we’re not using serverTimestamp, we must include SetOption when using Set.

The first time a document is created with Set, there’s no document in Firestore and so the concept of merging is moot. Firestore doesn’t balk at include merge options but, I think they’re redundant at this point:

freddie := &Dog{
    Name: "Freddie",
    Age:  2,
}
log.Printf("%+v", freddie)

ref := client.Collection("dogs").Doc("freddie")

if _, err := ref.Set(ctx, freddie); err != nil {
    log.Fatal(err)
}

All subsequent (!) times, we update (this doesn’t apply to deletes or gets) an (existing!) document, we must consider applying merge options.

Specifically, if we’ve a Created field regardless of whether we’re using serverTimestamp to set it initally or setting it in our code, we must remember to never overwrite this value and to do so we must exclude it from the merge options. This has the effect of leaving it unchanged.

However, for all the fields that we want to update, we must include these field names in the merge options so as to only overwrite these fields (and leave everything else unchanged).

This brings me to the final gotcha with the SDK. I was using SetOption incorrectly and was receiving errors. In the following code, note the very specific way in which fields must be represented:

freddie.Name = "Frederik Jack"
freddie.Updated = time.Now()

opt := firestore.Merge(
    []string{"name"},
    []string{"age"},
    []string{"updated"},
)

if _, err := ref.Set(ctx, freddie, opt); err != nil {
    log.Fatal(err)
}

snap, err := ref.Get(ctx)
if err != nil {
    log.Fatal(err)
}

d := &Dog{}
if err := snap.DataTo(d); err != nil {
    log.Fatal(err)
}

log.Printf("%+v", d)

NOTE firestore.Merge is variadic and takes an []string{} for each field. There appears to be no simpler mechanism (i.e. to provide just a string) and this complexity appears to result from the need to be able to support complex JSON documents with nested structs in which it’s necessary to path to a field.

NOTE Created is not included in Merge(...) because we don’t plan to overwrite it; it won’t be merged (just left)

This yields:

{
    Name:Freddie
    Age:2
}

{
    Name:Freddie
    Age:2
    Created:2021-05-06 17:32:39.935 +0000 UTC
}
{
    Name:Frederik Jack
    Age:2
    Created:2021-05-06 17:32:39.935 +0000 UTC
    Updated:2021-05-06 17:32:50.626336 +0000 UTC
}

NOTE I’ve removed the IsZero timestamps as these aren’t recorded in Firestore

What are we observing from the above?

Once Created has been assigned, we don’t overwrite it and it holds the original datetime
Because Created is not in the merge list, even if it changes in the code, the update value, not being in the merge list, won’t be persisted to Firestore.
We must manually update Updated

See this issue firestore: Set method zeros out serverTimestamp fields unexpectedly #1171 for more details including Google’s engineers’ explanation of this behavior.