gRPC and Protobuf: Implementing a partial update API

I came across a need to build a gRPC API endpoint to perform partial updates on a resource and went against the recommendation to use field masks made in:

• The Protobuf documentation
• This Netflix tech blog article

Here's why.

What are partial updates?

A partial update allows clients to specify which fields they want to update on a resource and the values to change them to. The alternative would be to update and re-write the entire resource where you would need to pass in the value of every field.

The advantages I saw in using partial updates over full updates in my use-case included:

1. The client doesn't need to make an additional query beforehand to fetch the current state of the other fields. Updates would also have to be synchronized so that another request doesn't change the resource before the current request completes its update.
2. The server doesn't need to unnecessarily update the whole database row including fields it hasn't changed

Dealing with nullable fields

The main challenge was handling nullable fields.

The gRPC API schema was defined with protobuf (proto 3) which doesn't have the concept of a nullable field.

The closest concept is fields being optional where they can be in one of two states: set or unset.

As a first step, we can say that if we've set a field in the request we want to update it. If we haven't set a field in the request, we don't want to update it.

However, this breaks down with nullable fields as we need to be able to differentiate setting fields as null or non-null values. This means we need to represent fields in one of three states:

1. Set to a non-null value
2. Set to null
3. Unset

❌ Option #1: Using field masks

A field mask is an additional field you pass in to declare the set of fields you want to affect with the request.

• In a partial update API, this could declare the set of fields you want to update on a resource
• In a get/fetch API, this could declare the set of fields to fetch if you don't want to return an entire resource

The advantage of this approach is that it only requires a minimal change to the Protobuf schema with an extra field.

It can also be added in a backward and forward compatible way.

• Backward compatible: If the server reads an older message without a field mask, the server can safely assume that all fields should be taken
• Forward compatible: If an old version of the server (which doesn't know how to parse the field mask) reads a message with a field mask, it can just ignore it as long as the field mask is optional

The main issue with this approach which resulted in this solution not being chosen is that field masks use field names to specify if they should be included or not.

This adds complexity because field names become significant:

1. Field names cannot be simply renamed without considering backward and forward compatibility. This goes against the typical assumption when using Protobuf that changing field names is safe (because field names aren't usually included in the binary encoded Protobuf message sent over the wire).
2. Server and client code need to directly reference field paths using strings. For example, you would reference a top-level field by the field name directly, or the entire path for a field nested within messages (e.g. some_request_message.some_nested_field.another_nested_field.the_field_we_want).

Trying to avoid this unnecessary complexity led to the following approach.

✅ Option #2: Defining our own nullable wrapper messages

We can define reusable wrapper messages that can represent all three states:

1. A wrapper for native types (e.g.string)

message NullableStringValue {
    oneof setAs {
        google.protobuf.NullValue null = 1;
        string value = 2;
    }
}

2. A wrapper for repeated fields

message NullableRepeatedStringValue {
    message RepeatedStringValue {
        repeated string value = 1;
    }
    oneof setAs {
        google.protobuf.NullValue null = 1;
        RepeatedStringValue value = 2;
    }
}

Note that we need to represent repeated fields in their own wrapper message anyway to mark them as optional. That is because combining optional directly with repeated doesn't work (e.g. optional repeated string ids = 2).

We can then reference the wrappers in our partial update API like this:

message PartialUpdateApiRequest {
    optional NullableStringValue name = 1;
    optional NullableRepeatedStringValue ids = 2;
}

Why this works

1. We can differentiate between 'set' and 'unset' because optional fields provide a .has<FieldName>() method to check this
2. We can differentiate between 'set as non-null' or 'set as null' by evaluating the value of setAs which indicates a value can be 'one of' a null or non-null value (see example Kotlin code below)

fun NullableStringValue.get(): String? =
    when (this.setAsCase) {
        NullableStringValue.SetAsCase.NULL -> null
        NullableStringValue.SetAsCase.VALUE -> value
        null,
        NullableStringValue.SetAsCase.SETAS_NOT_SET -> throw IllegalArgumentException("must be explicitly set to value or null")
    }
}