Using Rust to generate Kubernetes CRD

- 6 minutes read - 1234 words

For the first time, I chose Rust to solve a problem. Until this, I’ve been trying to use Rust to learn the language and to rewrite existing code. But, this problem led me to Rust because my other tools wouldn’t cut it.

The question was how to represent oneof fields in Kubernetes Custom Resource Definitions (CRDs).

CRDs use OpenAPI schema and the YAML that results can be challenging to grok.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: deploymentconfigs.example.com
spec:
  group: example.com
  names:
    categories: []
    kind: DeploymentConfig
    plural: deploymentconfigs
    shortNames: []
    singular: deploymentconfig
  scope: Namespaced
  versions:
  - additionalPrinterColumns: []
    name: v1alpha1
    schema:
      openAPIV3Schema:
        description: An example schema
        properties:
          spec:
            properties:
              deployment_strategy:
                oneOf:
                - required:
                  - rolling_update
                - required:
                  - recreate
                properties:
                  recreate:
                    properties:
                      something:
                        format: uint16
                        minimum: 0.0
                        type: integer
                    required:
                    - something
                    type: object
                  rolling_update:
                    properties:
                      max_surge:
                        format: uint16
                        minimum: 0.0
                        type: integer
                      max_unavailable:
                        format: uint16
                        minimum: 0.0
                        type: integer
                    required:
                    - max_surge
                    - max_unavailable
                    type: object
                type: object
            required:
            - deployment_strategy
            type: object
        required:
        - spec
        title: DeploymentConfig
        type: object
    served: true
    storage: true
    subresources: {}

I’ve developed several Kubernetes Operators using the Operator SDK in Go (which builds upon Kubebuilder).

But I worried that, because Go doesn’t (!?) directly represent union (oneOf) types. I have experienced this using Protobufs oneOf which is compiled to an interface type and uses (interface-)implementing types to represent the union’s types. I was also unsure (see below) how to represent this using Kubebuilder.

Interestingly, Gemini proposes the following which I will try:

// +kubebuilder:validation:Union
type DeploymentStrategy struct {
    // +unionDiscriminator
    Type string `json:"type"`

    // +unionMember
    RollingUpdate RollingUpdate `json:"rolling_update,omitempty"`

    // +unionMember
    Recreate Recreate `json:"recreate,omitempty"`
}

We want to able to represent the following variants:

kind: DeploymentConfig
apiVersion: example.com/v1alpha1
metadata:
  name: rolling-update
spec:
  deployment_strategy: # <--- Here's the union either rolling_update
    rolling_update:
      max_surge: 1
      max_unavailable: 1

Or:

kind: DeploymentConfig
apiVersion: example.com/v1alpha1
metadata:
  name: recreate
spec:
  deployment_strategy: # <--- Here's the union or recreate
    recreate:
      something: 1

I know from my adventures with Rust, that union types are permitted and I’ve experience using Rust with Protobuf oneOf types.

So I started with:

use kube_derive::CustomResource;
use schemars::JsonSchema;
use serde::{Serialize, Deserialize};

#[derive(CustomResource, Debug, Serialize, Deserialize, Default, Clone, JsonSchema)]
#[kube(group = "example.com", version = "v1alpha1", kind = "DeploymentConfig", namespaced)]
pub struct DeploymentConfigsSpec {
    pub deployment_strategy: DeploymentStrategy,
}

#[derive(Serialize,Deserialize,Clone,Debug,JsonSchema)]
pub enum DeploymentStrategy {
    RollingUpdate{
        max_surge: u16,
        max_unavailable: u16,
    },
    Recreate{
        something: u16
    },
}
impl Default for DeploymentStrategy {
    fn default() -> Self {
        DeploymentStrategy::Recreate{
            something: 1,
        }
    }
}

It’s then possible to create the CRD from the Spec and serialize it as JSON:

use kube::core::CustomResourceExt;

fn main() {
    // CRD as YAML
    let document_spec = DeploymentConfig::crd();
    println!("{}",serde_yaml::to_string(&document_spec).unwrap());
}

Yields:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: deploymentconfigs.example.com
spec:
  group: example.com
  names:
    categories: []
    kind: DeploymentConfig
    plural: deploymentconfigs
    shortNames: []
    singular: deploymentconfig
  scope: Namespaced
  versions:
  - additionalPrinterColumns: []
    name: v1alpha1
    schema:
      openAPIV3Schema:
        description: An example schema
        properties:
          spec:
            properties:
              deployment_strategy:
                oneOf:
                - required:
                  - RollingUpdate
                - required:
                  - Recreate
                properties:
                  Recreate:
                    properties:
                      something:
                        format: uint16
                        minimum: 0.0
                        type: integer
                    required:
                    - something
                    type: object
                  RollingUpdate:
                    properties:
                      max_surge:
                        format: uint16
                        minimum: 0.0
                        type: integer
                      max_unavailable:
                        format: uint16
                        minimum: 0.0
                        type: integer
                    required:
                    - max_surge
                    - max_unavailable
                    type: object
                type: object
            required:
            - deployment_strategy
            type: object
        required:
        - spec
        title: DeploymentConfig
        type: object
    served: true
    storage: true
    subresources: {}

You can then apply this to a (sacrificial) cluster and try to create some Custom Resources (CRs) against it. The examples above will suffice:

kubectl apply \
--filename=deploymentconfig.tests.yaml \
--namespace=${NAMESPACE}

error: error validating "/path/to/deploymentconfig.tests.yaml": error validating data: [ValidationError(DeploymentConfig.spec.deployment_strategy): unknown field "rolling_update" in com.example.v1alpha1.DeploymentConfig.spec.deployment_strategy, ValidationError(DeploymentConfig.spec.deployment_strategy): unknown field "recreate" in com.example.v1alpha1.DeploymentConfig.spec.deployment_strategy]; if you choose to ignore these errors, turn validation off with --validate=false

Uh-oh! The enum branches RollingUpdate and Recreate aren’t being correctly renamed when serialized to JSON.

This is straightforward to fix:

#[derive(Serialize,Deserialize,Clone,Debug,JsonSchema)]
pub enum DeploymentStrategy {
    // Add Serde rename
    #[serde(rename="rolling_update")]
    RollingUpdate{
        max_surge: u16,
        max_unavailable: u16,
    },
    // Add Serde rename
    #[serde(rename="recreate")]
    Recreate{
        something: u16
    },
}
impl Default for DeploymentStrategy {
    fn default() -> Self {
        DeploymentStrategy::Recreate{
            something: 1,
        }
    }
}

Rerun to regenerate the CRD YAML:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: deploymentconfigs.example.com
spec:
  group: example.com
  names:
    categories: []
    kind: DeploymentConfig
    plural: deploymentconfigs
    shortNames: []
    singular: deploymentconfig
  scope: Namespaced
  versions:
  - additionalPrinterColumns: []
    name: v1alpha1
    schema:
      openAPIV3Schema:
        description: An example schema
        properties:
          spec:
            properties:
              deployment_strategy:
                oneOf:
                - required:
                  - rolling_update # Correct
                - required:
                  - recreate       # Correct
                properties:
                  recreate:
                    properties:
                      something:
                        format: uint16
                        minimum: 0.0
                        type: integer
                    required:
                    - something
                    type: object
                  rolling_update:
                    properties:
                      max_surge:
                        format: uint16
                        minimum: 0.0
                        type: integer
                      max_unavailable:
                        format: uint16
                        minimum: 0.0
                        type: integer
                    required:
                    - max_surge
                    - max_unavailable
                    type: object
                type: object
            required:
            - deployment_strategy
            type: object
        required:
        - spec
        title: DeploymentConfig
        type: object
    served: true
    storage: true
    subresources: {}

Reapply the CRD to the cluster and then reapply the tests:

kubectl apply \
--filename=${PWD}/deploymentconfig.tests.yaml \
--namespace=${NAMESPACE}

deploymentconfig.example.com/rolling-update created
deploymentconfig.example.com/recreate created
The DeploymentConfig "bothof" is invalid: <nil>: Invalid value: "": "spec.deployment_strategy" must validate one and only one schema (oneOf). Found 2 valid alternatives
dazwilkin@hades-canyon:~/Projects/stackoverflow/78523396/rust (master)$

The last test fails intentionally:

kind: DeploymentConfig
apiVersion: example.com/v1alpha1
metadata:
  name: bothof
spec:
  deployment_strategy:
    rolling_update:
      max_surge: 1
      max_unavailable: 1
    recreate:
      something: 1

Because deployment_strategy is oneOf rolling_update or recreate, it should not contain both and so this Custom Resource’s creation fails.

So, as far as the CRD generation works, the Rust code is sufficient.

But, there’s another problem.

It’s possible to create the Custom Resources in Rust too:

fn main() {
    // CRD as YAML
    let document_spec = DeploymentConfig::crd();
    println!("{}",serde_yaml::to_string(&document_spec).unwrap());

    // CR with RollingUpdate as YAML
    let d1 = DeploymentConfigsSpec{
        deployment_strategy: DeploymentStrategy::RollingUpdate{
            max_surge:1,
            max_unavailable:1,
        },
    };
    println!("{}",serde_yaml::to_string(&d1).unwrap());

    // CR with Recreate as YAML
    let d2 = DeploymentConfigsSpec{
        deployment_strategy:DeploymentStrategy::Recreate{
            something: 1,
        }
    };
    println!("{}",serde_yaml::to_string(&d2).unwrap());
}

This yields (incorrectly):

deployment_strategy: !rolling_update
  max_surge: 1
  max_unavailable: 1

deployment_strategy: !recreate
  something: 1

The solution I found for this problem is the following. As always I’m interested in better ways to do this:

#[derive(Deserialize,Clone,Debug,JsonSchema)]
pub enum DeploymentStrategy {
    #[serde(rename="rolling_update")]
    RollingUpdate{
        max_surge: u16,
        max_unavailable: u16,
    },
    #[serde(rename="recreate")]
    Recreate{
        something: u16
    },
}
impl Default for DeploymentStrategy {
    fn default() -> Self {
        DeploymentStrategy::Recreate{
            something: 1,
        }
    }
}
impl Serialize for DeploymentStrategy {
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
        where
            S: serde::Serializer {
        match self {
            DeploymentStrategy::RollingUpdate{
                max_surge,
                max_unavailable
            } => {
                let rolling_update = RollingUpdate {
                    max_surge: *max_surge,
                    max_unavailable: *max_unavailable,
                };
                let mut map = std::collections::BTreeMap::new();
                map.insert("rolling_update", rolling_update);
                map.serialize(serializer)
            }
            DeploymentStrategy::Recreate{ 
                something 
            } => {
                let recreate = Recreate {
                    something: *something,
                };
                let mut map = std::collections::BTreeMap::new();
                map.insert("recreate", recreate);
                map.serialize(serializer)
            },
        }
    }
}

#[derive(Serialize,Deserialize,Clone,Debug,JsonSchema)]
pub struct RollingUpdate {
    max_surge: u16,
    max_unavailable: u16,
}

#[derive(Serialize,Deserialize,Clone,Debug,JsonSchema)]
pub struct Recreate{
    pub something: u16
}

This is slightly awkward because it duplicates (which gives me pause) the type definitions of RollingUpdate and Recreate. The first time these are defined as variants of the enum. The second time they’re defined explicitly as struct types.

Serialize is implemented explicity for the enum (DeploymentStrategy) and it matches on the variants and replaces the enum variant with the struct type and adds (!) the resulting struct to a BTreeMap. The map is then serialized.

This code yields the same CRD but yields expected YAML for Custom Resources:

deployment_strategy:
  rolling_update:
    max_surge: 1
    max_unavailable: 1

deployment_strategy:
  recreate:
    something: 1

Don’t forget to delete the CRD, Namespace and any CRs before you finish:

NAMES=$(\
  kubectl get deploymentconfigs \
  --namespace=${NAMESPACE} \
  --output=name\
)
for NAME in ${NAMES}
do
  kubectl delete ${NAME} \
  --namespace=${NAMESPACE}
done

kubectl delete crd/deploymentconfigs.example.com

That’s all!