Learning Snippets

Webhooks in Kubernetes

• k8s

Webhooks are a way to “intercept” requests to the k8s apiserver on their way from the apiserver’s HTTP handler to etcd persistency. There are two types of webhooks:

Like the name already says, mutating webhooks will modify submitted resources before they are persisted (e.g. filling in default values etc.). Validating webhooks will validate field values of a submitted resource but not change them. Conversion webhooks will convert between different versions of the same CRD for compatibility.

Serving Webhooks

Webhooks need to be served by a HTTP server so that the k8s apiserver can send requests to it. For admission webhooks, the POST requests have a AdmissionReview struct serialized to JSON as body. For conversion webhooks, a ConversionReview struct is used. Similarly, response bodies serialized the same structs to JSON.

The webhook server is a k8s Deployment, with a service wrapping it. For admission webhooks, the service is referenced in an admission webhook configuration from the admissionregistration.k8s.io API. This configuration also specifies invocation rules e.g. for which CRUD operation, which api groups, versions or resources an admission webhook should be used. The following configuration taken from the k8s docs is an example:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
...
webhooks:
- name: my-webhook.example.com
  clientConfig:
    caBundle: "Ci0tLS0tQk...<base64-encoded PEM bundle containing the CA that signed the webhook's serving certificate>...tLS0K"
    service:
      namespace: my-service-namespace
      name: my-service-name
      path: /my-path
      port: 1234
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: ["apps"]
    apiVersions: ["v1", "v1beta1"]
    resources: ["deployments", "replicasets"]
    scope: "Namespaced"
  ...

For conversion webhooks, the service is directly referenced in the CRD and there is no separate configuration file. This is described in the next section.

Multiversion support of CRDs and Webhooks

When several versions are present, some things need to be specified in the CRD yaml:

  1. Is custom logic necessary for conversion between different CRD versions? If there are schema changes from one version to the other, conversion webhooks should be used to implement conversion logic.
  2. In which version are the resources persisted? When a resource is created, it is persisted in a version selected as the storage version. This means that the stored version of the resource can be different from the one requested via kubectl get resource.version.group (when no version is specified, e.g. in kubectl get resource, kubectl will use the latest version by default). Reading from storage will not automatically changed the stored version and when a new storage version is set, there is no automatic process that converts the stored resources to the new storage version. The conversion will only happen when a stored resource is updated.
  3. Which versions are supported? As new versions are added, older versions can be disabled.

The following yaml taken from the k8s docs is an example CRD:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: crontabs.example.com
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: example.com
  # list of versions supported by this CustomResourceDefinition
  versions:
  - name: v1beta1
    # Each version can be enabled/disabled by Served flag.
    served: true
    # One and only one version must be marked as the storage version.
    storage: true
    # A schema is required
    schema:
      openAPIV3Schema:
        type: object
        properties:
          ...
  - name: v1
    served: true
    storage: false
    schema:
      openAPIV3Schema:
        type: object
        properties:
          ...
  # The conversion section is introduced in Kubernetes 1.13+ with a default value of
  # None conversion (strategy sub-field set to None).
  conversion:
    # None conversion assumes the same schema for all versions and only sets the apiVersion
    # field of custom resources to the proper value
    strategy: None
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: crontabs
    # singular name to be used as an alias on the CLI and for display
    singular: crontab
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: CronTab
    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - ct

Using Webhooks with Controller-Runtime

Let us now look at the inclusion of webhooks in controller-runtime. The starting point here is again the manager. When a manager is created, you can pass webhook-relevant config such as the port, host and certificate directory of the webhook server. After the manager is created, you add the webhook server to it:

import (
  ctrl "sigs.k8s.io/controller-runtime"
)
mgr, _ := ctrl.NewManager(<config>)
ctrl.NewWebhookManagedBy(mgr).
  For(<your crd>).
  Complete()

Under the hood, NewWebhookManagedBy() will return a builder that checks if your CRD implements the Defaulter (mutating webhook) and/or Validator (validating webhook) interfaces.

/* source code from https://github.com/kubernetes-sigs/controller-runtime/blob/release-0.7/pkg/webhook/admission/defaulter.go */
type Defaulter interface {
	runtime.Object
	Default()
}

/* source code from https://github.com/kubernetes-sigs/controller-runtime/blob/release-0.7/pkg/webhook/admission/validator.go */
type Validator interface {
	runtime.Object
	ValidateCreate() error
	ValidateUpdate(old runtime.Object) error
	ValidateDelete() error
}

If it does, it generates paths for your webhooks and registers the implemented interface functions as handlers to a http.ServeMux HTTP request multiplexer.

For conversion webhooks, the builder checks if one of your CRD versions implements the Hub interface and if all other versions implement the Convertible interface.

/* source code from https://github.com/kubernetes-sigs/controller-runtime/blob/release-0.7/pkg/conversion/conversion.go */
type Convertible interface {
	runtime.Object
	ConvertTo(dst Hub) error
	ConvertFrom(src Hub) error
}
type Hub interface {
	runtime.Object
	Hub()
}

Hubs and convertibles (or spokes) are a concept used by controller-runtime to reduce the combinatorial complexity of conversions. Imagine we have a three CRD versions: v1, v2, v3. Assume our hub version is v2, and a resource is persisted as v1. We now query the resource, but as v3 with kubectl get myresource.v3.mygroup. What happens is that v1 from the storage is converted into the hub version v2 and from there to the target version v3. To do so, we need to define how v1 and v3 can be converted to and from v2, and that is what the interface Convertible does. The introduction of a hub version eliminates the need to write logic for all possible version pairs.

After all the configuration is done, the manager starts up the webhook server - the webhook deployment in controller-runtime is part of the manager deployment. All that is left to do is to create a service yaml and to refer to the service in the admission webhook configuration or CRD yaml. If you use kubebuilder, you get the yaml files automatically.

Further Reading