This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Kubeadm

1: Audit policy
2: Auto-renewal of control plane certificates
3: Containerd metrics
4: DNS
5: Encryption At Rest
6: etcd
7: Extra API Server Certificate SANs
8: kube-proxy mode
9: Kubernetes Image Repository
10: Node registration configuration
11: Parallel Image Pulls

The customizations in this section are applicable to providers using kubeadm to bootstrap the cluster.

1 - Audit policy

Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself.

There are currently no configuration options for the Audit Policy customization and this customization will be automatically applied when the provider-specific cluster configuration patch is included in the ClusterClass.

2 - Auto-renewal of control plane certificates

autoRenewCertificates variable enables automatic renewal of control plane certificates by triggering a rollout of the control plane nodes when the certificates on the control plane machines are about to expire.

More information about certificate renewal: Automatically rotating certificates using Kubeadm Control Plane provider.

Example

To enable automatic certificate renewal use the following configuration, applicable to all CAPI providers supported by CAREN:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          controlPlane:
            autoRenewCertificates:
              daysBeforeExpiry: 30

Applying this configuration will result in the following configuration being applied:

KubeadmControlPlaneTemplate:

spec:
  template:
    spec:
      rolloutBefore:
        certificatesExpiryDays: 30

3 - Containerd metrics

Containerd exports metrics to a Prometheus endpoint. The metrics cover containerd itself, its plugins, e.g. CRI, and information about the containers managed by containerd.

There are currently no configuration options for metrics, and this customization will be automatically applied when the provider-specific cluster configuration patch is included in the ClusterClass.

4 - DNS

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

The DNS configuration can then be manipulated via the cluster variables. If the dns property is not specified, then the customization will be skipped.

CoreDNS

The CoreDNS configuration can then be manipulated via the cluster variables. If the dns.coreDNS property is not specified, then the customization will be skipped.

Example

The CoreDNS version can be updated automatically. To do this, set coreDNS to an empty object:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          dns:
            coreDNS: {}

Applying this configuration will result in the following value being set, with the version of the CoreDNS image being set based on the cluster's Kubernetes version:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      dns:
        imageTag: "v1.11.3"

To change the repository and tag for the container image for the CoreDNS pod, specify the following configuration:

Note do not include "coredns" in the repository, kubeadm already appends it.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          dns:
            coreDNS:
              image:
                repository: my-registry.io/my-org/my-repo
                tag: "v1.11.3_custom.0"

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      dns:
        imageRepository: "my-registry.io/my-org/my-repo"
        imageTag: "v1.11.3_custom.0"

5 - Encryption At Rest

encryptionAtRest variable enables encrypting kubernetes resources at rest using provided encryption provider. When this variable is set, kuberntetes secrets and configmaps are encrypted before writing them at etcd.

If the encryptionAtRest property is not specified, then the customization will be skipped. The secrets and configmaps will not be stored as encrypted in etcd.

We support following encryption providers

aescbc
secretbox

More information about encryption at-rest: Encrypting Confidential Data at Rest

Example

To encrypt configmaps and secrets kubernetes resources using aescbc encryption provider:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          encryptionAtRest:
            providers:
              - aescbc: {}

Applying this configuration will result in

<CLUSTER_NAME>-encryption-config secret generated.

A secret key for the encryption provider is generated and stored in <CLUSTER_NAME>-encryption-config secret. The APIServer will be configured to use the secret key to encrypt secrets and configmaps kubernetes resources before writing them to etcd. When reading resources from etcd, encryption provider that matches the stored data attempts in order to decrypt the data. CAREN currently does not rotate the key once it generated.

Configure APIServer with encryption configuration:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
          encryption-provider-config: /etc/kubernetes/pki/encryptionconfig.yaml
  files:
    - contentFrom:
        secret:
          key: config
          name: <CLUSTER_NAME>-encryption-config
      path: /etc/kubernetes/pki/encryptionconfig.yaml
      permissions: "0640"

6 - etcd

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

The etcd configuration can then be manipulated via the cluster variables. If the etcd property is not specified, then the customization will be skipped.

Example

To change the repository and tag for the container image for the etcd pod, specify the following configuration:

Note do not include "etcd" in the repository, kubeadm already appends it.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          etcd:
            image:
              repository: my-registry.io/my-org/my-repo
              tag: "v3.5.99_custom.0"

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      etcd:
        local:
          imageRepository: "my-registry.io/my-org/my-repo"
          imageTag: "v3.5.99_custom.0"

7 - Extra API Server Certificate SANs

If the API server can be accessed by alternative DNS addresses then setting additional SANs on the API server certificate is necessary in order for clients to successfully validate the API server certificate.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

To add extra SANs to the API server certificate, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          extraAPIServerCertSANs:
            - a.b.c.example.com
            - d.e.f.example.com

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        certSANs:
          - a.b.c.example.com
          - d.e.f.example.com

8 - kube-proxy mode

This customization allows configuration of the kube-proxy proxy mode. Currently, only iptables and nftables modes are supported. By default, kube-proxy is enabled in iptables mode by kubeadm.

Examples

Enabling nftables kube-proxy mode

Enabling nftables is done via the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          kubeProxy:
            mode: nftables

Applying this configuration will result in the following configuration being applied to create a KubeProxyConfiguration and append it to the kubeadm configuration that is used when kubeadm init is executed:

KubeadmControlPlaneTemplate:

spec:
  template:
    spec:
      kubeadmConfigSpec:
        files:
          - path: "/etc/kubernetes/kubeproxy-config.yaml"
            owner: "root:root"
            permissions: "0644"
            content: |-
              ---
              apiVersion: kubeproxy.config.k8s.io/v1alpha1
              kind: KubeProxyConfiguration
              mode: nftables
      preKubeadmCommands:
        - /bin/sh -ec 'cat /etc/kubernetes/kubeproxy-config.yaml >> /run/kubeadm/kubeadm.yaml'

Skipping kube-proxy installation

To disable the deployment and upgrade of kube-proxy, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          kubeProxy:
            mode: disabled

Applying this configuration will result in the following configuration being applied:

KubeadmControlPlaneTemplate:

spec:
  template:
    spec:
      kubeadmConfigSpec:
        initConfiguration:
          skipPhases:
            - addon/kube-proxy

9 - Kubernetes Image Repository

Override the container image repository used when pulling Kubernetes images.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

To configure HTTP proxy values, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          kubernetesImageRepository: "my-registry.io/my-org/my-repo"

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:
- /spec/template/spec/kubeadmConfigSpec/clusterConfiguration/imageRepository: my-registry.io/my-org/my-repo

10 - Node registration configuration

Below is a list of node registration configuration options that can be set for kubeadm init and kubeadm join.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

ignorePreflightErrors

Kubeadm runs preflight checks to ensure the machine is compatible with Kubernetes and its dependencies. The SystemVerification check is known to result in false positives. For example, it fails when the Linux Kernel version is not supported by kubeadm, even if the kernel has all the required features. For this reason, we skip the check by default.

Control plane

To configure ignorePreflightErrors for the control plane nodes, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          controlPlane:
            nodeRegistration:
              ignorePreflightErrors:
                - SystemVerification

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    initConfiguration:
      nodeRegistration:
        nodeRegistration:
          ignorePreflightErrors:
            - SystemVerification
    joinConfiguration:
        nodeRegistration:
          ignorePreflightErrors:
            - SystemVerification

Worker node

ignorePreflightErrors for individual nodepools can be configured similarly:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    workers:
      machineDeployments:
      - class: default-worker
        name: md-0
        variables:
          overrides:
          - name: workerConfig
            value:
              nodeRegistration:
                ignorePreflightErrors:
                  - SystemVerification

Applying this configuration will result in the following value being set:

KubeadmConfigTemplate:

spec:
  joinConfiguration:
    nodeRegistration:
      ignorePreflightErrors:
        - SystemVerification

By default, the following value will be set for both control plane and worker nodes:

    variables:
      - name: clusterConfig
        value:
          controlPlane:
            nodeRegistration:
              ignorePreflightErrors:
                - SystemVerification
      - name: workerConfig
        value:
          nodeRegistration:
            ignorePreflightErrors:
              - SystemVerification

This can be enabled by setting ignorePreflightErrors to an empty list:

    variables:
      - name: clusterConfig
        value:
          controlPlane:
            nodeRegistration:
              ignorePreflightErrors: []
      - name: workerConfig
        value:
          nodeRegistration:
            ignorePreflightErrors: []

11 - Parallel Image Pulls

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

The parallel image pull configuration can then be manipulated via the cluster variables. If the maxParallelImagePullsPerNode property is not specified, then the default value of 1 will be used which is equivalent to serialized image pulls.

Setting this value to 0 results in unlimited parallel image pulls.

Example

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          maxParallelImagePullsPerNodePerNode: 10

Applying this configuration will result in a KubeletConfiguration patch being added which will be applied by kubeadm on init and join:

KubeadmControlPlaneTemplate:

spec:
  template:
    spec:
      kubeadmConfigSpec:
        files:
          - path: "/etc/kubernetes/patches/kubeletconfigurationmaxparallelimagepulls+strategic.json"
            owner: "root:root"
            permissions: "0644"
            content: |-
              ---
              apiVersion: kubelet.config.k8s.io/v1beta1
              kind: KubeletConfiguration
              serializeImagePulls: false
              maxParallelImagePulls: 10

KubeadmConfigTemplate

spec:
  kubeadmConfigSpec:
    files:
      - path: "/etc/kubernetes/patches/kubeletconfigurationmaxparallelimagepulls+strategic.json"
        owner: "root:root"
        permissions: "0644"
        content: |-
          ---
          apiVersion: kubelet.config.k8s.io/v1beta1
          kind: KubeletConfiguration
          serializeImagePulls: false
          maxParallelImagePulls: 10