This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Generic

1: Audit policy
2: Auto-renewal of control plane certificates
3: Containerd metrics
4: Encryption At Rest
5: etcd
6: etcd
7: Extra API Server Certificate SANs
8: Global Image Registry Mirror
9: HTTP proxy
10: Image registries
11: Kubernetes Image Repository
12: Tainting nodes
13: Users

The customizations in this section are applicable to all providers.

1 - Audit policy

Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself.

There are currently no configuration options for the Audit Policy customization and this customization will be automatically applied when the provider-specific cluster configuration patch is included in the ClusterClass.

2 - Auto-renewal of control plane certificates

autoRenewCertificates variable enables automatic renewal of control plane certificates by triggering a rollout of the control plane nodes when the certificates on the control plane machines are about to expire.

More information about certificate renewal: Automatically rotating certificates using Kubeadm Control Plane provider.

Example

To enable automatic certificate renewal use the following configuration, applicable to all CAPI providers supported by CAREN:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          controlPlane:
            autoRenewCertificates:
              daysBeforeExpiry: 30

Applying this configuration will result in the following configuration being applied:

KubeadmControlPlaneTemplate:

spec:
  template:
    spec:
      rolloutBefore:
        certificatesExpiryDays: 30

3 - Containerd metrics

Containerd exports metrics to a Prometheus endpoint. The metrics cover containerd itself, its plugins, e.g. CRI, and information about the containers managed by containerd.

There are currently no configuration options for metrics, and this customization will be automatically applied when the provider-specific cluster configuration patch is included in the ClusterClass.

4 - Encryption At Rest

encryptionAtRest variable enables encrypting kubernetes resources at rest using provided encryption provider. When this variable is set, kuberntetes secrets and configmaps are encrypted before writing them at etcd.

If the encryptionAtRest property is not specified, then the customization will be skipped. The secrets and configmaps will not be stored as encrypted in etcd.

We support following encryption providers

aescbc
secretbox

More information about encryption at-rest: Encrypting Confidential Data at Rest

Example

To encrypt configmaps and secrets kubernetes resources using aescbc encryption provider:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          encryptionAtRest:
            providers:
              - aescbc: {}

Applying this configuration will result in

<CLUSTER_NAME>-encryption-config secret generated.

A secret key for the encryption provider is generated and stored in <CLUSTER_NAME>-encryption-config secret. The APIServer will be configured to use the secret key to encrypt secrets and configmaps kubernetes resources before writing them to etcd. When reading resources from etcd, encryption provider that matches the stored data attempts in order to decrypt the data. CAREN currently does not rotate the key once it generated.

Configure APIServer with encryption configuration:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
          encryption-provider-config: /etc/kubernetes/pki/encryptionconfig.yaml
  files:
    - contentFrom:
        secret:
          key: config
          name: <CLUSTER_NAME>-encryption-config
      path: /etc/kubernetes/pki/encryptionconfig.yaml
      permissions: "0640"

5 - etcd

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

The DNS configuration can then be manipulated via the cluster variables. If the dns property is not specified, then the customization will be skipped.

CoreDNS

The CoreDNS configuration can then be manipulated via the cluster variables. If the dns.coreDNS property is not specified, then the customization will be skipped.

Example

The CoreDNS version can be updated automatically. To do this, set coreDNS to an empty object:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          dns:
            coreDNS: {}

Applying this configuration will result in the following value being set, with the version of the CoreDNS image being set based on the cluster's Kubernetes version:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      dns:
        imageTag: "v1.11.3"

To change the repository and tag for the container image for the CoreDNS pod, specify the following configuration:

Note do not include "coredns" in the repository, kubeadm already appends it.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          dns:
            coreDNS:
              image:
                repository: my-registry.io/my-org/my-repo
                tag: "v1.11.3_custom.0"

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      dns:
        imageRepository: "my-registry.io/my-org/my-repo"
        imageTag: "v1.11.3_custom.0"

6 - etcd

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

The etcd configuration can then be manipulated via the cluster variables. If the etcd property is not specified, then the customization will be skipped.

Example

To change the repository and tag for the container image for the etcd pod, specify the following configuration:

Note do not include "etcd" in the repository, kubeadm already appends it.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          etcd:
            image:
              repository: my-registry.io/my-org/my-repo
              tag: "v3.5.99_custom.0"

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      etcd:
        local:
          imageRepository: "my-registry.io/my-org/my-repo"
          imageTag: "v3.5.99_custom.0"

7 - Extra API Server Certificate SANs

If the API server can be accessed by alternative DNS addresses then setting additional SANs on the API server certificate is necessary in order for clients to successfully validate the API server certificate.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

To add extra SANs to the API server certificate, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          extraAPIServerCertSANs:
            - a.b.c.example.com
            - d.e.f.example.com

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        certSANs:
          - a.b.c.example.com
          - d.e.f.example.com

8 - Global Image Registry Mirror

Add containerd image registry mirror configuration to all Nodes in the cluster.

When the globalImageRegistryMirror variable is set, files with configurations for Containerd default mirror.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

To provide an image registry mirror with a CA certificate, specify the following configuration:

If the registry mirror requires a private or self-signed CA certificate, create a Kubernetes Secret with the ca.crt key populated with the CA certificate in PEM format:

kubectl create secret generic my-mirror-ca-cert \
  --from-file=ca.crt=registry-ca.crt

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          globalImageRegistryMirror:
            url: https://example.com
            credentials:
              secretRef:
                name: my-mirror-ca-cert

Applying this configuration will result in following new files on the KubeadmControlPlaneTemplate and KubeadmConfigTemplate resources:

/etc/containerd/certs.d/_default/hosts.toml
/etc/certs/mirror.pem

To use a public hosted image registry (e.g. ECR) as a registry mirror, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          globalImageRegistryMirror:
            url: https://123456789.dkr.ecr.us-east-1.amazonaws.com

Applying this configuration will result in following new files on the KubeadmControlPlaneTemplate and KubeadmConfigTemplate resources:

/etc/containerd/certs.d/_default/hosts.toml

9 - HTTP proxy

In some network environments it is necessary to use HTTP proxy to successfuly execute HTTP requests. This customization will configure Kubernetes components (containerd, kubelet) with appropriate configuration for control plane and worker nodes, utilising systemd drop-ins to configure the necessary environment variables.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

To configure HTTP proxy values, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          proxy:
            http: http://example.com
            https: http://example.com
            additionalNo:
              - no-proxy-1.example.com
              - no-proxy-2.example.com

The additionalNo list will be added to default pre-calculated values that apply on k8s networking localhost,127.0.0.1,<POD CIDRS>,<SERVICE CIDRS>,kubernetes,kubernetes.default,.svc,.svc.cluster.local, plus provider-specific addresses as required.

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      files:
        - path: "/etc/systemd/system/containerd.service.d/http-proxy.conf"
          content: <generated>
        - path: "/etc/systemd/system/kubelet.service.d/http-proxy.conf"
          content: <generated>

KubeadmConfigTemplate:

spec:
  files:
    - path: "/etc/systemd/system/containerd.service.d/http-proxy.conf"
      content: <generated>
    - path: "/etc/systemd/system/kubelet.service.d/http-proxy.conf"
      content: <generated>

Applying this configuration will result in new bootstrap files on the KubeadmControlPlaneTemplate and KubeadmConfigTemplate.

10 - Image registries

Add image registry configuration to all Nodes in the cluster.

When the credentials variable is set, files and preKubeadmnCommands with configurations for Kubelet image credential provider and dynamic credential provider will be added.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

If your registry requires static credentials, create a Kubernetes Secret with keys for username and password:

kubectl create secret generic my-registry-credentials \
  --from-literal username=${REGISTRY_USERNAME} --from-literal password=${REGISTRY_PASSWORD}

If your registry requires a private or self-signed CA certificate, create a Kubernetes Secret with the ca.crt key populated with the CA certificate in PEM format:

kubectl create secret generic my-mirror-ca-cert \
  --from-file=ca.crt=registry-ca.crt

To set both image registry credentials and CA certificate, create a Kubernetes Secret with keys for username, password, and ca.crt:

kubectl create secret generic my-registry-credentials \
  --from-literal username=${REGISTRY_USERNAME} --from-literal password=${REGISTRY_PASSWORD} \
  --from-file=ca.crt=registry-ca.crt

To add image registry credentials and/or CA certificate, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          imageRegistries:
            - url: https://my-registry.io
              credentials:
                secretRef:
                  name: my-registry-credentials

Applying this configuration will result in new files and preKubeadmCommands on the KubeadmControlPlaneTemplate and KubeadmConfigTemplate.

11 - Kubernetes Image Repository

Override the container image repository used when pulling Kubernetes images.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

To configure HTTP proxy values, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          kubernetesImageRepository: "my-registry.io/my-org/my-repo"

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:
- /spec/template/spec/kubeadmConfigSpec/clusterConfiguration/imageRepository: my-registry.io/my-org/my-repo

12 - Tainting nodes

Tainting nodes prevents pods from being scheduled on them unless they explicitly tolerate the taints applied to the nodes. See the Kubernetes Taints and Tolerations documentation for more details.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

Control plane taints

To configure taints for the control plane nodes, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          controlPlane:
            taints:
              - key: some-key
                effect: NoSchedule
                value: some-value

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    initConfiguration:
      nodeRegistration:
        taints:
          - key: some-key
            effect: NoSchedule
            value: some-value
    joinConfiguration:
      nodeRegistration:
        taints:
          - key: some-key
            effect: NoSchedule
            value: some-value

Default control-plane taint applied by kubeadm

When using this customization, the default taint added by kubeadm to the control plane nodes will not be added unless explicitly specified as well.

To add the default taint back to the control-plane, add the following taint along with any custom taints you wish to add to the control-plane taints:

- key: node-role.kubernetes.io/control-plane
  effect: NoSchedule

Removing all taints from control-plane nodes

To remove the default control plane taints set by kubeadm (and therefore allow scheduling to control plane nodes without adding explicit tolerations to your pod manifests), set controlPlane.taints to an empty array:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          controlPlane:
            taints: []

Worker node taints

Taints for individual nodepools can be configured similarly:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    workers:
      machineDeployments:
      - class: default-worker
        name: md-0
        variables:
          overrides:
          - name: workerConfig
            value:
              taints:
                - key: some-key
                  effect: NoSchedule
                  value: some-value

Applying this configuration will result in the following value being set:

KubeadmConfigTemplate:

spec:
  joinConfiguration:
    nodeRegistration:
      taints:
        - key: some-key
          effect: NoSchedule
          value: some-value

13 - Users

Configure users for all machines in the cluster, the user's superuser capabilities using sudo user specifications, and the login authentication mechanism.

SSH authorized keys are just public SSH keys that are used to authenticate a login. See the SSH man page for more information.
For information on sudo user specifications, see the sudo documentation.
Local password authentication is disabled for the user by default. It is enabled only when a hashed password is provided.

Examples

Creates a user with the name admin, grants the user the ability to run any command as the superuser, and allows you to login via SSH using the username and private key corresponding to the authorized public key.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          users:
            - name: username
              sshAuthorizedKeys:
                - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAua0lo8BiGWgvIiDCKnQDKL5uERHfnehm0ns5CEJpJw optionalcomment"
              sudo: "ALL=(ALL) NOPASSWD:ALL"

Creates a user with the name admin, grants the user the ability to run any command as the superuser, and allows you to login via serial console using the username and password.

Note that this does not allow you to login via SSH using the username and password; in most cases, you must also configure the SSH server to allow password authentication.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          users:
            - name: admin
              hashedPassword: "$y$j9T$UraH8eN4XvapXBmmSaUrP0$Nyxdf1cJDGZcp0WDKu.CFHprrkPG4ubirqSqiD43Ix3"
              sudo: "ALL=(ALL) NOPASSWD:ALL"

Generic

1 - Audit policy

2 - Auto-renewal of control plane certificates

Example

3 - Containerd metrics

4 - Encryption At Rest

Example

5 - etcd

CoreDNS

Example

6 - etcd

Example

7 - Extra API Server Certificate SANs

Example

8 - Global Image Registry Mirror

Example

9 - HTTP proxy

Example

10 - Image registries

Example

11 - Kubernetes Image Repository

Example

12 - Tainting nodes

Example

Control plane taints

Default control-plane taint applied by kubeadm

Removing all taints from control-plane nodes

Worker node taints

13 - Users

Examples

Admin user with SSH public key login

Admin user with serial console password login