This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Generic

1: Audit policy
2: Containerd metrics
3: Encryption At Rest
4: etcd
5: Extra API Server Certificate SANs
6: Global Image Registry Mirror
7: HTTP proxy
8: Image registries
9: Kubernetes Image Repository
10: Users

The customizations in this section are applicable to all providers.

1 - Audit policy

Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself.

There are currently no configuration options for the Audit Policy customization and this customization will be automatically applied when the provider-specific cluster configuration patch is included in the ClusterClass.

2 - Containerd metrics

Containerd exports metrics to a Prometheus endpoint. The metrics cover containerd itself, its plugins, e.g. CRI, and information about the containers managed by containerd.

There are currently no configuration options for metrics, and this customization will be automatically applied when the provider-specific cluster configuration patch is included in the ClusterClass.

3 - Encryption At Rest

encryptionAtRest variable enables encrypting kubernetes resources at rest using provided encryption provider. When this variable is set, kuberntetes secrets and configmaps are encrypted before writing them at etcd.

If the encryptionAtRest property is not specified, then the customization will be skipped. The secrets and configmaps will not be stored as encrypted in etcd.

We support following encryption providers

aescbc
secretbox

More information about encryption at-rest: Encrypting Confidential Data at Rest

Example

To encrypt configmaps and secrets kubernetes resources using aescbc encryption provider:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          encryptionAtRest:
            providers:
              - aescbc: {}

Applying this configuration will result in

<CLUSTER_NAME>-encryption-config secret generated.

A secret key for the encryption provider is generated and stored in <CLUSTER_NAME>-encryption-config secret. The APIServer will be configured to use the secret key to encrypt secrets and configmaps kubernetes resources before writing them to etcd. When reading resources from etcd, encryption provider that matches the stored data attempts in order to decrypt the data. CAREN currently does not rotate the key once it generated.

Configure APIServer with encryption configuration:

KubeadmControlPlaneTemplate:

  spec:
    kubeadmConfigSpec:
      clusterConfiguration:
        apiServer:
          extraArgs:
            encryption-provider-config: /etc/kubernetes/pki/encryptionconfig.yaml
    files:
      - contentFrom:
          secret:
            key: config
            name: <CLUSTER_NAME>-encryption-config
        path: /etc/kubernetes/pki/encryptionconfig.yaml
        permissions: "0640"

4 - etcd

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

The etcd configuration can then be manipulated via the cluster variables. If the etcd property is not specified, then the customization will be skipped.

Example

To change the repository and tag for the container image for the etcd pod, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          etcd:
            image:
              repository: my-registry.io/my-org/my-repo
              tag: "v3.5.99_custom.0"

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      etcd:
        local:
          imageRepository: "my-registry.io/my-org/my-repo"
          imageTag: "v3.5.99_custom.0"

5 - Extra API Server Certificate SANs

If the API server can be accessed by alternative DNS addresses then setting additional SANs on the API server certificate is necessary in order for clients to successfully validate the API server certificate.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

To add extra SANs to the API server certificate, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          extraAPIServerCertSANs:
            - a.b.c.example.com
            - d.e.f.example.com

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        certSANs:
          - a.b.c.example.com
          - d.e.f.example.com

6 - Global Image Registry Mirror

Add containerd image registry mirror configuration to all Nodes in the cluster.

When the globalImageRegistryMirror variable is set, files with configurations for Containerd default mirror.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

To provide an image registry mirror with a CA certificate, specify the following configuration:

If the registry mirror requires a private or self-signed CA certificate, create a Kubernetes Secret with the ca.crt key populated with the CA certificate in PEM format:

kubectl create secret generic my-mirror-ca-cert \
  --from-file=ca.crt=registry-ca.crt

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          globalImageRegistryMirror:
            url: https://example.com
            credentials:
              secretRef:
                name: my-mirror-ca-cert

Applying this configuration will result in following new files on the KubeadmControlPlaneTemplate and KubeadmConfigTemplate resources:

/etc/containerd/certs.d/_default/hosts.toml
/etc/certs/mirror.pem

To use a public hosted image registry (e.g. ECR) as a registry mirror, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          globalImageRegistryMirror:
            url: https://123456789.dkr.ecr.us-east-1.amazonaws.com

Applying this configuration will result in following new files on the KubeadmControlPlaneTemplate and KubeadmConfigTemplate resources:

/etc/containerd/certs.d/_default/hosts.toml

7 - HTTP proxy

In some network environments it is necessary to use HTTP proxy to successfuly execute HTTP requests. This customization will configure Kubernetes components (containerd, kubelet) with appropriate configuration for control plane and worker nodes, utilising systemd drop-ins to configure the necessary environment variables.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

To configure HTTP proxy values, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          proxy:
            http: http://example.com
            https: http://example.com
            additionalNo:
              - no-proxy-1.example.com
              - no-proxy-2.example.com

The additionalNo list will be added to default pre-calculated values that apply on k8s networking localhost,127.0.0.1,<POD CIDRS>,<SERVICE CIDRS>,kubernetes,kubernetes.default,.svc,.svc.cluster.local, plus provider-specific addresses as required.

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      files:
        - path: "/etc/systemd/system/containerd.service.d/http-proxy.conf"
          content: <generated>
        - path: "/etc/systemd/system/kubelet.service.d/http-proxy.conf"
          content: <generated>

KubeadmConfigTemplate:

spec:
  files:
    - path: "/etc/systemd/system/containerd.service.d/http-proxy.conf"
      content: <generated>
    - path: "/etc/systemd/system/kubelet.service.d/http-proxy.conf"
      content: <generated>

Applying this configuration will result in new bootstrap files on the KubeadmControlPlaneTemplate and KubeadmConfigTemplate.

8 - Image registries

Add image registry configuration to all Nodes in the cluster.

When the credentials variable is set, files and preKubeadmnCommands with configurations for Kubelet image credential provider and dynamic credential provider will be added.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

If your registry requires static credentials, create a Kubernetes Secret with keys for username and password:

kubectl create secret generic my-registry-credentials \
  --from-literal username=${REGISTRY_USERNAME} --from-literal password=${REGISTRY_PASSWORD}

If your registry requires a private or self-signed CA certificate, create a Kubernetes Secret with the ca.crt key populated with the CA certificate in PEM format:

kubectl create secret generic my-mirror-ca-cert \
  --from-file=ca.crt=registry-ca.crt

To set both image registry credentials and CA certificate, create a Kubernetes Secret with keys for username, password, and ca.crt:

kubectl create secret generic my-registry-credentials \
  --from-literal username=${REGISTRY_USERNAME} --from-literal password=${REGISTRY_PASSWORD} \
  --from-file=ca.crt=registry-ca.crt

To add image registry credentials and/or CA certificate, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          imageRegistries:
            - url: https://my-registry.io
              credentials:
                secretRef:
                  name: my-registry-credentials

Applying this configuration will result in new files and preKubeadmCommands on the KubeadmControlPlaneTemplate and KubeadmConfigTemplate.

9 - Kubernetes Image Repository

Override the container image repository used when pulling Kubernetes images.

This customization will be available when the provider-specific cluster configuration patch is included in the ClusterClass.

Example

To configure HTTP proxy values, specify the following configuration:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          kubernetesImageRepository: "my-registry.io/my-org/my-repo"

Applying this configuration will result in the following value being set:

KubeadmControlPlaneTemplate:
- /spec/template/spec/kubeadmConfigSpec/clusterConfiguration/imageRepository: my-registry.io/my-org/my-repo

10 - Users

Configure users for all machines in the cluster, the user's superuser capabilities using sudo user specifications, and the login authentication mechanism.

SSH authorized keys are just public SSH keys that are used to authenticate a login. See the SSH man page for more information.
For information on sudo user specifications, see the sudo documentation.
Local password authentication is disabled for the user by default. It is enabled only when a hashed password is provided.

Examples

Creates a user with the name admin, grants the user the ability to run any command as the superuser, and allows you to login via SSH using the username and private key corresponding to the authorized public key.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          users:
            - name: username
              sshAuthorizedKeys:
                - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAua0lo8BiGWgvIiDCKnQDKL5uERHfnehm0ns5CEJpJw optionalcomment"
              sudo: "ALL=(ALL) NOPASSWD:ALL"

Creates a user with the name admin, grants the user the ability to run any command as the superuser, and allows you to login via serial console using the username and password.

Note that this does not allow you to login via SSH using the username and password; in most cases, you must also configure the SSH server to allow password authentication.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          users:
            - name: admin
              hashedPassword: "$y$j9T$UraH8eN4XvapXBmmSaUrP0$Nyxdf1cJDGZcp0WDKu.CFHprrkPG4ubirqSqiD43Ix3"
              sudo: "ALL=(ALL) NOPASSWD:ALL"

Generic

1 - Audit policy

2 - Containerd metrics

3 - Encryption At Rest

Example

4 - etcd

Example

5 - Extra API Server Certificate SANs

Example

6 - Global Image Registry Mirror

Example

7 - HTTP proxy

Example

8 - Image registries

Example

9 - Kubernetes Image Repository

Example

10 - Users

Examples

Admin user with SSH public key login

Admin user with serial console password login