The customizations in this section are applicable to providers using kubeadm to bootstrap the cluster.
This is the multi-page printable view of this section. Click here to print.
Kubeadm
- 1: Audit policy
- 2: Auto-renewal of control plane certificates
- 3: Containerd metrics
- 4: DNS
- 5: Encryption At Rest
- 6: etcd
- 7: Extra API Server Certificate SANs
- 8: kube-proxy mode
- 9: Kubernetes Image Repository
- 10: Node registration configuration
- 11: Parallel Image Pulls
1 - Audit policy
Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself.
There are currently no configuration options for the Audit Policy customization and this customization will be
automatically applied when the provider-specific cluster configuration patch is included in the
ClusterClass.
2 - Auto-renewal of control plane certificates
autoRenewCertificates variable enables automatic renewal of control plane certificates by triggering a rollout of the
control plane nodes when the certificates on the control plane machines are about to expire.
More information about certificate renewal: Automatically rotating certificates using Kubeadm Control Plane provider.
Example
To enable automatic certificate renewal use the following configuration, applicable to all CAPI providers supported by CAREN:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
spec:
topology:
variables:
- name: clusterConfig
value:
controlPlane:
autoRenewCertificates:
daysBeforeExpiry: 30
Applying this configuration will result in the following configuration being applied:
KubeadmControlPlaneTemplate:spec: template: spec: rolloutBefore: certificatesExpiryDays: 30
3 - Containerd metrics
Containerd exports metrics to a Prometheus endpoint. The metrics cover containerd itself, its plugins, e.g. CRI, and information about the containers managed by containerd.
There are currently no configuration options for metrics, and this
customization will be automatically applied when the provider-specific
cluster configuration patch is included in the
ClusterClass.
4 - DNS
This customization will be available when the
provider-specific cluster configuration patch is included in the ClusterClass.
The DNS configuration can then be manipulated via the cluster variables.
If the dns property is not specified, then the customization will be skipped.
CoreDNS
The CoreDNS configuration can then be manipulated via the cluster variables.
If the dns.coreDNS property is not specified, then the customization will be skipped.
Example
The CoreDNS version can be updated automatically. To do this, set coreDNS to an empty object:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
spec:
topology:
variables:
- name: clusterConfig
value:
dns:
coreDNS: {}
Applying this configuration will result in the following value being set, with the version of the CoreDNS image being set based on the cluster's Kubernetes version:
KubeadmControlPlaneTemplate:spec: kubeadmConfigSpec: clusterConfiguration: dns: imageTag: "v1.11.3"
To change the repository and tag for the container image for the CoreDNS pod, specify the following configuration:
Note do not include "coredns" in the repository, kubeadm already appends it.
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
spec:
topology:
variables:
- name: clusterConfig
value:
dns:
coreDNS:
image:
repository: my-registry.io/my-org/my-repo
tag: "v1.11.3_custom.0"
Applying this configuration will result in the following value being set:
KubeadmControlPlaneTemplate:spec: kubeadmConfigSpec: clusterConfiguration: dns: imageRepository: "my-registry.io/my-org/my-repo" imageTag: "v1.11.3_custom.0"
5 - Encryption At Rest
encryptionAtRest variable enables encrypting kubernetes resources at rest using provided encryption provider.
When this variable is set, kuberntetes secrets and configmaps are encrypted before writing them at etcd.
If the encryptionAtRest property is not specified, then
the customization will be skipped. The secrets and configmaps will not be stored as encrypted in etcd.
We support following encryption providers
- aescbc
- secretbox
More information about encryption at-rest: Encrypting Confidential Data at Rest
Example
To encrypt configmaps and secrets kubernetes resources using aescbc encryption provider:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
spec:
topology:
variables:
- name: clusterConfig
value:
encryptionAtRest:
providers:
- aescbc: {}
Applying this configuration will result in
<CLUSTER_NAME>-encryption-configsecret generated.
A secret key for the encryption provider is generated and stored in <CLUSTER_NAME>-encryption-config secret.
The APIServer will be configured to use the secret key to encrypt secrets and
configmaps kubernetes resources before writing them to etcd.
When reading resources from etcd, encryption provider that matches the stored data attempts in order to decrypt the data.
CAREN currently does not rotate the key once it generated.
- Configure APIServer with encryption configuration:
KubeadmControlPlaneTemplate:spec: kubeadmConfigSpec: clusterConfiguration: apiServer: extraArgs: encryption-provider-config: /etc/kubernetes/pki/encryptionconfig.yaml files: - contentFrom: secret: key: config name: <CLUSTER_NAME>-encryption-config path: /etc/kubernetes/pki/encryptionconfig.yaml permissions: "0640"
6 - etcd
This customization will be available when the
provider-specific cluster configuration patch is included in the ClusterClass.
The etcd configuration can then be manipulated via the cluster variables. If the etcd property is not specified, then
the customization will be skipped.
Example
To change the repository and tag for the container image for the etcd pod, specify the following configuration:
Note do not include "etcd" in the repository, kubeadm already appends it.
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
spec:
topology:
variables:
- name: clusterConfig
value:
etcd:
image:
repository: my-registry.io/my-org/my-repo
tag: "v3.5.99_custom.0"
Applying this configuration will result in the following value being set:
KubeadmControlPlaneTemplate:spec: kubeadmConfigSpec: clusterConfiguration: etcd: local: imageRepository: "my-registry.io/my-org/my-repo" imageTag: "v3.5.99_custom.0"
7 - Extra API Server Certificate SANs
If the API server can be accessed by alternative DNS addresses then setting additional SANs on the API server certificate is necessary in order for clients to successfully validate the API server certificate.
This customization will be available when the
provider-specific cluster configuration patch is included in the ClusterClass.
Example
To add extra SANs to the API server certificate, specify the following configuration:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
spec:
topology:
variables:
- name: clusterConfig
value:
extraAPIServerCertSANs:
- a.b.c.example.com
- d.e.f.example.com
Applying this configuration will result in the following value being set:
KubeadmControlPlaneTemplate:spec: kubeadmConfigSpec: clusterConfiguration: apiServer: certSANs: - a.b.c.example.com - d.e.f.example.com
8 - kube-proxy mode
This customization allows configuration of the kube-proxy proxy mode. Currently, only iptables and nftables
modes are supported. By default, kube-proxy is enabled in iptables mode by kubeadm.
Examples
Enabling nftables kube-proxy mode
Enabling nftables is done via the following configuration:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
spec:
topology:
variables:
- name: clusterConfig
value:
kubeProxy:
mode: nftables
Applying this configuration will result in the following configuration being applied to create a
KubeProxyConfiguration and append it to the kubeadm configuration that is used when kubeadm init
is executed:
KubeadmControlPlaneTemplate:spec: template: spec: kubeadmConfigSpec: files: - path: "/etc/kubernetes/kubeproxy-config.yaml" owner: "root:root" permissions: "0644" content: |- --- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: nftables preKubeadmCommands: - /bin/sh -ec 'cat /etc/kubernetes/kubeproxy-config.yaml >> /run/kubeadm/kubeadm.yaml'
Skipping kube-proxy installation
To disable the deployment and upgrade of kube-proxy, specify the following configuration:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
spec:
topology:
variables:
- name: clusterConfig
value:
kubeProxy:
mode: disabled
Applying this configuration will result in the following configuration being applied:
KubeadmControlPlaneTemplate:spec: template: spec: kubeadmConfigSpec: initConfiguration: skipPhases: - addon/kube-proxy
9 - Kubernetes Image Repository
Override the container image repository used when pulling Kubernetes images.
This customization will be available when the
provider-specific cluster configuration patch is included in the ClusterClass.
Example
To configure HTTP proxy values, specify the following configuration:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
spec:
topology:
variables:
- name: clusterConfig
value:
kubernetesImageRepository: "my-registry.io/my-org/my-repo"
Applying this configuration will result in the following value being set:
- KubeadmControlPlaneTemplate:
/spec/template/spec/kubeadmConfigSpec/clusterConfiguration/imageRepository: my-registry.io/my-org/my-repo
10 - Node registration configuration
Below is a list of node registration configuration options that can be set for kubeadm init and kubeadm join.
This customization will be available when the
provider-specific cluster configuration patch is included in the ClusterClass.
Example
ignorePreflightErrors
Kubeadm runs preflight checks to ensure the machine is compatible with Kubernetes and its dependencies.
The SystemVerification check is known to result in false positives.
For example, it fails when the Linux Kernel version is not supported by kubeadm,
even if the kernel has all the required features.
For this reason, we skip the check by default.
Control plane
To configure ignorePreflightErrors for the control plane nodes, specify the following configuration:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
spec:
topology:
variables:
- name: clusterConfig
value:
controlPlane:
nodeRegistration:
ignorePreflightErrors:
- SystemVerification
Applying this configuration will result in the following value being set:
KubeadmControlPlaneTemplate:spec: kubeadmConfigSpec: initConfiguration: nodeRegistration: nodeRegistration: ignorePreflightErrors: - SystemVerification joinConfiguration: nodeRegistration: ignorePreflightErrors: - SystemVerification
Worker node
ignorePreflightErrors for individual nodepools can be configured similarly:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
spec:
topology:
workers:
machineDeployments:
- class: default-worker
name: md-0
variables:
overrides:
- name: workerConfig
value:
nodeRegistration:
ignorePreflightErrors:
- SystemVerification
Applying this configuration will result in the following value being set:
KubeadmConfigTemplate:spec: joinConfiguration: nodeRegistration: ignorePreflightErrors: - SystemVerification
By default, the following value will be set for both control plane and worker nodes:
variables:
- name: clusterConfig
value:
controlPlane:
nodeRegistration:
ignorePreflightErrors:
- SystemVerification
- name: workerConfig
value:
nodeRegistration:
ignorePreflightErrors:
- SystemVerification
This can be enabled by setting ignorePreflightErrors to an empty list:
variables:
- name: clusterConfig
value:
controlPlane:
nodeRegistration:
ignorePreflightErrors: []
- name: workerConfig
value:
nodeRegistration:
ignorePreflightErrors: []
11 - Parallel Image Pulls
This customization will be available when the
provider-specific cluster configuration patch is included in the ClusterClass.
The parallel image pull configuration can then be manipulated via the cluster variables.
If the maxParallelImagePullsPerNode property is not specified, then the default value of 1 will be used
which is equivalent to serialized image pulls.
Setting this value to 0 results in unlimited parallel image pulls.
Example
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
spec:
topology:
variables:
- name: clusterConfig
value:
maxParallelImagePullsPerNodePerNode: 10
Applying this configuration will result in a KubeletConfiguration patch being added which will be
applied by kubeadm on init and join:
KubeadmControlPlaneTemplate:spec: template: spec: kubeadmConfigSpec: files: - path: "/etc/kubernetes/patches/kubeletconfigurationmaxparallelimagepulls+strategic.json" owner: "root:root" permissions: "0644" content: |- --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration serializeImagePulls: false maxParallelImagePulls: 10
KubeadmConfigTemplatespec: kubeadmConfigSpec: files: - path: "/etc/kubernetes/patches/kubeletconfigurationmaxparallelimagepulls+strategic.json" owner: "root:root" permissions: "0644" content: |- --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration serializeImagePulls: false maxParallelImagePulls: 10