This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Nutanix

The customizations in this section are applicable only to Nutanix clusters. They will only be applied to clusters that use the Nutanix infrastructure provider, i.e. a CAPI Cluster that references an NutanixCluster.

1 - Control Plane Endpoint

Configure Control Plane Endpoint. Defines the host IP and port of the Nutanix Kubernetes cluster.

Examples

Set Control Plane Endpoint

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          nutanix:
            controlPlaneEndpoint:
              host: x.x.x.x
              port: 6443
              virtualIP: {}

Applying this configuration will result in the following value being set:

  • NutanixCluster:
spec:
  template:
    spec:
      controlPlaneEndpoint:
        host: x.x.x.x
        port: 6443
  • KubeadmControlPlaneTemplate
  spec:
    kubeadmConfigSpec:
      files:
      - content: |
          apiVersion: v1
          kind: Pod
          metadata:
            name: kube-vip
            namespace: kube-system
          spec:
            containers:
            - name: kube-vip
              args:
              - manager
              env:
              - name: port
                value: '6443'
              - name: address
                value: 'x.x.x.x'
          ...
        owner: root:root
        path: /etc/kubernetes/manifests/kube-vip.yaml
        permissions: "0600"
      postKubeadmCommands:
        # Only added for clusters version >=v1.29.0
        - |-
          if [ -f /run/kubeadm/kubeadm.yaml ]; then
            sed -i 's#path: /etc/kubernetes/super-admin.conf#path: ...
          fi
      preKubeadmCommands:
        # Only added for clusters version >=v1.29.0
        - |-
          if [ -f /run/kubeadm/kubeadm.yaml ]; then
            sed -i 's#path: /etc/kubernetes/admin.conf#path: ...
          fi

Set Control Plane Endpoint and a Different Virtual IP

It is also possible to set a separate virtual IP to be used by kube-vip from the control plane endpoint. This is useful in VPC setups or other instances when you have an external floating IP already associated with the virtual IP.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          nutanix:
            controlPlaneEndpoint:
              host: x.x.x.x
              port: 6443
              virtualIP:
                configuration:
                  address: y.y.y.y

Applying this configuration will result in the following value being set:

  • NutanixCluster:
spec:
  template:
    spec:
      controlPlaneEndpoint:
        host: x.x.x.x
        port: 6443
  • KubeadmControlPlaneTemplate
  spec:
    kubeadmConfigSpec:
      files:
      - content: |
          apiVersion: v1
          kind: Pod
          metadata:
            name: kube-vip
            namespace: kube-system
          spec:
            containers:
            - name: kube-vip
              args:
              - manager
              env:
              - name: port
                value: '6443'
              - name: address
                value: 'y.y.y.y'
          ...
        owner: root:root
        path: /etc/kubernetes/manifests/kube-vip.yaml
        permissions: "0600"
      postKubeadmCommands:
        # Only added for clusters version >=v1.29.0
        - |-
          if [ -f /run/kubeadm/kubeadm.yaml ]; then
            sed -i 's#path: /etc/kubernetes/super-admin.conf#path: ...
          fi
      preKubeadmCommands:
        # Only added for clusters version >=v1.29.0
        - |-
          if [ -f /run/kubeadm/kubeadm.yaml ]; then
            sed -i 's#path: /etc/kubernetes/admin.conf#path: ...
          fi

2 - Machine Details

Configure Machine Details of Control plane and Worker nodes

Examples

(Required) Set Machine details for Control Plane and Worker nodes

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          controlPlane:
            nutanix:
              machineDetails:
                bootType: legacy
                cluster:
                  name: pe-cluster-name
                  type: name
                image:
                  name: os-image-name
                  type: name
                memorySize: 4Gi
                subnets:
                - name: subnet-name
                  type: name
                systemDiskSize: 40Gi
                vcpuSockets: 2
                vcpusPerSocket: 1
      - name: workerConfig
        value:
          nutanix:
            machineDetails:
              bootType: legacy
              cluster:
                name: pe-cluster-name
                type: name
              image:
                name: os-image-name
                type: name
              memorySize: 4Gi
              subnets:
              - name: subnet-name
                type: name
              systemDiskSize: 40Gi
              vcpuSockets: 2
              vcpusPerSocket: 1

Applying this configuration will result in the following value being set:

  • control-plane NutanixMachineTemplate:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: NutanixMachineTemplate
metadata:
  name: nutanix-quick-start-cp-nmt
spec:
  template:
    spec:
      bootType: legacy
      cluster:
        name: pe-cluster-name
        type: name
      image:
        name: os-image-name
        type: name
      memorySize: 4Gi
      providerID: nutanix://vm-uuid
      subnet:
      - name: subnet-name
        type: name
      systemDiskSize: 40Gi
      vcpuSockets: 2
      vcpusPerSocket: 1
  • worker NutanixMachineTemplate:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: NutanixMachineTemplate
metadata:
  name: nutanix-quick-start-md-nmt
spec:
  template:
    spec:
      bootType: legacy
      cluster:
        name: pe-cluster-name
        type: name
      image:
        name: os-image-name
        type: name
      memorySize: 4Gi
      providerID: nutanix://vm-uuid
      subnet:
      - name: subnet-name
        type: name
      systemDiskSize: 40Gi
      vcpuSockets: 2
      vcpusPerSocket: 1

Stretched Kubernetes clusters on an AHV Metro reject unsupported VM image OS versions

A stretched Kubernetes cluster on an AHV Metro rejects VM images whose operating system is on the unsupported list. The restriction applies to every Control Plane and Worker node pool. Clusters that are not stretched on an AHV Metro can use those OS versions.

The unsupported list currently contains rhel-8, which matches the RHEL 8 family. Each entry is matched as its own token, so rhel-8 matches rhel-8.10, rhel-8.1, and nkp-rhel-8.10-release-1.33.1-*, and does not match rhel-9 or rhel-80.

Preflight check NutanixMetroVMImage rejects the cluster when either of these matches an unsupported OS version:

  • imageLookup.baseOS
  • the Prism Central VM image name

Choose a VM image for a supported operating system, such as Rocky Linux or RHEL 9, and retry. To bypass the check, add NutanixMetroVMImage to the Cluster annotation preflight.cluster.caren.nutanix.com/skip.

(Optional) Using templates for VM Image lookup

Users can set format look up the image for a VM, It will be ignored if an explicit image is set. Supports substitutions for {{.BaseOS}} and {{.K8sVersion}} with the base OS and kubernetes version, respectively. The BaseOS will be the value in BaseOS and the K8sVersion is the value in the Machine .spec.version, with the v prefix removed. This is effectively the defined by the packages produced by kubernetes/release without v as a prefix: 1.13.0, 1.12.5-mybuild.1, or 1.17.3. For example, the default image format of {{.BaseOS}}-?{{.K8sVersion}}-* and BaseOS as "rhel-8.10" will end up searching for images that match the pattern rhel-8.10-1.30.5-* for a Machine that is targeting Kubernetes version v1.30.5. See also go text template

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          controlPlane:
            nutanix:
              machineDetails:
                bootType: legacy
                cluster:
                  name: pe-cluster-name
                  type: name
                imageLookup:
                  baseOS: "rockylinux-9"
                  format: {{.BaseOS}}-kube-v{{.K8sVersion}}.*

Applying this configuration will result in the following value being set:

  • control-plane NutanixMachineTemplate:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: NutanixMachineTemplate
metadata:
  name: nutanix-quick-start-cp-nmt
spec:
  template:
    spec:
      bootType: legacy
      cluster:
        name: pe-cluster-name
        type: name
      imageLookup:
        baseOS: "rockylinux-9"
        format: {{.BaseOS}}-kube-v{{.K8sVersion}}.*
      memorySize: 4Gi
      providerID: nutanix://vm-uuid
      subnet:
      - name: subnet-name
        type: name
      systemDiskSize: 40Gi
      vcpuSockets: 2
      vcpusPerSocket: 1

(Optional) Set Additional Categories for Control Plane and Worker nodes

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          controlPlane:
            nutanix:
              machineDetails:
                additionalCategories:
                - key: example-key
                  value: example-value
      - name: workerConfig
        value:
          nutanix:
            machineDetails:
              additionalCategories:
              - key: example-key
                value: example-value

Applying this configuration will result in the following value being set:

  • control-plane NutanixMachineTemplate:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: NutanixMachineTemplate
metadata:
  name: nutanix-quick-start-cp-nmt
spec:
  template:
    spec:
      additionalCategories:
      - key: example-key
        value: example-value
  • worker NutanixMachineTemplate:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: NutanixMachineTemplate
metadata:
  name: nutanix-quick-start-md-nmt
spec:
  template:
    spec:
      additionalCategories:
      - key: example-key
        value: example-value

(Optional) Set Project for Control Plane and Worker nodes

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          controlPlane:
            nutanix:
              machineDetails:
                project:
                  type: name
                  name: project-name
      - name: workerConfig
        value:
          nutanix:
            machineDetails:
              project:
                type: name
                name: project-name

Applying this configuration will result in the following value being set:

  • control-plane NutanixMachineTemplate:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: NutanixMachineTemplate
metadata:
  name: nutanix-quick-start-cp-nmt
spec:
  template:
    spec:
      project:
        type: name
        name: project-name
  • worker NutanixMachineTemplate:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: NutanixMachineTemplate
metadata:
  name: nutanix-quick-start-md-nmt
spec:
  template:
    spec:
      project:
        type: name
        name: project-name

(Optional) Add a GPU to a machine deployment

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
    - name: workerConfig
      value:
        nutanix:
          machineDetails:
            gpus:
            - type: name
              name: "Ampere 40"
    workers:
      - class: nutanix-quick-start-worker
        metadata:
          annotations:
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "1"
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1"
        name: gpu-0

Applying this configuration will result in the following value being set:

  • control-plane NutanixMachineTemplate:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: NutanixMachineTemplate
metadata:
  name: nutanix-quick-start-gpu-nmt
spec:
  template:
    spec:
      gpus:
      - type: name
        name: "Ampere 40"

3 - Prism Central Endpoint

Configure Prism Central Endpoint to create machines on.

Examples

Set Prism Central Endpoint

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          nutanix:
            prismCentralEndpoint:
              credentials:
                secretRef:
                  name: secret-name
              url: https://x.x.x.x:9440
              insecure: false

Applying this configuration will result in the following value being set:

  • NutanixClusterTemplate:
spec:
  template:
    spec:
      prismCentral:
        address: x.x.x.x
        insecure: false
        port: 9440
        credentialRef:
          kind: Secret
          name: secret-name

Provide an Optional Trusted CA Bundle

If the Prism Central endpoint uses a self-signed certificate, you can provide an additional trust bundle to be used by the Nutanix provider. This is a base64 PEM encoded x509 cert for the RootCA that was used to create the certificate for a Prism Central

See Nutanix Security Guide for more information.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <NAME>
spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          nutanix:
            prismCentralEndpoint:
              # ...
              additionalTrustBundle: "LS0...="

Applying this configuration will result in the following value being set:

  • NutanixClusterTemplate:
spec:
  template:
    spec:
      prismCentral:
        # ...
        additionalTrustBundle:
          kind: String
          data: |-
            -----BEGIN CERTIFICATE-----
            ...
            -----END CERTIFICATE-----