Alauda Container Platform

Prerequisites for Deploying on Immutable Infrastructure

This page describes the prerequisites for deploying Alauda Build of Rook-Ceph on Immutable Infrastructure clusters.

This page covers only the storage persistence and node-replacement requirements that are specific to deploying Alauda Build of Rook-Ceph on Immutable Infrastructure. For general Ceph requirements such as node count, CPU and memory, networking, time synchronization, capacity, versions, and health checks, see Planning Your Deployment and the applicable installation documentation. Immutable Infrastructure updates the operating system, Kubernetes, and node configuration by replacing nodes, so storage data must not depend on temporary state local to a node.

The following infrastructure types are covered:

  • Huawei DCS
  • Huawei Cloud Stack (HCS)
  • VMware vSphere
  • Bare Metal

For the global cluster, create and update the MachineTemplate, ConfigPool, MachineDeployment, and related resources in the global cluster, normally in the cpaas-system namespace. For an overview, see About Immutable Infrastructure. For the provider contract, see Immutable Infrastructure Resources, Managing Nodes, and API Reference.

TOC

Immutable Infrastructure RequirementsPersistence and Node ReplacementMON/Rook Data DirectoryOSD Data DisksRolling Node ReplacementHuawei DCSHuawei Cloud Stack (HCS)VMware vSphereBare MetalPre-installation Checklist

Immutable Infrastructure Requirements

Persistence and Node Replacement

MON/Rook Data Directory

The MON database and Rook management data are stored under dataDirHostPath. This directory must survive node replacement, so it must be on a persistent file system managed by the provider. It must not be located on the node system disk, a temporary disk, or a raw OSD device.

Choose one of the following configurations:

  • Mount the persistent file system at the product-supported default path /var/lib/rook and set spec.dataDirHostPath: /var/lib/rook in the StorageCluster/CephCluster configuration. This is the supported path for the Internal Mode installation flow.

  • A custom path, such as /var/cpaas/rook, is supported only when configuring the resource through YAML/API. The Internal Mode UI does not provide a control for changing this path; do not use a custom path with the UI installation flow.

    spec:
  dataDirHostPath: /var/cpaas/rook

Treat dataDirHostPath as immutable after the storage cluster is created. Select the final persistent mount path before creating the StorageCluster/CephCluster.

The persistent file system can use a dedicated disk or share a disk with other non-OSD persistent directories. Size it according to the general Ceph planning requirements.

OSD Data Disks

OSD data disks must be managed by the provider's persistent-disk mechanism and remounted on the corresponding worker after node replacement. Use a disk UUID, URN, slot, or an equivalent provider-defined identity to confirm that the same disk is used before and after replacement.

OSD data disks must use raw block mode and must not also provide the file system that contains dataDirHostPath. After the provider mounts or attaches the persistent disk, add the independent OSD device to Ceph through the Rook/ACP storage configuration. The exact device configuration depends on the current ACP Storage/Rook deployment mode. Capacities in the provider examples illustrate provider fields only and are not general sizing recommendations.

In the Internal Mode wizard, add the provider-presented raw disk to an OSD device class using the stable device path supplied by the provider, preferably /dev/disk/by-id/.... Do not use the MON/Rook file-system disk as an OSD device. See Deploying in Internal Mode for the device-class configuration.

Provider disk presentationInternal Mode configurationDevice value
Provider presents a blank raw diskAdd it to the OSD device classEnter the stable provider path, preferably /dev/disk/by-id/...; do not use a short /dev/sdX name
Provider presents the MON/Rook data disk as a mounted file systemDo not add it to an OSD device classKeep it mounted at /var/lib/rook for dataDirHostPath

Rolling Node Replacement

When Ceph data disks are managed by an Immutable Infrastructure provider pool, the worker MachineDeployment must use a fixed-slot model. maxSurge: 0 and maxUnavailable: 1 are mandatory because a surge VM cannot safely claim the same hostname, IP, or persistent-disk slot as the node it replaces, while more than one unavailable worker would remove multiple storage slots at once:

strategy:
  rollingUpdate:
    maxSurge: 0
    maxUnavailable: 1

Also ensure that:

  • A finite nodeDrainTimeout is not configured.
  • The Kubernetes Eviction API, PDBs, and the Rook disruption controller remain in the protection path.
  • Replacement is not forced by deleting PDBs or finalizers, or by forcibly cleaning up OSDs.

For newly created built-in storage clusters, keep the product default PDB protection enabled (managePodBudgets: true). Do not explicitly disable it for an Immutable Infrastructure worker pool. Existing clusters with PDB management disabled must be corrected before replacement. For the general PDB behavior, see Using PodDisruptionBudgets.

Huawei DCS

DCS environments must declare the persistent disks available to Rook through DCSIpHostnamePool. Configure the resource in the global cluster, normally in cpaas-system. For provider-specific infrastructure, node-management, and API details, see DCS Infrastructure Resources, DCS Node Management, and DCS Persistent IP Hostname Pool API:

The IP addresses, gateway, and DNS values in this example show the relationship between pool slots and node network identities. Replace them with values from your environment. They are provider node settings, not Rook-specific network requirements.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSIpHostnamePool
metadata:
  name: ceph-worker-pool
  namespace: cpaas-system
spec:
  pool:
    - ip: 192.0.2.101
      mask: "24"
      gateway: 192.0.2.1
      dns: 192.0.2.53
      hostname: worker-01
      persistentDisk:
        - slot: 0
          quantityGB: 30
          datastoreClusterName: worker-datastore-cluster
          path: /var/lib/rook
          format: xfs
        - slot: 1
          quantityGB: 300
          datastoreClusterName: worker-datastore-cluster
    - ip: 192.0.2.102
      mask: "24"
      gateway: 192.0.2.1
      dns: 192.0.2.53
      hostname: worker-02
      persistentDisk:
        - slot: 0
          quantityGB: 30
          datastoreClusterName: worker-datastore-cluster
          path: /var/lib/rook
          format: xfs
        - slot: 1
          quantityGB: 300
          datastoreClusterName: worker-datastore-cluster
    - ip: 192.0.2.103
      mask: "24"
      gateway: 192.0.2.1
      dns: 192.0.2.53
      hostname: worker-03
      persistentDisk:
        - slot: 0
          quantityGB: 30
          datastoreClusterName: worker-datastore-cluster
          path: /var/lib/rook
          format: xfs
        - slot: 1
          quantityGB: 300
          datastoreClusterName: worker-datastore-cluster

The corresponding DCSMachineTemplate must reference this pool through ipHostPoolRef:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSMachineTemplate
metadata:
  name: ceph-worker-template
  namespace: cpaas-system
spec:
  template:
    spec:
      ipHostPoolRef:
        name: ceph-worker-pool
      # Other VM, network, and resource settings omitted

Key requirements:

  • Configure Rook's disks in DCSIpHostnamePool.spec.pool[].persistentDisk; do not configure only ordinary virtual disks in the DCSMachine template.
  • DCS reuses persistent-disk identities by (IP, slot). After node replacement, confirm that each original OSD data disk still maps to the same persistent-disk identity.
  • Set path: /var/lib/rook and format: xfs for the MON/Rook data-directory disk. Omit path and format for OSD disks so that they remain raw block devices.
  • Set exactly one of datastoreClusterName and datastoreName; this example uses a datastore cluster. DCS reuses persistent disks by (IP, slot) in the pool entry and records the volume URN in status.persistentDiskStatus.

Huawei Cloud Stack (HCS)

For HCS, confirm that the provider supplies persistent, remountable Ceph data disks for each storage worker. Configure the resource in the global cluster, normally in cpaas-system. See HCS Infrastructure Resources, HCS Node Management, and HCS Machine Config Pool API for provider details:

The network settings in this example illustrate the node configuration and fixed IPs. Replace them with your environment's network parameters. If a usable ConfigPool already exists, retain its network settings and confirm that node replacement still reuses the same hostname and slot.

The HCS provider declares persistent disks for a fixed machine configuration through HCSMachineConfigPool:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HCSMachineConfigPool
metadata:
  name: hcs-acp-biz-worker-pool
  namespace: cpaas-system
spec:
  configs:
    - hostname: worker-01
      networks:
        - ipAddress: 192.0.2.101
          subnetName: worker-subnet
      persistentDisks:
        - slot: 0
          size: 30
          type: SSD
          mountPath: /var/lib/rook
          format: xfs
        - slot: 1
          size: 300
          type: SSD
    - hostname: worker-02
      networks:
        - ipAddress: 192.0.2.102
          subnetName: worker-subnet
      persistentDisks:
        - slot: 0
          size: 30
          type: SSD
          mountPath: /var/lib/rook
          format: xfs
        - slot: 1
          size: 300
          type: SSD
    - hostname: worker-03
      networks:
        - ipAddress: 192.0.2.103
          subnetName: worker-subnet
      persistentDisks:
        - slot: 0
          size: 30
          type: SSD
          mountPath: /var/lib/rook
          format: xfs
        - slot: 1
          size: 300
          type: SSD

The slot values in persistentDisks[] must be contiguous and start at 0. Together with the hostname, they form the persistent-disk identity. In this example, slot 0 is the MON/Rook data-directory disk and slot 1 is the raw OSD disk. Configure persistent disks in the ConfigPool rather than only in HCSMachineTemplate.spec.template.spec.dataVolumes[], which contains temporary data disks that are removed with the ECS instance.

Key requirements:

  • Configure persistent disks in HCSMachineConfigPool.spec.configs[].persistentDisks[], not only in HCSMachineTemplate.spec.template.spec.dataVolumes[].
  • Slot 0 is the MON/Rook data-directory disk and must use mountPath: /var/lib/rook with format: xfs; slot 1 is an independent raw OSD disk and must not have a file-system mount.
  • The (hostname, slot) identity must be reused after node replacement so that the original persistent disk is remounted.

The corresponding HCSMachineTemplate must reference this pool through configPoolRef and must also be created in the global cluster, normally in cpaas-system:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HCSMachineTemplate
metadata:
  name: ceph-worker-template
  namespace: cpaas-system
spec:
  template:
    spec:
      configPoolRef:
        name: hcs-acp-biz-worker-pool
      # Other ECS, image, and network settings omitted

Follow the provider API and node-management documentation for the target version for HCS disk types, mount policies, and machine-template fields.

VMware vSphere

The vSphere provider uses VSphereMachineConfigPool to declare persistent disks that are retained across VM deletion for a fixed machine configuration. Configure the resource in the global cluster, normally in cpaas-system. See vSphere Infrastructure Resources, vSphere Node Management, and vSphere Machine Config Pool API. The VSphereMachineTemplate then references this pool:

The example also lists network settings to show how fixed IPs are reused with pool slots. Replace the IP, gateway, DNS, and network name with values from your environment. These network fields are not Rook-specific, but node replacement must preserve the node network identity.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineConfigPool
metadata:
  name: ceph-worker-pool
  namespace: cpaas-system
spec:
  clusterRef:
    name: demo-cluster
  datacenter: dc-01
  configs:
    - hostname: worker-01
      network:
        primary:
          ip: 192.0.2.101
          gateway: 192.0.2.1
          dns:
            - 192.0.2.53
          networkName: workload-network
      persistentDisks:
        - name: rook-mon-data
          sizeGiB: 30
          datastore: ceph-datastore
          mountPath: /var/lib/rook
          fsFormat: xfs
          unitNumber: 1
          wipeFilesystem: false
        - name: rook-osd
          sizeGiB: 300
          datastore: ceph-datastore
          unitNumber: 2
          wipeFilesystem: false
    - hostname: worker-02
      network:
        primary:
          ip: 192.0.2.102
          gateway: 192.0.2.1
          dns:
            - 192.0.2.53
          networkName: workload-network
      persistentDisks:
        - name: rook-mon-data
          sizeGiB: 30
          datastore: ceph-datastore
          mountPath: /var/lib/rook
          fsFormat: xfs
          unitNumber: 1
          wipeFilesystem: false
        - name: rook-osd
          sizeGiB: 300
          datastore: ceph-datastore
          unitNumber: 2
          wipeFilesystem: false
    - hostname: worker-03
      network:
        primary:
          ip: 192.0.2.103
          gateway: 192.0.2.1
          dns:
            - 192.0.2.53
          networkName: workload-network
      persistentDisks:
        - name: rook-mon-data
          sizeGiB: 30
          datastore: ceph-datastore
          mountPath: /var/lib/rook
          fsFormat: xfs
          unitNumber: 1
          wipeFilesystem: false
        - name: rook-osd
          sizeGiB: 300
          datastore: ceph-datastore
          unitNumber: 2
          wipeFilesystem: false

The worker VSphereMachineTemplate must reference this pool:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  name: ceph-worker-template
  namespace: cpaas-system
spec:
  template:
    spec:
      machineConfigPoolRef:
        name: ceph-worker-pool

Key requirements:

  • Ceph data disks must not be temporary disks that are deleted with the VM.
  • Persistent disks must be configured in VSphereMachineConfigPool.spec.configs[].persistentDisks[], not only as temporary dataDisks in the MachineTemplate.
  • Configure fixed IPs in VSphereMachineConfigPool.spec.configs[].network.primary.ip. When node replacement reuses the same pool slot, the provider continues to use that IP for the new VM. Each IP must be uniquely available on the network and must not rely on DHCP allocation.
  • Set wipeFilesystem: false and use a fixed unitNumber so that data is retained and device ordering remains stable when the VM is rebuilt.
  • After node replacement, the original OSD data disk must be identified and remounted rather than replaced with a new disk of the same capacity.
  • dataDirHostPath must point to the mount point of the persistent MON/Rook VMDK. Do not format an OSD VMDK and use it for this path.

Bare Metal

Bare Metal environments must provide persistence equivalent to the Immutable Infrastructure node replacement model. Configure the resource in the global cluster, normally in cpaas-system. See Bare Metal Provider APIs, Bare Metal Node Management, and MachineInventoryPool API:

The network fields in this example only express the relationship between nodes and inventory. Use the actual network configuration from your environment. The exact bare-metal network management model is determined by the provider and machine configuration.

This provider binds fixed inventory entries to stable device identities through MachineInventoryPool.spec.machineInventories[].diskBindings[]:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: MachineInventoryPool
metadata:
  name: ceph-worker-pool
  namespace: cpaas-system
spec:
  clusterName: demo-cluster
  machineInventories:
    - name: worker-01
      hostname: worker-01
      diskBindings:
        - volumeName: cpaas-data
          selector:
            byID: /dev/disk/by-id/wwn-0x5000c50000010101
          expected:
            minimumSize: 30Gi
        - volumeName: rook-osd
          selector:
            byID: /dev/disk/by-id/wwn-0x5000c50000010102
          expected:
            minimumSize: 300Gi
    - name: worker-02
      hostname: worker-02
      diskBindings:
        - volumeName: cpaas-data
          selector:
            byID: /dev/disk/by-id/wwn-0x5000c50000010201
          expected:
            minimumSize: 30Gi
        - volumeName: rook-osd
          selector:
            byID: /dev/disk/by-id/wwn-0x5000c50000010202
          expected:
            minimumSize: 300Gi
    - name: worker-03
      hostname: worker-03
      diskBindings:
        - volumeName: cpaas-data
          selector:
            byID: /dev/disk/by-id/wwn-0x5000c50000010301
          expected:
            minimumSize: 30Gi
        - volumeName: rook-osd
          selector:
            byID: /dev/disk/by-id/wwn-0x5000c50000010302
          expected:
            minimumSize: 300Gi

The BaremetalMachineTemplate references this pool through machineInventoryPoolRef and declares mount and reclaim policies in storage.volumes[]:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: BaremetalMachineTemplate
metadata:
  name: ceph-worker-template
  namespace: cpaas-system
spec:
  template:
    spec:
      machineInventoryPoolRef:
        name: ceph-worker-pool
      storage:
        volumes:
          - name: cpaas-data
            filesystem:
              type: xfs
              initializePolicy: IfEmpty
              adoptPolicy: IfMatch
            mount:
              path: /var/lib/rook
              required: true
              options: [noatime]
            reclaimPolicy: Retain
          - name: rook-osd
            reclaimPolicy: Retain

The inventory and diskBindings.selector.byID preserve persistent-disk identity, while storage.volumes[].reclaimPolicy: Retain prevents data disks from being reclaimed during node reconfiguration. In this example, cpaas-data is the MON/Rook data-directory disk and rook-osd is an independent OSD disk. When the provider presents the OSD disk as a raw block device, do not format it or use it for dataDirHostPath. The example mounts the MON data disk at /var/lib/rook; configure CephCluster.spec.dataDirHostPath: /var/lib/rook explicitly.

The bare-metal provider must rediscover the original OSD disk by its stable device identity and preserve it during node reinstallation or replacement. Use the MachineInventoryPool API and Bare Metal Node Management procedures for device discovery and cleanup policies.

Pre-installation Checklist

Before installing Rook-Ceph, verify the following items for Immutable Infrastructure:

  • The provider manages OSD disks as persistent raw block disks, separate from the node system disk.
  • The MON/Rook data directory is configured as dataDirHostPath: /var/lib/rook for the supported Internal Mode flow, or explicitly configured through YAML/API when using another path; it is on a provider-managed persistent file system, is writable on the node, and does not use a raw OSD device.
  • The stable identity and remount process for data disks after node replacement have been confirmed.
  • The worker MachineDeployment uses maxSurge: 0 and maxUnavailable: 1.
  • No finite nodeDrainTimeout is configured, managePodBudgets: true is retained for newly created built-in storage clusters, and the PDB, Eviction API, and Rook disruption controller protection path is preserved.
  • The Immutable Infrastructure resource configuration documentation for the target provider version has been reviewed.

After these conditions are met, create the storage resources in the target business cluster and continue with Deploying in Internal Mode. The provider MachineTemplate, ConfigPool, and MachineDeployment resources described above belong to the global cluster, normally in cpaas-system; the StorageCluster/CephCluster is created in the business cluster. The Internal Mode UI uses /var/lib/rook; use YAML/API configuration instead if you need a custom dataDirHostPath.