#etcd Encryption

Install and operate etcd Encryption Manager in to automate etcd data encryption key rotation within your clusters.

It ensures that sensitive data stored in etcd, such as secrets and configmaps, is encrypted using a secure algorithm, enhancing your cluster's security.

#Key Strategies

The plugin supports two key strategies. Pick the one that matches your cluster role.

In deterministic mode, the runtime Active / Standby role is detected automatically; it is not a manual install parameter.

#Installation

The plugin supports two installation paths depending on the target cluster:

• Workload / DCS clusters — use the default random key strategy.
• Global clusters — must use the deterministic key strategy (see Global Clusters (Deterministic Mode) below).

See Cluster Plugin for the general plugin installation workflow.

#Workload / DCS Clusters (random strategy)

• Supported cluster types: On-Premises, DCS.
• No additional installation parameters are required — install via the standard Cluster Plugin workflow.

#Global Clusters (Deterministic Mode)

#Prerequisites

• The cluster must be a global cluster. When global.cluster.name is global, the plugin enforces etcdEncryption.keyStrategy=deterministic; the plugin configuration UI also defaults the field to deterministic for clusters labeled is-global=true.

#Plugin Parameters

The plugin form exposes the following parameters for deterministic installation.

The rootSecretRef.namespace is fixed to kube-system and is not exposed in the form.

#key_strategy

For global clusters this must be deterministic.

#replication_group_id

Recommendations:

• For DR clusters built with the etcd Synchronizer, both sides of the same DR group must share the same replication_group_id.
• To avoid leaking real business, environment, or cluster identity, do not reuse the cluster name even when the UI prefills it. Plan a separate opaque identifier ahead of installation.
• A UUID-style identifier may be a good fit, for example 6f1b9e2c-7c3a-4a9c-8b72-2fd0f8f3c1ab.

To generate a UUID-style value from the command line:

openssl rand -hex 16 | sed -E 's/^(.{8})(.{4})(.{4})(.{4})(.{12})$/\1-\2-\3-\4-\5/'

#root_secret_name

The plugin form exposes only the Secret name; the namespace is always kube-system at render time.

#Preparing the Root Secret

The deterministic root Secret must exist in kube-system before installing the plugin. Prepare it in two steps:

1. Generate 32 bytes of cryptographically strong random material into ./root-key.bin. The bytes must come from a high-entropy source — typically the operating system CSPRNG. openssl rand is a safe default; equivalent OS-CSPRNG-backed commands such as head -c 32 /dev/urandom > ./root-key.bin also work. Do not use shell $RANDOM, language-level rand(), or any time-seeded generator.

   openssl rand -out ./root-key.bin 32

2. Create the Secret in kube-system from that file:

   kubectl -n kube-system create secret generic etcd-derivation-root \
     --from-file=root-key=./root-key.bin

If you use a different Secret name, replace etcd-derivation-root above with the value you will enter as root_secret_name.

#Advanced (chart values, not exposed in the plugin form)

#How it Works

Upon installation, an etcd-encryption-manager controller is deployed in the kube-system namespace, which:

• Periodically rotates etcd data encryption keys.
• Retains the 8 most recent keys for rollback compatibility.
• Updates encryption configurations on all control nodes.
• Triggers kube-apiserver to hot reload new keys.
• Automatically migrates resources to re-encrypt data with new keys.

Cluster stability is maintained throughout these operations.

When the plugin is installed in deterministic mode on a global DR pair, the controller additionally:

• Detects whether it is acting as Active or Standby at runtime — this is not a manual configuration.
• On the Active side, generates the SeedBundle and lets the etcd Synchronizer replicate it to the Standby.
• On the Standby side, derives the same per-revision keys from the shared SeedBundle and the root Secret, so a failover produces identical encryption keys without manual key copy.

#Default Configuration

#Operations Guide

#Configuration Files

#Checking Status

Run the following command to check the current rotation status:

kubectl get EtcdEncryptionConfig default -o yaml

Example output:

apiVersion: cluster.alauda.io/v1alpha1
kind: EtcdEncryptionConfig
metadata:
  name: default
spec:
  resources:
    - secrets
    - configmaps
  rotationInterval: 168h0m0s
  type: aesgcm
status:
  deployStatus:
    192.168.100.1:
      revision: 3
      state: Success
    192.168.100.2:
      revision: 3
      state: Success
    192.168.100.3:
      revision: 3
      state: Success
  migration:
    completeTimestamp: "2025-05-27T05:47:01Z"
    resources:
      - secrets
      - configmaps
    revision: 3
    state: Success
  revision: 3

#Approving a Deterministic Key Rotation

When the encryption manager rotates a key in deterministic mode, the new revision enters a WaitingApproval phase on the Active cluster. The key does not take effect until the operator approves it. This gate exists because the Active cluster must not encrypt data with a key that the Standby cluster has not yet loaded — otherwise a failover would leave Standby unable to decrypt the new ciphertext.

Approval is always performed on the Active cluster only. Do not patch the Standby's EtcdKeySeedBundleState, and do not edit any resource's status field directly.

#Manual Approval (Recommended)

Step 1 — Identify the pending revision.

Log in to a control-plane node of the Active cluster and list all bundle states:

kubectl get eksbs \
  -o custom-columns='NAME:.metadata.name,REVISION:.spec.revision,PHASE:.status.phase,APPROVED:.spec.approved'

Find the row with PHASE=WaitingApproval and APPROVED=false. Record its revision number:

REVISION=<revision-number>
STATE="seed-bundle-state-${REVISION}"
BUNDLE="seed-bundle-${REVISION}"

Step 2 — Verify Standby readiness.

Log in to a control-plane node of the Standby cluster and check the same revision:

kubectl get eksbs "$STATE" \
  -o custom-columns='REVISION:.spec.revision,PHASE:.status.phase,PREPARED_AT:.status.preparedAt'

The output must show PHASE=Prepared for the expected revision. Stop if the object is missing, still Preparing, or Failed — investigate the etcd Synchronizer path before proceeding.

Step 3 — Cross-check derivation inputs.

Run the following command on both the Active and Standby clusters and compare the output. The replicationGroupID, dimension, and derivationAlgorithm must match exactly:

kubectl get eksb "$BUNDLE" \
  -o custom-columns='REVISION:.spec.revision,GROUP:.spec.replicationGroupID,DIMENSION:.spec.dimension,ALGORITHM:.spec.derivationAlgorithm'

Step 4 — Approve.

On the Active cluster, approve the pending revision:

kubectl patch eksbs "$STATE" \
  --type=merge -p '{"spec":{"approved":true}}'

The patch records the approval. Activation, deployment to control-plane nodes, and data migration happen asynchronously after this point.

Step 5 — Confirm activation completes.

On the Active cluster, check the bundle state and migration progress:

kubectl get eksbs "$STATE" \
  -o custom-columns='REVISION:.spec.revision,PHASE:.status.phase,MESSAGE:.status.message'
kubectl get eec default \
  -o custom-columns='REVISION:.status.revision,MIGRATION_STATE:.status.migration.state,MIGRATION_REVISION:.status.migration.revision'

The rotation is complete when:

• The bundle state reaches PHASE=Active.
• eec/default shows the new REVISION with MIGRATION_STATE=Success.

To check per-node deployment progress on the Active cluster:

kubectl get eec default \
  -o go-template='{{range $node, $status := .status.deployStatus}}{{printf "%s\t%s\t%d\t%s\n" $node $status.state $status.revision $status.message}}{{end}}'

Every control-plane node must report state=Success for the new revision. A bundle that remains Activating indicates deployment or migration is still in progress (or retrying after a failure).

#Timed Approval (Not Recommended)

When activationPolicy is set to timed-approval, the controller automatically approves the pending revision once approvalDelay elapses after the EtcdKeySeedBundleState is created. The delay must be long enough for the full Standby preparation pipeline to complete:

The default 30m provides margin for typical single-region deployments with a small control plane. Increase it if your setup involves cross-region replication, large control planes, or clusters without automatic provider reload.

#Required monitoring

If you choose timed-approval, you must verify that monitoring and alerting are working on both the Active and Standby clusters before enabling this policy.

The encryption manager exposes the following Prometheus metrics:

#Metrics

#Built-in alert rules

The plugin includes the following alerts:

These alerts fire on each cluster independently. For Global clusters, the critical alert to act on is EtcdEncryptionManagerTimedApprovalDeadlineApproaching on the Active cluster — it gives you a 10-minute window to check whether the Standby is Prepared and switch to manual if it is not.

#Emergency response

If the EtcdEncryptionManagerTimedApprovalDeadlineApproaching alert fires and the Standby is not ready, switch the Active to manual to halt auto-approval. On the Active cluster:

kubectl patch eec default \
  --type=merge -p '{"spec":{"deterministic":{"activationPolicy":"manual"}}}'

Verify the current pending bundle has switched:

kubectl get eksbs \
  -o custom-columns='NAME:.metadata.name,REVISION:.spec.revision,PHASE:.status.phase,APPROVED:.spec.approved'

The bundle must still show APPROVED=false. Switching back to timed-approval later affects only future bundles — the current bundle remains under manual control. If the bundle has already moved to Activating, the deadline has already taken effect; follow the activation checks above and do not assume that changing the policy rolls it back.