GKE Cluster Secrets Not KMS-Encrypted: Why It Matters and How to Fix It

Kubernetes Secrets are the default home for database passwords, API tokens, TLS private keys, and service account credentials. On GKE, those Secrets live in etcd, the cluster's backing store. By default Google encrypts etcd at rest at the infrastructure level, but the Secret payloads themselves are only base64-encoded inside the key-value store. Anyone who reaches the etcd data, or who can read Secrets through the API without the right guardrails, sees the cleartext.

Application-layer secret encryption closes that gap. It uses a Cloud KMS key you own to encrypt Secret data before it is written to etcd. This is an envelope encryption setup: GKE generates a data encryption key, encrypts your Secrets with it, then encrypts that data key with your KMS key encryption key. The gke_clusterencryption check fires when a cluster has no such configuration.

---

What this check detects

The check inspects each GKE cluster's databaseEncryption setting. When that field is absent or set to DECRYPTED, the cluster relies only on Google-managed infrastructure encryption for etcd. There is no customer-managed KMS key in the path, and no application-layer envelope encryption for Secret objects.

A passing cluster has databaseEncryption.state set to ENCRYPTED with a valid keyName pointing at a Cloud KMS key.

---

Why it matters

Base64 is encoding, not encryption. Run base64 -d and the secret is back in cleartext. The risk is not theoretical:

• etcd exposure. If an attacker, an insider, or a misconfigured backup process gains read access to the etcd store, every Secret is recoverable. With KMS encryption enabled, the data is unreadable without permission to use the key.
• Blast radius from a single leaked credential. A GKE cluster commonly holds dozens of Secrets. One compromised etcd snapshot can hand over database passwords, third party API keys, and TLS keys all at once.
• Key access becomes an auditable control. When Secrets are KMS-encrypted, every decrypt operation is gated by IAM on the key and logged in Cloud Audit Logs. You gain a clear paper trail and a kill switch: disable the key and the encrypted data is inert.
• Compliance expectations. Frameworks like PCI DSS, HIPAA, SOC 2, and CIS GKE Benchmark expect sensitive data at rest to be encrypted with managed keys. CIS GKE 4.6.5 specifically calls for KMS-based secret encryption.

Treat etcd as if it will eventually be copied somewhere you did not intend. Application-layer encryption is what makes that copy worthless.

---

How to fix it

The fix has two parts: create a Cloud KMS key (if you do not already have one), grant the GKE service agent permission to use it, then enable database encryption on the cluster.

1. Create a KMS key ring and key

# Set your variables
PROJECT_ID="my-project"
LOCATION="us-central1"   # match your cluster region
KEYRING="gke-secrets"
KEY="gke-secrets-key"

# Create the key ring
gcloud kms keyrings create "$KEYRING" \
  --location="$LOCATION" \
  --project="$PROJECT_ID"

# Create the key
gcloud kms keys create "$KEY" \
  --location="$LOCATION" \
  --keyring="$KEYRING" \
  --purpose="encryption" \
  --project="$PROJECT_ID"

2. Grant the GKE service agent access to the key

GKE uses a Google-managed service account, the Kubernetes Engine Service Agent, to call KMS on the cluster's behalf. Grant it the encrypter/decrypter role on the key.

PROJECT_NUMBER=$(gcloud projects describe "$PROJECT_ID" --format="value(projectNumber)")
GKE_SA="service-${PROJECT_NUMBER}@container-engine-robot.iam.gserviceaccount.com"

gcloud kms keys add-iam-policy-binding "$KEY" \
  --location="$LOCATION" \
  --keyring="$KEYRING" \
  --member="serviceAccount:${GKE_SA}" \
  --role="roles/cloudkms.cryptoKeyEncrypterDecrypter" \
  --project="$PROJECT_ID"

3. Enable database encryption on the cluster

You can enable this on an existing cluster without recreating it.

KEY_NAME="projects/${PROJECT_ID}/locations/${LOCATION}/keyRings/${KEYRING}/cryptoKeys/${KEY}"

gcloud container clusters update my-cluster \
  --region="$LOCATION" \
  --database-encryption-key="$KEY_NAME" \
  --project="$PROJECT_ID"

Verify the change

gcloud container clusters describe my-cluster \
  --region="$LOCATION" \
  --project="$PROJECT_ID" \
  --format="value(databaseEncryption.state, databaseEncryption.keyName)"
# Expected: ENCRYPTED  projects/.../cryptoKeys/gke-secrets-key

Terraform

For clusters managed as code, set the database_encryption block:

resource "google_kms_key_ring" "gke" {
  name     = "gke-secrets"
  location = "us-central1"
}

resource "google_kms_crypto_key" "gke_secrets" {
  name     = "gke-secrets-key"
  key_ring = google_kms_key_ring.gke.id
  purpose  = "ENCRYPT_DECRYPT"
}

resource "google_kms_crypto_key_iam_member" "gke_sa" {
  crypto_key_id = google_kms_crypto_key.gke_secrets.id
  role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
  member        = "serviceAccount:service-${data.google_project.current.number}@container-engine-robot.iam.gserviceaccount.com"
}

resource "google_container_cluster" "primary" {
  name     = "my-cluster"
  location = "us-central1"

  database_encryption {
    state    = "ENCRYPTED"
    key_name = google_kms_crypto_key.gke_secrets.id
  }

  depends_on = [google_kms_crypto_key_iam_member.gke_sa]
}

---

How to prevent it from happening again

One-off fixes drift. Bake the requirement into the path that creates clusters.

Policy as code with OPA Gatekeeper or Kyverno

If you provision clusters through a GitOps pipeline that applies Terraform plans, use a Conftest policy to reject any cluster without encryption:

package gke.encryption

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "google_container_cluster"
  not resource.change.after.database_encryption[_].state == "ENCRYPTED"
  msg := sprintf("Cluster '%s' must enable KMS database encryption", [resource.address])
}

CI/CD gate

Run the policy check before any plan reaches apply:

terraform plan -out=tfplan
terraform show -json tfplan > plan.json
conftest test plan.json --policy ./policies
# Non-zero exit blocks the merge

Organization policy

For broader control, use Cloud Organization Policy to require customer-managed encryption keys, or enforce GKE configuration with Policy Controller across the fleet. This catches clusters created outside your IaC pipeline, which is where most drift starts.

---

Best practices

• Rotate the KMS key on a schedule. Set automatic rotation (for example every 90 days) on the key. GKE re-encrypts the data key with each new version, and old versions remain available to decrypt existing data.
• Separate the key project from the cluster project. Hosting KMS keys in a dedicated security project limits who can manage or destroy keys, and keeps key administration separate from workload teams.
• Lock down key IAM tightly. Only the GKE service agent needs encrypter/decrypter on the key. Avoid granting broad roles/cloudkms.admin to application teams.
• Do not store everything in Kubernetes Secrets. For high-value or frequently rotated credentials, consider Secret Manager with Workload Identity, so secrets are fetched at runtime rather than persisted in etcd at all. KMS-encrypted Secrets and Secret Manager solve overlapping problems and work well together.
• Audit decrypt operations. Enable Cloud Audit Logs for Cloud KMS and alert on unexpected decrypt patterns. A spike in decrypt calls can signal data exfiltration in progress.

Application-layer secret encryption is a low-effort, high-value control. It takes a few commands to enable, costs almost nothing, and turns a stolen etcd snapshot from a full credential dump into an inert blob. If the check is firing, treat it as a quick win and move it to the top of your remediation list.