Pub/Sub Subscription Has No Dead Letter Topic

Pub/Sub is the backbone of a lot of event-driven systems on GCP. It buffers messages between producers and consumers, retries delivery when a subscriber fails to ack, and generally hides a lot of distributed systems pain. But that retry behavior has a dark side: a single message your subscriber cannot process will be redelivered over and over, sometimes for days. Without a dead letter topic, you have no safety net for those messages.

This Lensix check, pubsub_nodeadletter, looks at every Pub/Sub subscription in your GCP projects and flags the ones that have no dead letter topic configured.

---

What this check detects

The check inspects each subscription's deadLetterPolicy field. If that field is empty, the subscription has nowhere to send messages that have exhausted their delivery attempts. The check raises a finding for that subscription.

A dead letter policy has two parts:

• Dead letter topic — the Pub/Sub topic that undeliverable messages get forwarded to.
• Max delivery attempts — how many times Pub/Sub tries to deliver a message before giving up and forwarding it to the dead letter topic. Valid values are 5 through 100.

When both are set, a message that fails to be acked after the configured number of attempts is published to the dead letter topic instead of being redelivered indefinitely.

---

Why it matters

Without a dead letter topic, a message that your subscriber cannot handle has two possible fates, and both are bad.

Poison messages get redelivered forever

Say a producer publishes a malformed payload, or a message references a record that was deleted, or there is a bug in your deserialization code. Your subscriber nacks the message every time. Pub/Sub, doing exactly what it is designed to do, keeps the message in the backlog and redelivers it according to your retry policy until the message expires (the default retention is 7 days, configurable up to 31 days).

In the meantime that single bad message can:

• Trigger the same error and the same alert thousands of times, drowning out real signal.
• Burn CPU and egress as your subscriber repeatedly pulls, fails, and nacks it.
• Block ordered delivery. If you use message ordering keys, a stuck message holds up every message behind it on the same key.

Messages disappear with no audit trail

Once a message hits the retention window, Pub/Sub deletes it. If that message represented a payment event, an audit log, or a provisioning request, it is simply gone. There is no record that it ever failed, and no copy to replay once you fix the bug. For regulated workloads this is a real compliance problem, because you cannot prove what happened to data you no longer have.

The business impact

Dead letter topics turn an invisible, unbounded failure mode into a bounded, observable one. Instead of "messages are being lost somewhere," you get a dedicated topic you can alert on, inspect, and replay. That difference matters most during an incident, when you are trying to figure out what data was affected and whether you can recover it.

---

How to fix it

Fixing this is a three-step process: create a dead letter topic, give Pub/Sub permission to publish to it, and attach the dead letter policy to your subscription. Skipping the IAM step is the most common mistake, so do not skip it.

Step 1: Create the dead letter topic

gcloud pubsub topics create my-subscription-dead-letter \
  --project=my-project

It is also good practice to create a subscription on the dead letter topic so messages there are retained and you can pull them later for inspection.

gcloud pubsub subscriptions create my-subscription-dead-letter-sub \
  --topic=my-subscription-dead-letter \
  --project=my-project

Step 2: Grant the Pub/Sub service account permission

Pub/Sub forwards dead-lettered messages using a Google-managed service account specific to your project. That account needs the Publisher role on the dead letter topic and the Subscriber role on the source subscription.

# Get your project number
PROJECT_NUMBER=$(gcloud projects describe my-project --format="value(projectNumber)")
SERVICE_ACCOUNT="service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com"

# Allow Pub/Sub to publish to the dead letter topic
gcloud pubsub topics add-iam-policy-binding my-subscription-dead-letter \
  --member="serviceAccount:${SERVICE_ACCOUNT}" \
  --role="roles/pubsub.publisher" \
  --project=my-project

# Allow Pub/Sub to ack messages on the source subscription
gcloud pubsub subscriptions add-iam-policy-binding my-subscription \
  --member="serviceAccount:${SERVICE_ACCOUNT}" \
  --role="roles/pubsub.subscriber" \
  --project=my-project

Step 3: Attach the dead letter policy to the subscription

gcloud pubsub subscriptions update my-subscription \
  --dead-letter-topic=my-subscription-dead-letter \
  --max-delivery-attempts=5 \
  --project=my-project

Set --max-delivery-attempts to a value that gives transient failures room to recover but does not let a poison message thrash for too long. A value between 5 and 10 works for most workloads.

Doing it in the console

1. Open Pub/Sub → Subscriptions and click the subscription.
2. Click Edit.
3. Under Dead lettering, check Enable dead lettering.
4. Select or create the dead letter topic and set the maximum delivery attempts.
5. The console offers to grant the required IAM roles automatically. Accept that prompt, then click Update.

Terraform

If you manage infrastructure as code, the policy lives in the google_pubsub_subscription resource. Here is a complete example including the topic and IAM bindings.

resource "google_pubsub_topic" "dead_letter" {
  name = "my-subscription-dead-letter"
}

resource "google_pubsub_subscription" "main" {
  name  = "my-subscription"
  topic = google_pubsub_topic.main.id

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.dead_letter.id
    max_delivery_attempts = 5
  }
}

data "google_project" "current" {}

resource "google_pubsub_topic_iam_member" "dead_letter_publisher" {
  topic  = google_pubsub_topic.dead_letter.id
  role   = "roles/pubsub.publisher"
  member = "serviceAccount:service-${data.google_project.current.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}

resource "google_pubsub_subscription_iam_member" "subscriber" {
  subscription = google_pubsub_subscription.main.name
  role         = "roles/pubsub.subscriber"
  member       = "serviceAccount:service-${data.google_project.current.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}

---

How to prevent it from happening again

A one-time fix does not stop the next subscription from shipping without a dead letter topic. Bake the requirement into the places where infrastructure is defined and reviewed.

Policy as code with OPA / Conftest

If you run Terraform through CI, add a policy check on the plan output. This Rego rule fails any subscription that lacks a dead letter policy.

package pubsub

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "google_pubsub_subscription"
  not resource.change.after.dead_letter_policy
  msg := sprintf("Subscription '%s' has no dead_letter_policy", [resource.change.after.name])
}

Organization policy and gatekeeping

• Run this Lensix check on a schedule so any drift, including subscriptions created by hand or by other teams, surfaces quickly.
• Add a pre-merge gate in your pipeline that blocks Terraform changes introducing a subscription without dead_letter_policy.
• If you use a service catalog or internal platform for creating Pub/Sub resources, make the dead letter topic non-optional in the request form.

---

Best practices

• Always pair a dead letter topic with a subscription. Messages forwarded to a topic with no subscription expire on the topic's retention schedule. Create a subscription so dead-lettered messages stick around for inspection.
• Tune retry policy alongside max delivery attempts. Use exponential backoff on the source subscription so transient failures get time to recover before the message counts toward the delivery limit.
• Monitor the dead letter topic actively. Treat any message landing there as a signal that something needs attention, not as a place messages go to be forgotten.
• Build a replay path. Once you have fixed the underlying bug, you want a way to move messages from the dead letter subscription back to the main topic. A small script or Cloud Function that reads from the dead letter subscription and republishes is usually enough.
• Avoid making the dead letter topic point at itself. A subscription on the dead letter topic should not use that same topic as its own dead letter destination, or you create a redelivery loop.
• Use separate dead letter topics per subscription. Sharing one dead letter topic across many subscriptions makes it hard to tell where a failed message came from.

A dead letter topic does not fix bad messages. It makes them visible and recoverable, which is the difference between a five-minute investigation and a postmortem about data you can no longer find.

Configuring dead letter policies is cheap, the IAM setup is the only fiddly part, and the payoff is a system that fails loudly and recoverably instead of silently and permanently. Run the pubsub_nodeadletter check across your projects, fix the subscriptions it flags, and add a policy gate so the problem does not come back.