CloudTrail Log Delivery Failing: Diagnose and Fix the Gap in Your Audit Trail

CloudTrail is the system of record for almost everything that happens in an AWS account. Every API call, every console login, every IAM change leaves a trace there. When a trail stops delivering logs, that record goes dark, and you often do not find out until you actually need the data, which is the worst possible time to discover it.

The CloudTrail Log Delivery Failing check looks at each trail's delivery status and raises a finding when CloudTrail reports that it cannot write to its configured S3 bucket. This post covers what triggers it, why a silent logging gap is dangerous, and how to fix and prevent it.

---

What this check detects

CloudTrail records two timestamps for the health of each trail: the last successful log delivery and the last delivery attempt. It also surfaces an error message when an attempt fails. This check inspects the trail status and fires when:

• The most recent delivery attempt returned an error (for example AccessDenied or NoSuchBucket).
• There is a significant gap between the last delivery attempt and the last successful delivery.

You can see the same data CloudTrail uses with a single call:

aws cloudtrail get-trail-status \
  --name management-events-trail \
  --query '{LastDeliveryError:LatestDeliveryError, LastDelivery:LatestDeliveryTime, LastAttempt:LatestDeliveryAttemptTime}'

A healthy trail returns an empty LatestDeliveryError and a LatestDeliveryTime that is recent. If LatestDeliveryError contains text, that string is the root cause, and it is usually one of a handful of issues.

---

Why it matters

A failing trail is not a cosmetic problem. It removes the evidence trail that security, compliance, and operations all depend on.

You lose visibility during the exact window an attacker wants you blind

A common step in cloud attacks is to disrupt logging. An attacker with sufficient permissions might modify the S3 bucket policy, delete the bucket, or remove the KMS key grant so that delivery quietly fails while CloudTrail still reports as enabled. From the outside the trail looks fine. Inside, nothing is being written. Any actions taken after that point leave no durable record in S3.

Compliance frameworks assume the logs exist

PCI DSS, SOC 2, HIPAA, and CIS AWS Foundations all require continuous, retained audit logging. A trail that stops delivering creates a gap that you cannot reconstruct later. During an audit or a breach investigation, "we had a few weeks where logs were not being written" is not an acceptable answer.

Incident response falls apart without source data

When something goes wrong, the first question is usually "who did what, and when." If delivery has been failing, the answer for the affected period simply does not exist. You cannot retroactively generate CloudTrail history that was never delivered.

---

How to fix it

Start by reading the error string, because it tells you which of the common causes you are dealing with.

aws cloudtrail get-trail-status --name management-events-trail \
  --query 'LatestDeliveryError' --output text

Cause 1: The S3 bucket policy denies CloudTrail

The most frequent cause is a bucket policy that no longer grants the CloudTrail service principal write access. This happens when someone tightens the policy, applies an organization-wide deny, or replaces the policy entirely. Fetch the current policy:

aws s3api get-bucket-policy \
  --bucket my-cloudtrail-logs \
  --query Policy --output text | jq .

CloudTrail needs two grants: permission to check the bucket ACL, and permission to write objects under the trail prefix with the bucket-owner-full-control ACL. A correct policy looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSCloudTrailAclCheck",
      "Effect": "Allow",
      "Principal": { "Service": "cloudtrail.amazonaws.com" },
      "Action": "s3:GetBucketAcl",
      "Resource": "arn:aws:s3:::my-cloudtrail-logs",
      "Condition": {
        "StringEquals": {
          "aws:SourceArn": "arn:aws:cloudtrail:us-east-1:111122223333:trail/management-events-trail"
        }
      }
    },
    {
      "Sid": "AWSCloudTrailWrite",
      "Effect": "Allow",
      "Principal": { "Service": "cloudtrail.amazonaws.com" },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-cloudtrail-logs/AWSLogs/111122223333/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control",
          "aws:SourceArn": "arn:aws:cloudtrail:us-east-1:111122223333:trail/management-events-trail"
        }
      }
    }
  ]
}

Apply the corrected policy:

aws s3api put-bucket-policy \
  --bucket my-cloudtrail-logs \
  --policy file://cloudtrail-bucket-policy.json

Cause 2: The bucket was deleted or renamed

If the error is NoSuchBucket, the target no longer exists. You either recreate a bucket with the original name and the correct policy, or point the trail at a new bucket:

aws cloudtrail update-trail \
  --name management-events-trail \
  --s3-bucket-name my-cloudtrail-logs-new

Cause 3: KMS key permissions are missing

If the bucket uses SSE-KMS, CloudTrail needs kms:GenerateDataKey* on the key. A revoked grant or an edited key policy produces a KMS-related delivery error. Add CloudTrail to the key policy:

{
  "Sid": "AllowCloudTrailEncrypt",
  "Effect": "Allow",
  "Principal": { "Service": "cloudtrail.amazonaws.com" },
  "Action": "kms:GenerateDataKey*",
  "Resource": "*",
  "Condition": {
    "StringLike": {
      "kms:EncryptionContext:aws:cloudtrail:arn": "arn:aws:cloudtrail:*:111122223333:trail/*"
    }
  }
}

Cause 4: A Block Public Access or Object Lock setting interferes

Rarely, an Object Lock retention rule or an overly aggressive SCP blocks the write. Check the trail's region against the bucket region as well, since a region mismatch in older setups can cause issues. Once you believe the cause is resolved, force a fresh delivery attempt by toggling logging:

aws cloudtrail stop-logging  --name management-events-trail
aws cloudtrail start-logging --name management-events-trail

# Wait a few minutes, then confirm recovery
aws cloudtrail get-trail-status --name management-events-trail \
  --query '{Error:LatestDeliveryError, Last:LatestDeliveryTime}'

When LatestDeliveryError comes back empty and LatestDeliveryTime updates, the trail is healthy again.

---

How to prevent it from happening again

Most delivery failures come from someone changing a bucket policy, a KMS key, or a bucket without realizing CloudTrail depended on it. Prevention is mostly about making those resources hard to break and loud when they do.

Manage the trail and its bucket as code

Defining the trail, bucket, and policy together in Terraform keeps the dependency explicit and reviewable. Anyone who weakens the policy has to do it through a pull request.

resource "aws_cloudtrail" "main" {
  name                          = "management-events-trail"
  s3_bucket_name                = aws_s3_bucket.cloudtrail.id
  include_global_service_events = true
  is_multi_region_trail         = true
  enable_log_file_validation    = true
  kms_key_id                    = aws_kms_key.cloudtrail.arn

  depends_on = [aws_s3_bucket_policy.cloudtrail]
}

Alert on delivery health, not just on the trail being on

Create a CloudWatch alarm that watches the time since the last successful delivery, or run a scheduled check that calls get-trail-status and pages you when LatestDeliveryError is non-empty. This is exactly the kind of continuous check Lensix runs for you, so a broken trail surfaces within minutes rather than during an audit.

Protect the bucket policy with policy-as-code

Add an OPA or Conftest rule to your pipeline that rejects any change which removes the CloudTrail service principal from the bucket policy:

# Run in CI before applying infrastructure changes
conftest test bucket-policy.json --policy ./policies/cloudtrail

An SCP that prevents anyone other than your pipeline role from modifying CloudTrail configuration and its logging bucket stops accidental and malicious tampering at the org level.

---

Best practices

• Use a dedicated, locked-down logging account. Send organization trail logs to a central S3 bucket in a separate account that almost no one can touch. This limits blast radius and prevents a workload account from breaking its own logging.
• Enable an organization trail. A single org-level trail covers all member accounts and is harder for an individual account owner to disable.
• Turn on log file validation everywhere. It is free and gives you tamper evidence.
• Apply MFA delete and a deny-delete policy on the logging bucket. Making the destination hard to remove removes one of the most common failure causes.
• Monitor delivery continuously. A trail you set up two years ago and never looked at again is exactly the kind that fails silently. Automated checking is the only reliable way to catch it.

The cost of fixing a delivery failure is a few minutes of policy editing. The cost of discovering one during an incident is a permanent hole in your audit history. Treat the gap between "last attempt" and "last success" as a number that should always be near zero, and wire up automation so you never have to remember to look.