GCP: Alert on Custom IAM Role Changes | Lensix

TL;DR

Without a log-based alert on custom IAM role creation and updates, an attacker or insider can quietly craft privileged roles and you will never hear about it. Create a logging metric on iam.roles.create and iam.roles.update activity and wire it to an alerting policy.

Custom IAM roles are one of the quietest ways to escalate privileges in a GCP project. They look administrative and boring, which is exactly why they get ignored. This check flags projects where no log-based alert exists to notify you when someone creates or modifies a custom role. If that gap exists, a change that grants a service account a sensitive permission can slip through with no one watching.

What this check detects

Lensix inspects your GCP project for a log-based metric and an associated alerting policy that fire when custom IAM role activity appears in Cloud Audit Logs. Specifically, it looks for monitoring coverage of these Admin Activity log events:

google.iam.admin.v1.CreateRole — a new custom role is created
google.iam.admin.v1.UpdateRole — an existing custom role's permission set is changed
google.iam.admin.v1.DeleteRole and UndeleteRole — roles removed or restored

If there is no metric filtering on these methods, or there is a metric but no alerting policy attached to it, the check fails. The logs themselves are almost always present (Admin Activity logging is on by default and cannot be turned off), so this is purely about whether anyone gets notified.

Note: Cloud Audit Logs split into Admin Activity, Data Access, System Event, and Policy Denied streams. IAM role changes are Admin Activity events, which means they are captured for free and retained for 400 days by default. You are not missing data, you are missing the trigger that turns that data into a page.

Why it matters

Custom roles are a privilege escalation vector that defenders frequently overlook. Consider a few realistic scenarios.

The slow privilege creep

A developer has permission to manage IAM roles in a project. They create a custom role to "make their service account work" and bundle in iam.serviceAccounts.getAccessToken or resourcemanager.projects.setIamPolicy. That role gets bound to a service account, and now anything that can act as that account can impersonate further or rewrite the project's IAM policy. No deny, no warning, just a working pipeline and a wide open door.

The attacker who already has a foothold

An attacker who compromises a principal with iam.roles.create permission rarely uses obviously malicious-looking grants. Instead they modify an existing, legitimately named custom role and add a single dangerous permission. UpdateRole events are easy to miss in a noisy log stream. Without an alert, the change persists until your next manual IAM review, which might be months away.

Warning: Editing a custom role applies the change to every principal already bound to it. A single UpdateRole call can escalate privileges across dozens of service accounts and users at once, with no per-binding audit trail beyond the role change itself.

Compliance and audit expectations

The CIS Google Cloud Foundations Benchmark explicitly recommends a log metric filter and alert for custom role changes (control 2.6 in recent versions). If you are pursuing SOC 2, ISO 27001, or PCI DSS, "we monitor changes to access control definitions" is a control you will be asked to evidence. An unalerted role change is a finding waiting to happen during an audit.

How to fix it

Remediation has two parts: create a logging metric that counts custom role events, then attach an alerting policy that notifies a channel when the count goes above zero.

Step 1: Create the log-based metric (gcloud)

gcloud logging metrics create custom-iam-role-changes \
  --description="Custom IAM role create, update, or delete events" \
  --project=YOUR_PROJECT_ID \
  --log-filter='resource.type="iam_role"
    AND (protoPayload.methodName="google.iam.admin.v1.CreateRole"
    OR protoPayload.methodName="google.iam.admin.v1.UpdateRole"
    OR protoPayload.methodName="google.iam.admin.v1.DeleteRole")'

This defines a counter metric that increments every time one of the listed methods is recorded. Swap YOUR_PROJECT_ID for your actual project.

Step 2: Create a notification channel (if you do not have one)

gcloud beta monitoring channels create \
  --display-name="Security On-Call" \
  --type=email \
  [email protected] \
  --project=YOUR_PROJECT_ID

Note the channel ID returned, you will reference it in the next step.

Step 3: Create the alerting policy

Save the following as iam-role-alert-policy.json:

{
  "displayName": "Alert on Custom IAM Role Changes",
  "combiner": "OR",
  "conditions": [
    {
      "displayName": "Custom IAM role change detected",
      "conditionThreshold": {
        "filter": "metric.type=\"logging.googleapis.com/user/custom-iam-role-changes\" AND resource.type=\"iam_role\"",
        "comparison": "COMPARISON_GT",
        "thresholdValue": 0,
        "duration": "0s",
        "aggregations": [
          {
            "alignmentPeriod": "60s",
            "perSeriesAligner": "ALIGN_COUNT"
          }
        ]
      }
    }
  ],
  "notificationChannels": [
    "projects/YOUR_PROJECT_ID/notificationChannels/CHANNEL_ID"
  ]
}

Then apply it:

gcloud alpha monitoring policies create \
  --policy-from-file=iam-role-alert-policy.json \
  --project=YOUR_PROJECT_ID

Console alternative

Go to Logging > Log-based Metrics and click Create Metric.
Set type to Counter and paste the log filter from Step 1.
Open Monitoring > Alerting and click Create Policy.
Select your new metric, set the condition to any time series is above 0, and choose your notification channel.

Tip: Use an ALIGN_COUNT aligner with a 60s period and a threshold of 0 so the alert fires on the first event rather than waiting for a sustained rate. IAM role changes are rare enough that one event is worth a notification.

How to prevent it from happening again

Fixing one project by hand does not scale. Bake the metric and alert into your infrastructure as code so every new project inherits the coverage.

Terraform

resource "google_logging_metric" "custom_iam_role_changes" {
  name    = "custom-iam-role-changes"
  project = var.project_id
  filter  = <<-EOT
    resource.type="iam_role"
    AND (protoPayload.methodName="google.iam.admin.v1.CreateRole"
    OR protoPayload.methodName="google.iam.admin.v1.UpdateRole"
    OR protoPayload.methodName="google.iam.admin.v1.DeleteRole")
  EOT

  metric_descriptor {
    metric_kind = "DELTA"
    value_type  = "INT64"
  }
}

resource "google_monitoring_alert_policy" "custom_iam_role_changes" {
  project      = var.project_id
  display_name = "Alert on Custom IAM Role Changes"
  combiner     = "OR"

  conditions {
    display_name = "Custom IAM role change detected"
    condition_threshold {
      filter          = "metric.type=\"logging.googleapis.com/user/${google_logging_metric.custom_iam_role_changes.name}\" AND resource.type=\"iam_role\""
      comparison      = "COMPARISON_GT"
      threshold_value = 0
      duration        = "0s"
      aggregations {
        alignment_period   = "60s"
        per_series_aligner = "ALIGN_COUNT"
      }
    }
  }

  notification_channels = [var.security_channel_id]
}

Put this in a shared module and require it for every project factory you run. New projects then ship with monitoring already wired up.

Tip: If you manage your org with the Google Cloud project factory or a landing zone, add this module to the baseline that every project consumes. That turns "we should monitor IAM roles" from a checklist item into a default you cannot forget.

Policy as code in CI

Add an OPA or Conftest gate to your Terraform pipeline that fails the plan if a project module does not include a custom role logging metric. A simple Rego rule can check that every google_logging_metric set for a project includes one filtering on iam.admin role methods. This stops a teammate from spinning up a project without the alert in the first place.

Note: You can also centralize this. Route all project audit logs into a security-managed log bucket or sink and build the metric and alert once at the aggregation point, rather than per project. That trades a little setup complexity for guaranteed coverage across the org.

Best practices

Alert on the whole IAM verb family, not just create. UpdateRole is the sneakiest event because it mutates an existing trusted role. Make sure your filter covers create, update, and delete.
Include the actor in your notification. Configure the alert to surface protoPayload.authenticationInfo.principalEmail so responders immediately know who made the change without digging through logs.
Pair the alert with periodic review. An alert tells you about new changes. A scheduled review of existing custom roles catches drift that predates your monitoring. Run both.
Prefer predefined roles where possible. Every custom role is a thing you now have to monitor and govern. If a predefined role fits, use it and reduce the surface you need to watch.
Route the alert somewhere a human reads it. An email to a shared inbox no one checks is the same as no alert. Send it to a paged on-call channel or a security Slack with an owner.

Warning: Log-based metrics only count what your filter matches. If you typo a method name or scope the filter to the wrong resource type, the alert will sit silent forever and look healthy. Test it by creating a throwaway custom role and confirming the alert fires, then delete the test role.

Custom IAM role monitoring is cheap to set up and pays for itself the first time someone makes a change they should not have. Wire it into your project baseline once, verify it fires, and move it from your worry list to your defaults.

No Alert for Custom IAM Role Changes in GCP