Case Study: CloudWatch Alarms That Never Fired

Jul 2, 2026

Context

We run a serverless revenue-cycle backend on AWS. One of its critical paths calls a third-party eligibility API. We emit custom CloudWatch metrics on every call and wire alarms for success rate, error rate, timeouts, and p95 latency.

On paper the setup looked solid. Dashboards showed traffic. Alarms existed in CDK. SNS topics were subscribed. Nobody ever got paged.

The symptom

During a reliability review we noticed that four alarms for the eligibility vendor had never left INSUFFICIENT_DATA:

  • success rate below 99%
  • error rate above 5%
  • any timeout
  • p95 latency over 10 seconds

Production was not quiet. The same dashboards showed hundreds of errors per day. Latency widgets had data. Timeouts appeared in graphs. The alarms sat still.

Investigation

We compared three things side by side:

  1. What the Lambda emitted
  2. What the dashboard widgets queried
  3. What the alarms selected

Dashboards used SEARCH expressions over rich dimension sets, roughly:

Endpoint + Method + Status (+ StatusCode / ErrorCode)

That matched the emitter. Widgets looked healthy.

Alarms cannot use SEARCH. They need a fixed metric identity: name, namespace, and an exact dimension map. Ours expected:

  • ApiCallCount with only { Status: success | error }
  • TimeoutCount with no dimensions
  • ApiLatencyMs with no dimensions

The emitter never published those series. It only published the full-dimension variants.

In CloudWatch, ApiCallCount with dimensions {Endpoint, Method, Status, ...} is a different time series from ApiCallCount with {Status}. Same metric name is not enough. Dimension sets must match exactly.

Root cause

Two design choices compounded:

  1. Dashboards and alarms need different metric shapes. Dashboards want high-cardinality series for drill-down. Alarms want low-cardinality, stable series they can address without SEARCH.
  2. treatMissingData: NOT_BREACHING. Missing data does not breach. If the alarm never sees its series, it never alarms. It also never looks broken in the way an ALARM state would. It just sits in insufficient data forever.

So we had a false sense of coverage: graphs worked, alarm definitions existed, and the monitoring path silently watched nothing.

Sibling clients in the same metrics module already emitted both shapes. This vendor path only emitted the dashboard shape.

The fix

Emit-side only. No alarm or dashboard changes.

On every eligibility API call we kept the existing full-dimension datums and added alarm-friendly aggregates:

  • ApiCallCount with { Status } only
  • ApiLatencyMs with no dimensions
  • TimeoutCount with no dimensions (on timeouts)

Dashboard SEARCH expressions require Endpoint and Method, so they do not match the new series. No double counting in widgets. Alarms finally received the series they were already configured to watch.

Unit tests asserted both the rich series and the aggregate series left the emitter.

Aftermath

After deploy, the alarms left insufficient data and began evaluating real traffic. We did not need to retune thresholds or rewrite alarm math. The definitions had been correct the whole time. Their inputs were not.

Lessons

  1. Verify the series identity, not just the metric name. In CloudWatch, dimensions define the series. Name collision is not a match.
  2. Treat dashboards and alarms as separate consumers. If a dashboard uses SEARCH over many dimensions, plan a parallel low-cardinality emission for alarms.
  3. NOT_BREACHING hides missing wiring. Prefer a deploy-time or test-time check that the alarm's exact dimension set is actually published. An alarm that has never left insufficient data is a smell, not a green check.
  4. Copy patterns across emitters carefully. Other clients in the codebase already had the dual-emit pattern. One path drifted, and monitoring failed quietly for that path alone.
  5. A quiet observability stack is not a healthy one. Absence of alerts is only meaningful if you have proof the alert path has data.

Closing

The bug was not a bad threshold or a flaky vendor. It was a contract mismatch between producers and consumers of metrics, papered over by a missing-data policy that fails closed into silence.

If your CloudWatch alarms have lived in INSUFFICIENT_DATA since creation while dashboards look fine, compare dimension sets before you trust the green.