At 11:20 UTC on November 18, 2025, Cloudflare’s edge began returning 5xx errors at scale. Within minutes, X was down, ChatGPT was throwing 500s, and roughly one-fifth of the public web was affected. A scheduled database permissions change fifteen minutes earlier had quietly altered how a Bot Management feature file was generated; the file doubled in size, hit a runtime limit of 200 features, and the Rust proxy panicked on a Result::unwrap(). Cloudflare didn’t identify the root cause until 13:37 UTC. Core traffic resumed around 14:30. Their postmortem is unusually detailed.
The interesting story about that morning isn’t the outage itself - Cloudflare has been thorough about owning it. It’s the gap between their customers’ responses. Some teams were posting status-page updates within three minutes. Others learned from users on their support channels two hours later. Same outage, same impact window, completely different operational experience.
What separated them was almost entirely which Cloudflare alerts they had configured - and which alerts the Cloudflare Notifications page makes hard to find, or doesn’t offer natively at all. Six specific gaps determined who knew first.
Why these gaps exist in the first place
Cloudflare’s Notifications surface is a flat opt-in checklist of around 80 alert_type values. There is no “recommended for your account” view. The default state after signing up is empty and remains empty until someone walks the list manually and ticks boxes.
Coverage across the product surface is uneven. SSL, Tunnel Health, DDoS L7, Load Balancer pool health, and Logpush job state all have native notification policies. Workers errors, R2 5xx rates, cache hit ratio drops, origin latency regressions - none do. The available_alerts API is the canonical proof; you can list it yourself in one curl.
And there is no native “audit my coverage” view. A team that configured three policies in 2022 has no way to see what they’re missing in 2026 without manually walking the full list. The six gaps below recur across deployments because the platform doesn’t surface them.
Detection gap 1 - Edge 5xx and origin 5xx are not the same signal
Cloudflare ships two distinct 5xx-rate notification types, and both should be configured, routed to different destinations:
http_alert_edge_error: Cloudflare-side problem. Codes 520–526 dominate (origin unreachable, SSL handshake failed, connection timed out). Should page the team that knows how to escalate to Cloudflare.http_alert_origin_error: Customer backend problem. Codes 500–504 originating from the customer’s server. Should page the backend on-call.
Most teams configure one or the other, often whichever was easier to find. During the November outage, teams with only http_alert_origin_error saw nothing - their backend was healthy. Teams with only http_alert_edge_error during an unrelated backend deploy gone wrong wake up the wrong on-call.
The fix is mechanical: two cloudflare_notification_policy Terraform blocks, two distinct destinations, ten lines of code. The customers who had this distinction in place on November 18 routed their pages directly to “investigate Cloudflare,” not “roll back yesterday’s deploy” - and saved themselves the first thirty minutes of misdirected debugging.
Detection gap 2 - Workers and Workers KV have no native notification
The most pointed gap of the November outage. Workers KV cascaded - customers using KV for feature flags, session storage, or rate-limit counters silently broke as Bot Management’s dependencies failed.
There is no workers_error_rate or workers_kv_error_rate in available_alerts. Run the API yourself and check. Workers metrics live in the GraphQL Analytics API, not in Notifications. To alert on them, customers need to take one of three paths:
GraphQL polling. A small Worker on a Cron Trigger queries the
workersInvocationsAdaptiveorkvOperationsAdaptiveGroupsdataset and pushes a derived rate to an alerting system.lablabs/cloudflare-exporter + Prometheus. Exposes
cloudflare_worker_errors_countandcloudflare_worker_requests_countas Prometheus metrics; the alert is a ratio over a time window.Logpush → SIEM with downstream rule. For teams already on Datadog or Splunk.
Each path adds 5–7 minutes of latency end-to-end. That’s worse than a native Notification (which fires in seconds). It’s also infinitely better than the multi-hour delay between cascade and discovery that most teams experienced on November 18. Teams with the alert in place switched feature flags off, served graceful fallbacks, and posted to their status page proactively. Teams without it found out when users started complaining about specific broken features.
Detection gap 3 - R2 is monitored less than your laptop’s disk
R2 in production is typically the destination for log archive, backups, asset hosting, build artifacts. None of those scream when they fail silently. On March 21, 2025, R2 write operations failed at 100% for 1 hour and 7 minutes after a credential rotation mishap. Most affected customers learned about it through Cloudflare’s status page, not their own monitoring.
There is no native Notifications policy for R2 error rates. The data lives in GraphQL (r2OperationsAdaptiveGroups, filterable by actionStatus) or in the lablabs exporter’s cloudflare_r2_operation_count. A useful alert combines two signals: a 5xx-rate threshold over five minutes, and a flatline detector for storage byte growth if R2 is the sink for a Logpush job or backup pipeline.
Backups that never restore are not backups. R2 writes that fail silently are the same shape of problem.
Detection gap 4 - Origin latency regressions with no baseline
On August 21, 2025, a traffic surge from a single Cloudflare customer overloaded the network links between Cloudflare and AWS us-east-1. Customers with origins in that region saw their p95 latency creep up; almost none had alerts for it because their alerts were thresholded on uptime, not latency.
Static thresholds miss latency drift entirely. Endpoints continue to return 200, they just take longer to do it. The fix is to alert on a percentile (p95 over five minutes) compared against a trailing baseline (50% above the seven-day median, say). The GraphQL field is originResponseDurationMs; the lablabs exporter exposes per-status-code, per-country latency histograms.
Why this is rarely configured: percentile-of-percentile arithmetic isn’t in the default “monitor your origin” tutorial. Most stop at uptime. Teams that go beyond it catch slowdowns the rest of the market discovers through support tickets.
External observability - the blind spot when Cloudflare can’t tell you what just happened
This is the conceptual heart of the post.
On February 20, 2026, an internal cleanup task - introduced as part of Cloudflare’s “Code Orange: Fail Small” resilience initiative - withdrew approximately 1,100 customer BGP prefixes from the internet. Twenty-five percent of all BYOIP prefixes vanished from BGP for over six hours. No Cloudflare-side dashboard could have warned affected customers, because Cloudflare’s own systems didn’t know yet. The customer’s IPs were gone from the routing table; the customer’s Cloudflare console showed everything green.
The principle this surfaces is simple but load-bearing: observability about a system must originate outside that system. Three concrete requirements follow:
External synthetic checks from outside Cloudflare’s network. Pingdom, StatusCake, UptimeRobot, or self-hosted probes from a non-Cloudflare-fronted IP. Every one to two minutes against the customer’s flagship endpoints.
BGP route monitoring for BYOIP customers. BGPalerter, RIPE RIS, or a paid offering. The February 20 outage is the textbook case.
A status-page subscription for
cloudflarestatus.comitself, wired into the same incident channel as everything else.
One further point. If a team’s operational runbook says “first, check the Cloudflare dashboard,” that runbook has a circular dependency. The November 18 outage took the Dashboard down. The teams that had non-Cloudflare-fronted health checks knew immediately. The teams that didn’t were locked out of their primary diagnostic surface during the worst window of the outage.
External observability is the only category of alert that fires reliably when Cloudflare itself is the broken thing.
Operational hygiene - notification policies that exist but never deliver
The “everything is configured” trap. A team configures a Notifications policy in 2022, points it at a Slack webhook, and never tests it again. Three years later the webhook is dead: the channel was renamed, the bot integration was removed, the PagerDuty integration key was rotated and never updated in Cloudflare. The policy still exists. The dashboard shows it as healthy. Nothing actually pages.
This gap doesn’t tie to a specific outage because it doesn’t surface during a specific outage - it surfaces during every outage, as a multiplier on response time. The fix is mechanical and almost universally skipped. Cloudflare’s notification destination test endpoint (POST /accounts/{account_id}/alerting/v3/destinations/webhooks/{id}/test) supports a synthetic delivery against every configured target. A quarterly fire-drill, ten minutes per quarter, catches every silent failure before it matters.
The most expensive alert is the one that’s configured but doesn’t deliver. It creates false confidence, and false confidence is worse than no confidence.
What Cloudflare’s UX optimizes for
Step back from the six gaps and a pattern emerges. Cloudflare’s Notifications surface is discovery-oriented: it tells you what’s available. It is not coverage-oriented: it doesn’t tell you what you’re missing given your actual configuration. AWS, GCP, and Azure have analogous gaps but ship at least partial “recommended monitors” surfaces - CloudWatch alarm templates, GCP recommended dashboards, Azure Monitor recommended alerts. Cloudflare’s flat opt-in list is more austere.
The category emerging in response is alert coverage auditing - third-party tools that compare a customer’s configured policies against a recommended baseline and surface what’s missing. It is still a young category, the way external observability of one’s own services was young a decade ago.
A practical audit, this week
Things worth checking in the Cloudflare account this week:
Open Notifications. Count configured policies. Fewer than six is statistically under-alerted.
Test every destination - fire a synthetic alert through each one.
Confirm both
http_alert_edge_errorandhttp_alert_origin_errorexist, on separate destinations.Confirm at least one external (non-Cloudflare-fronted) synthetic check on each flagship endpoint.
If using Workers, confirm a Workers error-rate alert sourced outside Cloudflare Notifications.
If using R2, confirm an R2 5xx-rate alert exists.
If on BYOIP, confirm BGP route monitoring is in place.
Closing
The recent Cloudflare outages weren’t caused by missing customer alerts. Cloudflare owns those, and their postmortems make that clear. But the response delta between teams - the gap between knowing within three minutes and learning from users two hours later - was caused almost entirely by missing customer alerts.
That gap is closing as the expectation shifts. Auditing a SaaS dependency’s alerting against a recommended baseline is becoming a normal expectation, the way auditing one’s own services was a decade ago. The platforms that build the muscle for it first will be the ones whose teams stop showing up on the slow side of the next outage’s response-delta chart.


