Introduction
Over the past two and a half quarters, Cloudflare’s engineering teams have been engaged in an intensive internal initiative code-named "Code Orange: Fail Small". The goal was straightforward yet ambitious: make the network infrastructure more resilient, secure, and reliable for every customer. Earlier this month, that work was officially completed—but as the team notes, resilience is never a finish line; it’s an ongoing priority woven into the entire development lifecycle. The immediate driver was to prevent the kind of global outages that occurred on November 18 and December 5, 2025. This article explores the key changes, what was shipped, and what it all means for you.
Key Areas of Focus
The project targeted several critical areas: safer configuration changes, reducing the impact of any single failure, overhauling break glass procedures, and improving incident management. Additionally, Cloudflare introduced measures to prevent configuration drift and regressions over time, and strengthened customer communication during incidents. Let’s dive into the most impactful changes.
Safer Configuration Changes
Historically, many internal configuration changes could propagate across the entire network almost instantly. That approach carried significant risk: a misconfigured setting could affect customer traffic before anyone noticed. Now, Cloudflare has shifted to a health-mediated deployment methodology for all configuration changes—the same rigorous process used for software releases. This means changes are rolled out progressively, with real-time health monitoring that can automatically detect problems and revert changes before they impact your traffic.
To achieve this, Cloudflare built a new internal component called Snapstone. Snapstone bundles configuration changes into packages, then releases them gradually with health checks at each step. Before Snapstone, implementing health-mediated deployment for configurations was possible but required significant, custom effort per team—leading to inconsistent application. Snapstone closes that gap by providing a unified system for progressive rollout, automated rollback, and live health monitoring. What makes it especially powerful is its flexibility: it can handle any unit of configuration, whether it’s a data file (like the one that caused the November 18 outage) or a control flag (like the one involved in the December 5 outage).
Reducing the Impact of Failure
Another major area was minimizing the blast radius when something does go wrong. By isolating failures to smaller segments of the network, Cloudflare ensures that a problem in one region or service doesn’t cascade globally. This fail small philosophy is embedded in new architectural guardrails and automated isolation mechanisms.
Improved Break Glass and Incident Management
The term "break glass" refers to emergency procedures that allow engineers to bypass normal safeguards during critical incidents. Cloudflare revised these procedures to include clearer escalation paths, pre-approved emergency actions, and post-incident reviews that feed back into the system. Incident management was also enhanced with better coordination tools and faster communication channels.
Preventing Drift and Regressions
To ensure the improvements last, Cloudflare introduced new automated tests and monitoring checks that run continuously. These catch configuration drift—small unauthorized changes that can accumulate over time—and flag regressions before they become problems. Regular audits ensure that all teams adhere to the new practices.
Customer Communication During Outages
Transparency is critical during any service disruption. Cloudflare overhauled its customer communication protocols to provide faster, more detailed updates. The new approach includes structured status pages, real-time alerts via dashboards, and post-incident reports that clearly explain root causes and corrective actions. This ensures that even when issues do occur, you stay informed every step of the way.
What This Means for You
For most Cloudflare customers, the most visible change is increased stability. Configuration changes that once took effect everywhere in seconds are now rolled out gradually, with safety nets in place. The risk of a global outage from a single misconfiguration has been dramatically reduced. Moreover, the improved communication means you’ll receive timely updates if anything does go wrong.
Behind the scenes, Cloudflare’s network is now more resilient by design. The Snapstone system and health-mediated deployments are already being used by the product teams directly affected by the November and December incidents, and the lessons are being applied across all teams. The company has also committed to ongoing resilience investments, so this isn’t a one-time fix but a new baseline.
Looking Ahead
While "Code Orange: Fail Small" is complete, Cloudflare emphasizes that improving resiliency is an endless journey. The tools and processes introduced—such as Snapstone and health-mediated deployment—will continue to evolve. New failure modes will be discovered, and the network will adapt. But the foundation laid over the past two quarters gives customers confidence that the network is stronger than ever.
To learn more about specific technical details, refer to the official engineering blog or the post-incident reports for the November and December outages. For a deeper dive into Snapstone, see the Safer Configuration Changes section above.