When a seemingly harmless kernel optimization collides with a modern transport protocol, the results can be puzzling. This is the story of a bug in Cloudflare's QUIC implementation, quiche, where the CUBIC congestion controller's window got permanently stuck at its minimum after a loss event. The root cause traced back to a Linux kernel change meant to align CUBIC with RFC 9438§4.2-12—a fix for TCP that, when ported to QUIC, uncovered unexpected behavior. Here are ten things you need to know about this bug, from its symptoms to the elegant one-line fix that restored order.
1. CUBIC: The Default Congestion Controller in Linux
CUBIC, standardized in RFC 9438, is the default congestion controller used by the Linux kernel. This means it governs how most TCP and QUIC connections on the public internet probe for available bandwidth, react to loss, and recover after congestion. Its widespread adoption makes any bug in CUBIC a critical issue for global network performance.
2. QUIC and Quiche: Cloudflare's Open-Source Implementation
Cloudflare's QUIC implementation, quiche, uses CUBIC as its default congestion controller. This places the CUBIC code directly in the critical path for handling a significant share of Cloudflare's traffic. Any subtle interaction between CUBIC and QUIC's unique semantics can have outsized effects on real-world connections.
3. The Symptom: A Test That Failed 61% of the Time
The investigation began with inexplicable failures in Cloudflare's ingress proxy integration test pipeline. When testing CUBIC under heavy early connection loss, the test failed in 61% of runs. Recovery after a congestion collapse is an uncommon regime, but it's exactly the scenario a congestion controller must handle. Most tests focus on steady-state growth; bugs at minimum cwnd are often invisible in throughput data.
4. What Is a Congestion Window (cwnd)?
The congestion window is the sender-side cap on how many bytes can be in flight at any moment. A larger cwnd allows more data per round trip; a smaller one throttles the sender. Loss-based algorithms like CUBIC grow cwnd when there is no loss and shrink it when loss occurs, aiming to infer available bandwidth without explicit feedback.
5. The Linux Kernel Change That Sparked It All
To bring CUBIC in line with RFC 9438§4.2-12, the Linux kernel introduced a fix for the app-limited exclusion. This change modified how CUBIC handles periods when the application does not have enough data to send. While correct for TCP, this change had unintended consequences when ported to QUIC.
6. How the Bug Manifested in QUIC
When the Linux kernel fix was ported to quiche, the CUBIC congestion window became permanently pinned at its minimum after a congestion collapse event. Instead of recovering by probing for bandwidth, the window stayed locked, never allowing the connection to regain its prior sending rate. This led to sustained underutilization of the network path.
7. The Root Cause: A Subtle Timing Interaction
The bug arose from a subtle interaction between the app-limited exclusion logic and QUIC's different acknowledgment rules. In TCP, the timing of ACKs and application writes prevented the problematic state. In QUIC, with its multiplexed streams and different delivery semantics, the same logic could cause CUBIC to misinterpret a paused application as a sign of persistent congestion.
8. The Elegant Near One-Line Fix
The fix was surprisingly simple: a single change that broke the cycle causing the permanent minimum window. By adjusting when CUBIC considers the connection to be app-limited, the update allowed the controller to recover normally after congestion. This one-line adjustment restored correct behavior without reintroducing the original TCP bug.
9. Lessons for Congestion Control Implementation
This case highlights the dangers of porting TCP-specific optimizations to other protocols without careful analysis. Even with the same congestion controller, subtle transport differences can reveal hidden assumptions. It also underscores the need for testing corner cases like recovery after severe congestion, not just steady-state throughput.
10. Why This Matters for the Internet
Given CUBIC's ubiquity, any bug in its QUIC implementation could impact millions of connections. The successful fix ensures that Cloudflare's infrastructure—and the many services relying on QUIC—remain robust. More broadly, it demonstrates that even well-tested algorithms require re-validation when adapted to new environments.
This story ends happily: a one-line fix resolved a confounding bug. But the journey from a failing test to a root cause analysis reveals how deep the interplay between kernel optimizations and protocol design can go. As QUIC adoption grows, such lessons will be essential for maintaining a resilient, high-performance internet.