Your Failover Path Has a Control Plane Too

High availability covers the failures you designed for. Resilience covers the ones you didn’t.

A team I worked with upgraded their public ingress load balancers to TLS 1.3 for compliance. Handshakes completed, services stayed healthy, metrics stayed flat. Nothing suggested a problem.

Then traffic stopped arriving.

What happened

As of this writing, Route 53 HTTPS health checks require the target endpoint to support TLS 1.2. With 1.2 disabled, the health checker couldn’t complete the handshake and marked the endpoint unhealthy. The CDN took that at face value and stopped routing traffic to the region.

Inside the region, everything looked fine. Services were up, load balancers were healthy, dashboards showed nothing unusual. Meanwhile users were rerouted to a region thousands of kilometres away, latency spiked, and the failover region started scaling under load it had never been provisioned for. It took roughly forty minutes to isolate, because the failure wasn’t in the data plane. It was in the control plane, the part that decides where traffic goes.

Availability is not resilience

The system was highly available. It was not resilient.

High availability is about surviving failures you expected: an instance crash, a zone outage, a dependency timeout. Resilience is about recovering from conditions the system was never designed to handle. Availability is easy to measure through uptime, failover time and replication lag. Resilience resists measurement, because many failure modes only appear under real pressure, and testing them is often too expensive and risky.

The control plane nobody draws

Many multi-region architectures are multi-region only in the data plane. Their recovery still depends on a few shared control-plane pieces such as DNS health checks, identity and routing. The global AWS STS endpoint is one example: per AWS’s documentation it is hosted in a single region and doesn’t fail over automatically.

Monitoring that gates traffic is part of the data path, even though it lives in the control plane. So the useful design question isn’t “what if a region fails?” but “what if the thing that detects regional failure is itself degraded?”

For the few customer-facing paths where that dependency is unacceptable, AWS Application Recovery Controller (explicit routing controls plus readiness checks) is an alternative to DNS-based failover. It is operationally heavier, so I’d keep it for payments, authentication and similar paths, and leave DNS failover for everything else.

Recovery paths rot

Most organisations know who owns uptime: on-call, SLOs, escalation channels. Few know who owns recovery. Runbooks go stale, permissions drift, and the first thirty minutes of an incident turn into dependency archaeology.

This incident had its own version of that. Afterwards, two alerting gaps were found: the CDN had moved traffic away from an entire region without an alert, and the load balancers had dropped to zero inbound traffic just as silently. The CDN had also been failing health checks for a week on the first upgraded batch, while traffic kept flowing to the remaining TLS 1.2 targets. Nobody knew.

What I took away from this

1. Map the control-plane dependencies of your failover path. DNS, identity, routing, certificates, configuration. If failover depends on it, it belongs on the diagram.
2. Alert on absence, not just errors. A green application dashboard means little if the failure happens before requests reach the application.
3. Give recovery an owner separate from on-call, and drill it. On-call is about response; ownership is about keeping procedures, tooling and tests current. Start small, with one service and one AZ shift. Every drill finds something broken.

In sufficiently complex systems, resilience is probabilistic, not provable. The realistic goal is more confidence: fewer unknowns, rehearsed coordination, and a shorter gap between failure and detection.

This article was originally published on InfoQ on 01/10/2026: High Availability is Not Resilience: Why Cloud Systems Fail When it Matters Most.