Why five nines, and what it actually cost us to promise it
For a little over five years I was the DevOps and Infrastructure Lead for Colombia’s first fully digital bank, built on AWS. Signing up to a 99.999% availability SLO sounds like a slide-deck number until you do the arithmetic. It leaves roughly five minutes of error budget for the entire year. That constraint drove almost every architectural decision we made, and it forced us to be honest about the difference between “highly available” and “actually measured.” This post is about how we designed to that target, how we operated against it, and where the money and the pain went.
Instrumenting the SLO before trusting it
The first thing I pushed back on internally was the temptation to declare five nines based on infrastructure redundancy alone. A number you can’t measure is a number you can’t defend to a regulator, and we were operating under SOC 2, PCI DSS, and ISO 27001 pressure while securing the operational banking license. So we started with SLIs, not architecture diagrams.
We defined availability at the level that mattered to customers, meaning successful, timely responses on the request paths that move money or show a balance, rather than raw host uptime. Those SLIs fed an explicit error-budget policy: as long as we had budget left, teams shipped features; when we burned through it, reliability work took priority over roadmap. That policy was the single most useful reliability tool we had, because it turned “should we slow down?” from an argument into a rule. It also made blameless postmortems land better. A burned budget was a shared signal, not a search for someone to blame.
On the delivery side we ran daily production releases and held change-failure rate at or below 15% with sub-hour recovery. Automated promotion through environments (Jenkins, Terraform, GitHub Actions), plus SAST via SonarQube and cloud-posture remediation through Dome9, kept the release rate from becoming the thing that ate the budget.
Multi-region HA/DR, and the drills that made it real
Five nines is not survivable in a single region, so the platform was multi-region by design. The data tier is where most of the interesting decisions lived, because every storage engine has a different failover story:
- DynamoDB Global Tables gave us active-active replication for the workloads that could tolerate last-writer-wins semantics, so a regional loss didn’t mean a data-plane outage there.
- Amazon Aurora MySQL carried the relational, transactional workloads, where we accepted a more managed replica/failover model rather than pretend multi-writer was free.
- S3 cross-region replication covered object data and artifacts.
- AWS Elastic Disaster Recovery (DRS), which we adopted after starting on CloudEndure, handled the servers that didn’t fit a native-managed pattern. It did block-level replication into a standby region we could spin up on demand.
The architecture only counts if you rehearse it. We ran scheduled failover drills and validated recovery to a 5-minute RPO and a 4-hour RTO, so at most five minutes of data at risk and four hours to full service in the recovery region. The drills were where assumptions went to die: DNS TTLs, warm-vs-cold capacity, secrets availability in the second region, and the ordering of which tier you bring up first. An untested DR plan is a hope, and hope has a bad RTO.
FinOps: paying for reliability without overpaying
Redundancy across regions is expensive by construction. You are, by design, paying for capacity you hope never to fully use. The job wasn’t to make it cheap, it was to stop it being wasteful. We cut peak compute cost by roughly 30% using AWS Savings Plans for the always-on baseline and Spot Instances for the fault-tolerant, interruptible workloads. The discipline that made Spot safe was the same discipline that made DR safe: if a workload can survive an instance vanishing without warning, it can survive a much larger failure too. Reliability engineering and cost engineering turned out to be the same muscle.
Running it day to day
None of this holds together without an operating rhythm. We staffed a real on-call rotation with defined incident response, and every significant incident produced a blameless postmortem whose action items were tracked to closure rather than filed and forgotten. Over time I led and mentored the DevOps, SRE, and Cloud teams up to about 25 people, and the culture mattered as much as the tooling. The error-budget policy only works if leadership actually honors it when the budget runs dry and a launch is on the line.
Takeaways
- Measure the SLO from the customer’s request path, or don’t claim it. Infrastructure redundancy is an input, not evidence.
- The error-budget policy is a decision-making tool, not a dashboard. Its value is settling the “ship or stabilize” argument in advance.
- Pick a failover model per data engine. Global Tables, Aurora, S3 CRR, and DRS each buy a different guarantee, so don’t force one pattern on all of them.
- Untested DR is fiction. The drills, not the diagrams, are what validated our 5-minute RPO and 4-hour RTO.
- Spot and DR reward the same design. Build workloads that tolerate sudden loss and you get cost savings and resilience from one investment.