We had two data centres.
But nobody wanted to lose one.
That bothered me.
The platform had been designed with redundancy. There was infrastructure in both locations, services running on both sides, and an expectation that if one data centre became unavailable, the other would keep the platform running.
On paper, that sounded reassuring.
But when my team started reviewing what had actually been built, that confidence disappeared quite quickly.
The infrastructure existed. The resilience didn’t.
I had initially been asked to review the infrastructure behind a new payments platform.
My team hadn’t designed it. We hadn’t selected the equipment, built the original environment or written the software.
We were coming into something that was already well underway.
One of my engineers reviewed the network and firewall configuration between the two data centres. Another worked through the virtualisation, storage and how workloads were expected to behave if one location disappeared.
As we went deeper, the problem became clearer.
We had two data centres, but the environment was not going to fail over in the way everyone expected.
That created an uncomfortable situation.
A significant amount of infrastructure already existed. There was equipment in two locations. The architecture had been designed around redundancy.
Yet if we actually lost one side, we couldn’t rely on the other side taking over.
Having two data centres was giving us confidence.
It wasn’t giving us resilience.
So we started changing things
We worked through the network configuration, virtualisation and storage.
We also found that some of the platform services had effectively been built as separate clusters in each data centre.
Again, it looked redundant.
There were services on both sides.
But two independent sets of services don’t necessarily give you one resilient platform.
We changed that architecture so services could operate across the two locations rather than treating each data centre as its own island.
There was a lot of technical work involved.
But the part that mattered most came afterwards.
We started breaking it.
Deliberately.
We wanted to know what would actually happen if part of the infrastructure disappeared.
Not what the architecture said should happen.
Not what somebody remembered being designed.
What actually happened.
Eventually, one data centre could disappear
We tested different failure scenarios repeatedly.
Initially, they didn’t work.
We changed things and tested again.
Eventually, we reached the point where we could fail from one data centre to the other.
That was a significant milestone.
The platform continued operating on the other side.
It would have been very easy to stop there.
We had deliberately failed a data centre and the other data centre had taken over.
We could say failover worked.
But there was another obvious question.
Could we move everything back?
So we tried.
It didn’t work properly.
That was an important moment for me.
We could have stopped after the successful failover and reported that resilience had been tested.
We would have been wrong.
We had proved one direction of one failure scenario.
That isn’t the same as proving resilience.
Every test gave us another answer we hadn’t expected
So we kept going.
Fail over.
Observe.
Fix.
Try again.
Fail back.
Observe.
Fix.
Try again.
And as we exercised the platform in ways it apparently hadn’t been exercised before, we discovered other behaviours.
During one of those tests, for example, messaging queues became locked.
That wasn’t a known problem somebody had handed over to us.
We found it because we were now doing something very simple but very revealing:
we were making the platform fail deliberately.
This became the pattern.
Every time we removed an assumption and replaced it with a test, we learned something.
Some things behaved exactly as expected.
Others didn’t.
And occasionally fixing one problem allowed us to reach the next problem that had previously been hidden behind it.
That could be frustrating.
But I’d much rather discover those problems while my team is deliberately creating the failure than discover them when a real data centre has disappeared and the business is waiting for us to recover.
The real problem wasn’t the second data centre
Looking back, one of the most important lessons wasn’t particularly technical.
The business already had many of the things normally associated with resilience.
Two locations.
Redundant equipment.
Multiple instances of services.
People who had designed the architecture.
Specialist technology.
If I had asked, “Do we have redundancy?” the answer would have been yes.
But that was the wrong question.
The question should have been:
What have we actually proved?
That distinction has stayed with me.
Technology organisations accumulate confidence surprisingly easily.
“We have backups.”
“We have a disaster recovery environment.”
“We run across multiple availability zones.”
“We have another data centre.”
“We can rebuild from infrastructure as code.”
All of those things may be true.
But there is a gap between having something designed to recover and knowing that it will recover.
You only close that gap with evidence.
You also have to recover from the recovery
The difficulty moving back to the original data centre taught me something else.
When people talk about disaster recovery, the conversation naturally focuses on surviving the initial failure.
Can we keep operating if this disappears?
Of course that matters.
But eventually the failed component, service or location comes back.
Then what?
You have to get from the recovery state back to normal operations.
For us, proving we could move from one data centre to another wasn’t enough.
We needed to understand what happened afterwards too.
That meant testing the return journey.
And when the return journey didn’t behave as expected, we kept working until we understood why.
I think that part is easily overlooked.
A successful failover can feel like the end of a disaster recovery test.
Sometimes it is only the halfway point.
The testing changed our relationship with the platform
Over time, we ran many data-centre failure tests.
The technical environment improved, but something else happened as well.
We understood it better.
We knew how different components behaved when something disappeared.
We knew which assumptions had already proved wrong.
We knew where failure could expose unexpected behaviour.
And, importantly, the team became increasingly comfortable testing it again.
That is a very different position from having a diagram showing two data centres and hoping the arrows between them work when you need them.
Before, we had infrastructure that was supposed to be resilient.
Afterwards, we had people who had watched it fail, recovered it, found problems, changed it and done it again.
That gave me much more confidence than the architecture ever could.
What that experience changed for me
Years later, when somebody tells me a platform is resilient, I don’t get much comfort from hearing that it operates across two locations.
I want to know what happened the last time one of them disappeared.
When was it tested?
What broke unexpectedly?
Could the team recover?
Could they return the platform to normal operation afterwards?
What did they learn?
What did they change?
And perhaps most importantly:
Would they be comfortable doing the test again?
Because if everyone is nervous about deliberately failing something that is supposed to be resilient, that nervousness is telling you something.
It doesn’t necessarily mean the architecture is wrong.
It means there is still uncertainty.
And uncertainty around resilience tends to remain invisible while everything is working.
Our two data centres looked reassuring before we started testing them.
The testing removed some of that reassurance.
For a while, it actually made the situation look worse.
But that was useful.
We were replacing confidence with evidence.
And that is the lesson I’ve carried with me ever since.
Resilience isn’t the infrastructure you’ve bought.
It isn’t the architecture somebody designed.
And it isn’t what the documentation says should happen.
Resilience is what you’ve actually proved happens when something fails.