I once became responsible for the infrastructure behind a new payments platform that another team had already been building.
My team hadn’t designed it.
We hadn’t selected the equipment.
We hadn’t written the software.
Initially, we were asked to look at the infrastructure.
One of my engineers started reviewing the network configuration between two data centres. A third-party provider had supplied and configured much of the equipment and was proposing that we spend more money to solve one of the problems.
My engineer reviewed the proposal and came back with a different conclusion.
The additional equipment wasn’t going to solve the underlying problem.
That was the beginning of several months of asking questions.
Every answer seemed to create another question
We started working through the infrastructure.
Network configurations needed changing. Parts of the switching configuration needed correcting. The way the virtual machines and storage had been configured meant failover between the two data centres wasn’t going to work.
Some services were effectively operating independently in each location rather than as one resilient platform.
So we started changing things.
More importantly, we started testing them.
As we made changes, we got the platform failing over from one data centre to the other. Then we discovered we couldn’t reliably bring it back again.
We fixed that and tested again.
Other tests exposed different problems.
This wasn’t a case of somebody giving us a list of known issues.
We were discovering how the platform actually behaved by deliberately testing scenarios that hadn’t been properly exercised before.
But the technical problems weren’t what made the situation most difficult.
It was understanding why things were the way they were.
Who made that decision?
I’d ask why something had been designed a particular way.
Someone would point me to somebody else.
Some of the people involved had already left.
Development had separate leadership, so although our infrastructure changes sometimes required changes to the software, I wasn’t responsible for the development team.
There were boundaries I couldn’t simply ignore.
And not all the history of the platform had been handed to us when we became involved.
One example was the production environment.
The infrastructure had originally been intended for staging.
Building another equivalent environment was expensive. The business didn’t want to spend more money, so at some point a decision had been made to turn what had been built into production.
By the time my team became involved, I understood what we’d been given to be the production infrastructure.
It was only later that more of the history became clear.
And that raised another question.
What had actually been tested?
We believed the application had already been tested properly against the external payment network’s staging service.
That was our understanding.
We had no reason at the time to assume otherwise.
Later, when further staging testing was required, we started discovering that the situation wasn’t as clear as we’d believed.
Then we connected to production
Months after we’d first become involved, the platform reached its initial controlled production phase.
I was online with the developers in the early hours of the morning when the external payment network told us the connection was live.
They sent us a payment.
We didn’t receive it.
From their side, they could see the connection being established and then dropping. They would restart it, it would connect and drop again.
We were stuck in a continuous loop.
The remaining developer from the original team initially believed the problem was with the external provider.
I found that difficult to accept.
They were already operating successfully with other participants. We were the new participant that wasn’t working.
My developers and I thought we needed to look at our own software first.
So two of my developers and I started going through an application we hadn’t written.
The code was difficult to follow. Methods called other methods which called more methods, and understanding what was happening wasn’t straightforward.
I was debugging locally while my developers worked through different parts of the application and we shared what we were finding.
Meanwhile, the external network was waiting for us to operate and our CIO was having to explain why we couldn’t.
As we investigated, something more concerning became clear.
The end-to-end staging testing we believed had happened hadn’t actually happened in the way we’d understood it had.
We had gone into production with confidence based partly on testing that hadn’t proved what we thought it had proved.
Finding the problem wasn’t the end
Eventually, one of my developers found a problem that was causing the application to fail.
We made a change, deployed it across the services and waited.
We started seeing more activity.
But the logs weren’t telling us enough.
So we added more logging.
Deployed again.
Watched again.
Found where execution was getting further.
Repeated.
This continued over the following days.
The next stage of the production launch was already scheduled for the following Friday.
We got there.
But the platform wasn’t something we simply switched on and forgot about.
It was operating, but we monitored it closely and dealt with issues as they appeared.
We weren’t only debugging the software
Looking back, what stays with me isn’t one particular technical problem.
It’s how much time we spent reconstructing decisions.
Why was this configured this way?
What had actually been tested?
Which environment had been used?
Who had made that decision?
What assumptions had been made when the application was written?
Sometimes we found the answer.
Sometimes the person who knew had already left.
Sometimes different people had different parts of the answer.
And sometimes we discovered the answer only when something failed.
Because when an incident happens, you’re not only diagnosing what the system is doing.
You’re trying to reconstruct why the system was built that way in the first place.
Gradually, we established control
After the launch, we continued changing the way the platform operated.
We built a proper staging environment so changes could be deployed and tested there before reaching production.
Production deployments became controlled. The application ran across four services split between the two data centres, allowing us to remove one service at a time, deploy the change and return it to service.
We restricted access to the production servers. Previously, developers could access them and make changes directly. I wanted production changes controlled by the team responsible for operating the environment.
We improved logging and built dashboards that made it easier to understand what the platform was doing.
Even then, following a payment through all the different parts of the platform was difficult.
There wasn’t a simple way to start with an incoming transaction and trace its journey from beginning to end.
We worked to improve that visibility because when something goes wrong, knowing that something failed isn’t enough.
You need to know where.
We were also learning operational requirements that hadn’t initially been clear to us.
At one point I was given contractual and operational documentation and started reading through it myself. That was how I discovered some of the requirements around maintaining and patching the platform.
We then established a controlled process with the external network: traffic could be moved away from one data centre while we patched that side, returned it to service, and then repeated the process on the other.
Piece by piece, we weren’t just fixing technology.
We were learning how to operate it.
What stayed with me
I don’t look back at this experience and think there was one obvious decision I should have made differently on the first day.
I had responsibility for part of the platform, not all of it. Development had separate leadership. People who had made earlier decisions were leaving or had already left. Information was fragmented across teams.
What the experience changed was the questions I ask when taking responsibility for technology.
I don’t only want to know what exists.
I want to understand why it exists.
What has actually been tested?
What do we only assume has been tested?
Who understands the important architectural decisions?
What happens when something fails?
Who is allowed to change production?
Can we trace what the system is doing when something goes wrong?
And if the people who originally built it disappeared tomorrow, could the people left behind still operate it?
Architecture diagrams can look good.
Infrastructure can be duplicated.
Tests can be green.
And a system can be described as production-ready.
None of those things, by themselves, mean you have operational control.
Sometimes the first step in transforming technology isn’t changing it.
It’s finding out what you actually have — and whether anybody still knows why.