There are two ways of getting production control wrong.
One is allowing people to change infrastructure whenever they need to and relying on logs to tell you afterwards what happened.
The other is locking production down so tightly that when something goes wrong, the people responsible for restoring the service cannot act.
Over time, I’ve found that good technology control sits somewhere between the two.
I learned this while thinking about something that initially seemed much simpler: the audit trail for infrastructure changes.
We could see what had changed
We already had good change history for software.
Changes were held in source control. There was a record of what had changed, and changes could be reviewed before reaching production.
Infrastructure wasn’t quite the same.
With on-premise infrastructure managed by an external provider, changes were generally recorded through the ticketing process.
Cloud infrastructure was different.
Engineers could make changes directly through the cloud portal. The platform itself kept logs, so we could go back afterwards and reconstruct who had done what.
From an audit perspective, there was evidence.
But I became uncomfortable with that being our definition of control.
Being able to explain what somebody changed yesterday is useful.
I’d rather know how that change was allowed to reach production in the first place.
The question became bigger than compliance
I wanted infrastructure changes to have the same discipline we expected from software changes.
There was also another concern.
We operated separate development, UAT and production environments. If those environments were changed manually over time, they could gradually become different.
Eventually you risk testing against one environment and operating something different in production.
So this wasn’t simply about producing evidence for an audit.
I wanted us to be able to answer some basic questions.
What should production look like?
How does a change get there?
Who reviews it?
And how do we know that what is actually running still matches what we think is running?
Those are technology leadership questions as much as compliance questions.
We changed the operating model
We moved towards defining infrastructure as code.
Development, UAT and production could now be based on the same infrastructure definitions.
The normal route for an infrastructure change became similar to the route for a software change.
Someone proposes the change.
Another engineer reviews it.
It is merged and deployed through the controlled process.
That gave us something more valuable than an audit trail.
It gave us a defined way of changing infrastructure.
But I didn’t want that process to become absolute.
Production still needs an emergency door
I restricted the ability to make direct production infrastructure changes to two people.
But I deliberately didn’t remove it completely.
If production is failing, restoring the service matters.
I don’t want somebody sitting in the middle of a serious incident unable to make a necessary change because the only permitted route requires them to first change the infrastructure code and work through the normal deployment process.
There are times when you need to act.
So direct intervention remained possible.
The difference was that it became an exception rather than the normal way of working.
And if we used that exception, I wanted it to be visible.
We introduced a daily check comparing the infrastructure we had defined with what was actually running.
If someone had made a manual change, we could see that production had drifted from our source of truth and deal with it.
Control isn’t the same as restriction
This changed how I think about technology controls.
It’s easy to interpret control as removing permissions, adding approvals and making it harder for people to change things.
Sometimes those controls are necessary.
But technology leadership also has to consider what happens at 2 a.m. when something isn’t working.
A control that looks perfect in an audit but prevents the team from responding to a production incident isn’t necessarily a good control.
Equally, giving trusted engineers unrestricted freedom because we can always check the logs afterwards isn’t enough either.
The balance I’ve come to prefer is different.
Make the controlled path the normal path.
Limit who can use the exception.
Keep the exception available when the situation genuinely requires it.
And make deviations visible so they cannot quietly become the new normal.
Compliance was the starting point, not the outcome
The original requirement was easy to understand.
We needed evidence of infrastructure changes.
But implementing that properly forced a broader question:
What does a controlled production environment actually look like?
The answer affected access, infrastructure, deployment, environment consistency, peer review and how we respond when something goes wrong.
That’s something I’ve seen repeatedly in regulated technology environments.
A requirement can arrive looking like a compliance problem.
Eventually, somebody has to turn it into decisions about how the technology organisation actually operates.
Good technology control isn’t about preventing people from acting.
It’s about making the normal way of working controlled, the exceptions deliberate, and both visible.