Rolling Out with Confidence: Part 1 — the journey towards an AI-assisted path to production at Deel

The first release I ever ran at Deel, I ran from a coworking space in Seville, during a September day, wedged into a meeting booth that was somehow both cramped and boiling, on a three hour call with Nick Rudenko.

Three hours and not because the code was hard. It was the backmerges. Across five repositories, by hand, dev, the release branch and master all had to be synced before the release branch could go into master. Only then, could a fresh release branch be cut from dev. The conflicts had to be resolved one at a time, while Nick talked me through what to run and more importantly, what not to touch. My hand was sweating on the trackpad, certain I was one wrong click away from overwriting the release branch with the wrong content, merging it into master and taking production down with it. Nick was watching like a hawk so I wouldn't do something stupid, braced to clean up whatever mess I was about to make if an automation fired the wrong way.

When it finally went out, we both exhaled, sank back and relaxed a little. For about a day. Then came the next one and the one after that. Rinse and repeat. Successfully, every time and never once less stressful. Not until we started changing the process itself.

That was Deel's release process when I inherited it: two people on a call, praying a script does what they think it does. This is the beginning of the story of how we turned it into something a machine mostly runs, a human blesses with one click and nobody has to sweat through.

01 · What "release" meant when I got here

We had a Release Candidate, an RC. The idea is sound: you don't push dev straight to production. You cut a branch, you freeze it, you stabilise it in a production-like environment for a couple of days under monitoring and test runs and only then you push the button.

The problem was the size of the thing. Our RC was a snapshot of dev from up to two weeks earlier, plus every cherry-pick that had landed since. During month-end freeze windows it could reach a thousand tickets. A thousand. That is not a release candidate, that is a small kingdom and I was the reluctant queen of it. The regression suite took four hours. The moment an engineer merged to dev, they lost all control over when their work would ship and a single team's bug could hold the whole train for everyone else.

That is not a release candidate, that is a small kingdom — and I was the reluctant queen of it.

We started at a couple of RCs a week, except during the month-end freeze, when nothing moved at all for a week. Then, because each one was so heavy to wrangle, we slowed the cadence right down. Slowing down didn't make it lighter. It made each release bigger, which made it scarier, which made us want to slow down again. If you've ever tried to lose weight by eating less often, you know how that went.

And here's the part the official narrative never highlighted: the slow lane wasn't the real lane. The real way things shipped was engineers cherry-picking straight to production, a fast lane that was the norm, used constantly, with essentially no quality gates on it at all. I say this with love, because I was one of the people using it. When the front door takes a week, everybody learns where the side door is.

So we had the worst of both worlds: a release process too heavy to trust and an escape hatch too easy to abuse. Meanwhile, we were expected to ship over forty fixes and features every day, all mixed together. You cannot pair that pace with a once-a-week, thousand-ticket release and an ungated side door and call it healthy.

I became, for a while, a professional nagger. Chasing sign-offs across sixty Jira projects before we dared promote anything. It isn't on my CV. It probably should be.

02 · The idea that unlocked everything: release is not rollout

Here is the reframe the rest of this rests on.

Deploying code and exposing it to a customer are two different events and they deserve two different systems. Deploy belongs to engineering: code reaches production. Rollout belongs to product: a customer finally sees the feature. For years we'd tangled the two together, which is why every deploy felt so heavy. It carried the full weight of customer exposure.

Once you separate them, deploy can be small, constant and boring, because nothing is visible until a second, deliberate switch is flipped. That switch, the feature flags and phased cohorts, together with the question of who sees what and when, is a system of its own and it's the subject of the next post. For now, just hold the idea: code reaching production is not the same as a customer seeing it. Separating the two let us make deploy safe first, without customer exposure riding on every merge.

Which is the half of the story we had to address before anything else.

03 · The fast lane ... that wasn't

Our first real attempt to fix it was a trunk-style release: small, frequent releases carrying only cherry-picks, this time with gates on them. The idea was to take the side door everyone was already using and put a proper lock and a camera on it. We called it the release trunk. And honestly? It worked, for a while. Three clean daily releases in a row, no incidents. I was ready to declare victory.

Then the environment underneath it wobbled, several times and everything I believed about just moving faster fell over. Tests wouldn't run reliably, so releases got delayed and here's the lesson I paid full price for. When you keep delaying releases because the environment is flaky, engineers don't become more careful. They go back to cherry-picking straight to production, right past every check you built to protect them. Instability doesn't buy you caution. It buys you people routing around you.

Instability doesn't buy you caution. It buys you people routing around you.

I had genuinely believed the trunk could be the answer. It couldn't and I eventually said so out loud. You don't get to keep a nice-to-have on life support just because it was your idea. So we cut the detour short and pivoted.

04 · Fast-healing staging: RC's rigour at trunk's speed

The version that stuck is almost embarrassingly simple to describe.

First, a distinction worth making up front, because two different things share the name staging. There's the staging branch, which is our release candidate: after each release it's recreated from dev, so it's always a fresh, honest snapshot of everything that's ready to ship. And there's the staging environment, a long-living, production-shaped place that always runs whatever is on the staging branch. The branch is the candidate; the environment is where we prove it.

With that split, the flow is simple. After each release, the staging branch is reset from dev. The staging environment picks it up automatically, no manual branch cutting, no conflict resolution rituals at 2pm, no sweating in a booth in Seville. The full automated suite lives on top of it: end-to-end tests, public API tests, performance tests, all on production-shaped data. Then, once a day, the staging branch promotes to master.

dev → staging → master. Fast forward, no detours.

The trick is that preparing a release stops being a task at all. There's nothing to stabilise on release day, because the staging environment was never allowed to rot in the first place: the branch feeding it is continuously refreshed from a dev we now keep green by default. We didn't throw RC out. We took everything it taught us, the monitoring, the quality gates, the discipline of validating before you promote and moved all of it onto an environment that maintains itself, instead of one I had to resuscitate every week.

05 · Guardrails over trust

A daily release only works if the gates are automatic and genuinely hard to bypass. "Please be careful" is not a control. Four principles carried us.

Push the gate left. We stopped trying to catch problems at the release branch, which was always too late and too fragile. Instead we block at the merge into dev: if the tests for the code you actually touched don't pass, your PR doesn't merge. Keep dev green and the release stays green by construction.

Zero bypass, not zero-trust theatre. Remember the ungated side door? It finally got its guardrails. Cherry-picks now run through rules that simply cannot be waived: the commit has to already exist in dev and a tracked request has to exist in our system, or the merge is blocked. Cherry-picks to staging are the default now and cherry-picks straight to master are the rare, justified exception that a director has to sign for. The urgent door still exists. It's just narrow, watched and it locks behind you.

Automated end-to-end and API coverage is the promotion signal. The decision that a release candidate is good enough to promote isn't anyone's judgement call, it's the automated end-to-end and public API suite going green in the staging environment. That suite is the real quality gate and it's substantial enough that it earns its own breakdown just below. If it's red, the candidate does not move.

Approve by exception, not by roll call. Chasing explicit approvals was the single biggest bottleneck (remember the professional nagger), so we inverted it. Every release is approved by default and any registered approver can place a veto if they have a concern. No veto and we ship, automatically, with no human confirmation step in the way. A veto blocks the release until it's cleared and the moment the last veto is withdrawn, we proceed on our own. It's the same safety net, anyone can stop the train with one click, but without the dead time of waiting for approvals that were always going to come anyway.

Those four guardrails decide what is allowed to move. The third of them, automated coverage, is where most of the real work lives, so let me open it up. It's a ladder of gates that a release candidate climbs in the staging environment before it earns promotion.

Gate 0 — is the environment even usable? Services up, no 5xx spikes, migrations applied. If the staging environment itself is unhealthy we say so in Slack and don't waste a run pretending otherwise.

Gate 1 — the smoke run. We rank our flows by priority. The P0 flows are the paths a customer cannot afford to lose: authentication and single sign-on, contract creation, the other core journeys. A smoke suite covers exactly those and it runs on every release. It has to be perfectly green: a hundred percent pass rate, no flaky tests tolerated. If smoke fails, we do not promote.

Gate 2 — E2E test runs for the domains you actually touched. We look at what the change modified and run the affected domains, skipping the rest. For each of those domains we run the full set, every priority level from P0 downward. We don't reduce the result to a blunt pass or fail. We look at how many tests failed at each priority and judge the impact and safety of the release from those numbers: a P0 failure stops everything, while a handful of low-priority failures is a considered decision rather than an automatic block.

Gate 3 — API schema tests. Contracts drift quietly and break integrations loudly, so a schema mismatch is a blocker.

Gate 4 — Tracking the non-functional signals. 95th-percentile latency and error rate measured against the pre-release baseline, plus basic security checks.

Holding all of it up is a single rule: no silent failures. Every skipped test is tracked, every flaky test is quarantined out of the blocking set with a linked bug, an owner and an SLA to fix it and"it's just the environment" is not an argument unless you can prove it. A flaky gate everyone has learned to ignore is worse than no gate at all. During the final validation window and in the hours before a release, the staging branch freezes, so nothing new lands and the thing we validated is the thing we ship.

And once we've shipped, one more gate watches production for us. A passive dashboard tracks error rates and latency against the pre-release baseline, an agent hunts for error patterns that weren't there before and a small set of P0 journeys runs against production every few minutes for the first couple of hours. If error rate or latency jumps past baseline, we know in minutes, not in support tickets and we decide whether to roll back.

06 · The co-pilot, not the captain

Now the part I care about most.

At more than forty deploys a day, no human can hold all the gates. But I also refuse to let a machine make the release decision itself. So we drew a clear line: AI surfaces and assists, humans decide. The veto is that human, kept in the loop on purpose, as a counterweight to everything we automated around it.

AI surfaces and assists. Humans decide.

Everywhere else, AI does the judgement work that used to eat expert hours.

AI risk analysis on every cherry-pick. A score and a plain-language verdict (SAFE, CAUTION, DANGER), flagging migrations, an oversized API surface, thin testing evidence. The reviewer opens the request already knowing where to look.

Performance regressions, narrated. Our performance gate doesn't just fail a run. An agent explains the k6 regression, telling you which endpoint slowed down and by how much against baseline, straight into Slack.

Migrations, briefed before they scare anyone. Database migrations, historically the scariest part of any release, get an AI review that reads the migration, classifies the lock type and rollback feasibility and briefs the DBA, instead of dropping a raw "please review this PR" on them and hoping for the best.

"Is my ticket in production yet?" The question every engineer asks a hundred times a week is now a question you put to an agent. It reads the release state and tells you exactly where your code is: still on staging, shipping tonight, or already live.

This is where Akai, our internal agent platform, comes in and it's where most of my attention goes now. I think about it as two families of agents around every release: preventive and corrective.

Preventive — guard the integrity of the system before anything breaks. The same tedious checks on every single release that a person never quite manages to run every single release.

Corrective — act once something has already slipped.

The direction, in one line, is a path to production that increasingly runs itself. The guardrails stop being a checklist a person walks through, but rather become a layer that's simply always on.

And I'll be straight about the buzzword fatigue, because I feel it as much as anyone. We didn't reach for AI because it's fashionable. We reached for it so that the boring, repetitive, error-prone parts of shipping stop needing a human, which enables humans to ponder on the one decision that actually matters. Should this go to customers?

07 · Where we are going next

I've watched this process go from two people on a Google Meet, on opposite sides of a continent, sweating over merge conflicts to a pipeline that heals itself, gates itself and increasingly explains itself. The endgame we're building toward is the one I pitched to the leadership team some time ago and only half believed at the time: deployment as a button click, releases many times a day and a path to production where the guardrails are agentic by default and a human is there for judgement, not grunt work.

We're not all the way there. As I write this, some of the agents are further along than others and I still get the occasional 3AM alert to remind me the machine is a work in progress. But the shape is set and it's reproducible. None of it depends on our particular stack. It rests on five ideas you can borrow today.

And then there's the other half of the story: how a feature actually reaches a customer once the code is safely in production. Flags, cohorts, phased rollouts, kill switches and the art of letting exactly the right people see exactly the right thing at exactly the right time. That's a whole system of its own and it's where rolling out with confidence really earns its name.

One more thing. The software release approach we have today didn't build itself. Massive credit to the technical team that rolled up their sleeves and got it off the ground. Going fast and safely is a team sport and I got to play it with the best of the best. They are the main reason I'm able to roll out with confidence today.