Rolling Out with Confidence, Part 3: The agentic team you won't find on the org chart

Part 3 of the "Rolling Out with Confidence" series. Part 1: The Path to Production. Part 2: The Release Control Center.

Friday, 8:06 am. I got a message from our QA tech lead, Maia. The message said "Staging is down."

Kostiantyn, our release tech lead, was already in the thread with a dashboard link. The release was due to ship in two hours. Between the three of us, the next two hours were mapped out before anyone had finished typing, because we'd walked this exact path so many times we could do it with our eyes closed.

And that was the problem. We could do it with our eyes closed.

Every morning before the release, the same checklist. Is the environment actually up, or is a dependency lying? Of the forty-seven red tests on the board, how many are real? Which PR broke them and did it arrive through dev, through a sync, or through a cherry-pick somebody filed at midnight? Is that migration safe to run? Did production get worse after yesterday's release or does it just look that way because a service restarted?

An hour each, median, every day. Not because any step was hard. Because every step needed a brain and the brains were two of the best people I have had the opportunity to work with.

In Part 1, I wrote that the guardrails should become "a layer that's simply always on". That was September. This is what Maia and Kostiantyn did with that sentence. They built the agents to take over their release validation work. This article is about our agent team members, what they're allowed to do, what they're not, and what changed for the two humans who used to validate daily the checklist.

Part 1 ended on two families of agents, preventive and corrective and a promise that they'd take the tedious checks off people's plates. Here is the crew, as it stands around every release today.

Before you hire a new team member, you write a job description. Ours turned out to be very simple “As our AI agent team member, you surface and assist, but still let us humans decide”.

An agent in our release machine may read anything, from test results, pull requests and logs to even migrations, metrics and Slack threads. It may check what it reads against rules we wrote down. It may narrate, which means group, explain, attribute and recommend. And it may post, in the thread where the humans already look.

It may not approve a release. It may not merge, retry, skip, roll back or force anything. It may not change a verdict. Where a verdict exists, code computes it and the agent explains it.

That sounds like a small job. It's the reason the whole thing worked. Nobody had to trust an agent with production before they'd watched it be right about staging a few hundred times. And because every agent posts where people already read, adoption wasn't a rollout. It was a thread people started following, and then guiding themselves by.

Who broke staging?

Some of our agents live in the staging quality loop and the first one answers the question Maia was asked more frequently than any other question.

The E2E RCA Agent. The staging tests run finishes, half the board is red and the release is waiting on you. Is it a bug? A stale test? Did the database reset mid-run? Whose PR was it? The agent does the digging. You type one command in the test-results channel, go get the coffee you were going to get anyway and come back to a report in the thread.

The report doesn't list forty-seven failures. It lists a handful of causes, with the tests that broke for the same reason grouped together. For each cause it names the PR behind it, across code and test repositories. When the cause isn't a PR at all, it says so. A database reset, a rollout that landed mid-run or a service going down is named as the cause, instead of being pinned on whoever merged last. Each cause carries a risk level for production, with the reason attached. And the noise, the flaky tests and the long-broken ones, is listed separately so the real problems don't get buried under it.

One run from late September had 692 tests, with 47 failures and 4 real causes. The agent has already traced failures to real bugs, including a pagination loop that was quietly running a backend service out of memory. This helped us patch an issue that could have remained undetected and would have taken down production with it on release.

The Staging Forecaster. Then we went one step earlier in time. If a test is going to fail on staging on release day, it is very often already failing in dev today; nobody used to look actively in this direction. The forecast reads our close to 700 release-regression tests against the six dev separate runs and tells each team which of their staging tests will be broken, before the dev code ever reaches staging. In one recent run it caught 21 across 7 teams. Twenty-one fixes on a quiet afternoon instead of during the release window.

The Regression Suite Curator. A suite that only grows is a suite nobody trusts. Part 1's small, trusted, prioritised test suite doesn't stay small on its own. Reviewing close to 700 tests by hand isn't realistic, so an agent reads every release-regression test and says whether it belongs in pre-release or in the dev validation. Additionally, it decides whether its priority tag is right, with a reason and a link to the covered code for every verdict. The first full audit came back blunt. 240 tests should stay, 137 can move to dev-only, 295 need a decision from the owning team, 20 are broken, and 271 carry the wrong priority tag. Critical flows such as login, payments and contracts are always protected. The team makes the final call, and the same check runs on a pull request adding new tests before new tests even get to the staging suite.

Staging Health Reporter. This one generates a per-team failure report on their service availability, which together offers an overview of the staging health, without anyone scrolling through runs and matching request IDs in the logging system to detect the cause of a failure was a non healthy environment.

Notice what none of them do. None of them fixes a test, retries a run or merges anything. They report. The humans who used to do the digging now read a summary and decide.

The release validation is no longer silent

Part 1 described the gate ladder. A staging release candidate climbs Gates 0 to 4 in the staging environment before it earns promotion, and a fifth gate watches production afterwards. The ladder hasn't changed. What changed is that it now speaks.

The readiness brief. At the approval gate, every release gets a brief. Seven sections, each green, amber, red or unknown. The verdict, GO, GO-WITH-RISK or NO-GO, is computed in code from rules anyone can read. The AI's only job is to narrate the top three concerns in plain language and post them to the release thread. It cannot change the verdict.

The gate learned the difference between red and flaky. Before the brief is written, every failing test is classified against thirty days of its own history as a new regression, a known flaky test, or unknown. Only regressions turn the gate red. Failures map to owning teams by Slack handle. Part 1's rule was "no silent failures"; this is the part that stops the loud ones from being false alarms.

And the two agents Part 1 already introduced have simply kept working. The cherry-pick risk scoring was run on 618 approved feature or bugfix requests in one October week. The post-release watch still samples production at fifteen, thirty and sixty minutes after each deployment and says all is clear, we should continue watching or that regression is already suspected. It recommends a rollback. It has never performed one and that's intended.

The migration that left the deploy window

Part 1 mentioned, almost in passing, that migrations get an AI briefing before they even reach the DBA team for approval. What it didn't say is where the pre-requisite migrations used to run. Each migration ran inside the deployment, as a step of the process. If one migration locked a table, we learned about it from the alerts, when damage was already done and the release could not proceed according to schedule.

For some time now, our migrations have run through the Migrations Control Center. They left the deploy window entirely and got a process of their own. Each one passes four checks before it's allowed anywhere near production. We check what category it is, whether it's isolated, whether it's been modified since we validated it and whether its dependencies are in the release scope. The AI briefing from Part 1 is still there, now attached to a migration that has a status, a place in the queue and an owner.

The reading is advisory. It never blocks. The DBA holds the only write permission and every action, run, skip, manual apply or override lands in an audit log and a Slack thread. A release cannot ship while a pre-requisite migration is unapplied or skipped without someone confirming the skip.

In one of our conversations, Kostiantyn summarised the situation very well. Before this, releases shipped blind on migrations. On a given morning the gate on our release now reads something along the lines of "blocked, 47 pending, 4 applied this window". A month earlier, that sentence could not have been written by anyone.

One message to production

The last hire is the one that surprised people, because it looks like a shortcut. It is actually the opposite.

Getting a merged change to production out of band used to mean filing a cherry-pick request to staging, applying CI fixes and chasing reviewers, filing another one to master, chasing them again and finally checking the deployment. Since October, you tag the release agent in our fast lane channel with a pull request or a Jira key and tell it to release. One request now covers all the steps. One card shows the whole route and updates in place. You're tagged only when you need to act. Our QA agent runs on the way through; missing evidence generates a warning and red CI or a failing behaviour still blocks.

The agent never approves, merges or forces. People still review and merge on GitHub. It went from a sketch to production in a couple of days. It didn't build a new lane. It drove the one we'd spent a year making trustworthy and it drives it using fewer permissions than the humans needed to have.

The rest of the crew

Those are the hires with the most hours on the clock. Around them sits the rest of the team Part 1 sketched, each one taking a check that used to live in somebody's head.

Commit parity. The same commit should exist in dev, staging and master and a sync of branches can quietly drop one. An agent compares the three before every release and names what's not compliant.

Release versus reality. Do the contents of the release match what the ticket tracker thinks we shipped? An agent reads both and flags the differences, so when you ask "Is my ticket in production?" the answer relies directly on the codebase changes.

Cause and revert. The post-release watch finds the spike. Its partner ties the spike to the change that most likely caused it and drafts the recommendation to revert, with the evidence attached, for a human to act on.

Escalation intelligence. Client escalations get clustered by root cause and correlated with the release calendar, so a pile of tickets becomes a signal about our own pipeline instead of about whoever shouted loudest.

The Ghost User. Our newest project is an agent whose only job is to know the platform. Every repository, every component, every API call a test has ever made, fed continuously into it, so it never goes stale. With that context, root-cause analysis stops depending on test artefacts. Fixes applied to broken tests become accurate instead of hopeful and new one-time validations can be reduced to a sentence, such as "validate that contract creation works", rather than a script. It walks the product like a customer would, remembers every way to do a thing and re-validates all of them when a change touches that area. A test framework is not required and a full reasoning trace is provided for every step.

What the humans do now

We still start the day with the release. We start it by reading. The brief is on the thread. The red run already has an owner. The migration already has a status. The hour previously used to assemble the evidence is now spent on deciding what to do with the evidence.
That's how our way of working has changed in the past months. Agents read, check and narrate. A release captain still clicks go. A DBA still clicks run. However, the release team got a lot of new supportive colleagues that help us make decisions.

If you're building your own version, the ideas travel as they did in the first two parts.

1. Compute the verdict in code and let the AI narrate it, because a verdict you can read as a rule is one you can argue with, fix and trust.

2. Give every red run an owner, not a dashboard; the expensive part of a failure is the hour finding out whose it is. Say "it's not your code" out loud, because half the value of root-cause analysis is the engineers you didn't page.

3. When an agent drives a process humans trust, give it fewer permissions than the humans, not more.

4. Let the people who do the chore build the agent. The best agents we have were built by the people doing the manual work. They knew exactly the expected result. They'd been producing it by hand every morning.

Our new team members are not part of the org chart. But the people who brought them to the team are the reason I can now calmly read a "Staging is down" message on a Friday at 8 am. Together with the message, I also receive a fully automated explanation and action plan, before I even have the first sip of my coffee.