Pablo Albaladejo

AGENTIC DEV · · 8 min read

A gate is only as real as its run: three of mine that were not

An AI-native delivery loop is a chain of gates. Three of my own gates looked real and were not: one never ran, one ran and checked nothing, one failed and nobody listened.

A gate is a check that can stop a change before it reaches users. An AI-native delivery loop is mostly gates: hooks, evals, reviews, required checks, a merge, a deploy. In August, Anthropic published a playbook for that loop, and I agree with most of it. But this week I checked my own gates against one simple question: did this gate run, and could it block? Three of them failed that question. Below are the three, what the playbook leaves out, and the merge rule I use now.

What the playbook gets right

The AI-native SDLC playbook describes six stages, from plan to maintain. Every stage commits an artifact that the next stage reads. Hooks are deterministic guardrails. Autonomy is tiered by environment, and one line sets the limit: “The agent may act up to the production gate and cannot pass it.” Monitoring uses fixed statistical rules, and an agent is called only when a band is breached. The playbook says it plainly: “Detection stays deterministic.”

I run most of this already. Kaiord, my open-source training platform, is built by agents. Every non-trivial change starts as a written proposal, a design and a task list. Hooks block architecture violations before a commit. Required checks guard main. So when I read the playbook, I did not find a missing stage. I found a missing question.

Gate one: the eval that never ran

For months, two of my essays quoted this line as a gate I shipped:

process.exit(report.passRate >= 90 ? 0 : 1);

The line is real. It is line 65 of the eval runner, as it was. But the runner gets its model through a loader that throws without a provider API key. The project has no key. So the comparison never executed, not once. In #1232 I removed the threshold. The file header now says what is true: “A comparison that cannot run is not a gate, and one dressed as a gate is the failure this suite exists to remove.”

This is the gate the playbook leans on most. It asks teams to run 20 to 50 real tasks whenever CLAUDE.md, skills or hooks change, and to review any change that drops the pass rate. It does not say how to prove the suite ran. It also does not say how many runs one decision needs. That matters, because one run is noisy. A drop in the pass rate can be pure chance. Take a fixed 90% bar as an example. With 20 to 50 tasks, a system with a true 92% pass rate falls under it in one run of every five at best, and in two of every five at worst. With Kaiord’s 22 benchmarks, it is about one run in four. A rule that fires on any drop fires even more often. Good changes fail, people learn to re-run until green, and then the gate is noise. I wrote about what one aggregate number hides in A 90% pass rate is not a gate. What I plan instead is k-of-n runs measured against an A/A baseline, where I first run the same version against itself to learn the noise. I have not shipped that yet.

Gate two: the review that ran and checked nothing

Kaiord pull requests get an AI code review from CodeRabbit. On #1333, the CodeRabbit status check is green: SUCCESS. The CodeRabbit comment on the same pull request says “Review limit reached”. The free review quota was empty. No review happened, and the check still reported success.

I merged that pull request. The other gates on it were real, so nothing broke. I knew the review had not run, because I asked for it again and it was rate limited again. But the checks list still showed CodeRabbit green, and a reader of that list sees a review that never happened. This is the quiet failure mode of AI review. A reviewer that runs out of budget does not fail. It reports success with nothing inside. The playbook puts an AI reviewer on every pull request. It should also say what that reviewer reports when it does not review.

Gate three: the alarm that nobody hears

Kaiord has a security audit workflow. It runs on every push to main. Until this month, when it found high or critical advisories, it failed the job and opened a new issue. From late September, it found at least two high advisories, and for one week a critical one too. So the job failed on every push: from 30 September to 17 October, 97 of 97 runs on main were red. And each red run on main opened one more issue. On 2026-10-17 there were 99 open issues whose title starts with “Security Alert”.

The job was not in the list of required checks. So it failed, and merges continued. In April, I closed four older alerts by hand in a cleanup. The issues came back, and there were more of them each week. In my essay on shipping with agents, I wrote that any red job must be treated as red. My own repo broke that rule on 97 pushes in a row.

On 17 October I fixed both halves. #1296 updated the packages behind the advisories, so the audit on main is green again. #1510 changed the workflow. It now keeps one open alert issue, updates it on every red run, and closes it when the audit is clean. A report that cannot be read now fails the job, so an audit that did not run cannot look green. By 23 October, every old copy was closed. The job is still not a required check, and that decision is still open.

The playbook is right that detection should stay deterministic. This audit is deterministic. The problem is what happens after detection. A red check that blocks nothing and an issue queue that only grows are not a closed loop. They teach everyone to stop looking.

The question the loop needs

All three gates had a status. None had proof. So I now ask three questions of every gate:

  1. Did it run? On this change, with real inputs, not on a path that exits early.
  2. Did it check something? It should report a count: tasks scored, files reviewed, advisories found. Zero is a result to see, not to hide.
  3. Can it block? If it goes red, does something stop? If not, it is a log line, and it should be named like one.

This is also how I think about the merge now. The playbook’s tiered autonomy is the right shape, so here is mine. On this personal site, an agent merges its own pull request. It may do that only after a separate review agent approves and CI is green on the exact commit. After the merge, it watches the deploy until it finishes. New essays, like this one, also need my approval of the content. On Kaiord and my other repos, I press merge. Before this month, the shipping essay said the merge was “the one thing I never delegate”. That stopped being true, so I corrected it.

What this does not fix

This essay has four limits. First, the eval gate I describe as the alternative is not shipped. Today Kaiord has no live eval gate at all, and I would rather say that than show a line that does not run.

Second, you cannot check my site’s merge gate yourself. That repository is private. The Kaiord gates in this essay are public, and every link above opens the real file or pull request.

Third, there are probably more. While checking a draft of this essay, a review agent found a fourth. Kaiord has a Claude hook that should flag architecture violations in packages/core. It filters changed files with includes("/packages/core/src/"), but git prints those paths with no leading slash. So the hook runs and never matches a file. The layer rules still hold before every commit, because ESLint checks the architecture boundaries there. But the hook was one more check that passed with nothing inside. I fixed the filter in #1516, with tests for the path format git really prints.

Fourth, the audit trail is weaker than the playbook assumes. GitHub records every merge on my site as mine, because the agent uses my token. The record cannot tell who pressed the button. Today the proof that an independent review ran lives in pull request comments and in the session, not in a field that GitHub enforces.

A checklist you can lift

A gate is a promise that a check ran and could have stopped you. Check the promise, not the color.

FAQ

What does "a gate is only as real as its run" mean?
A gate protects you only if three things are true on every change. It executed, it checked something real, and its result can stop the merge. A gate that shows a green or red status without all three is decoration. In my open-source repo, Kaiord, I found three that failed the test. One never executed, one executed and checked nothing, and one failed on every push without blocking anything.
What does Anthropic's AI-native SDLC playbook leave out?
The playbook gets the loop right: committed artifacts at every stage, hooks as deterministic guardrails, tiered autonomy, and deterministic detection. It asks teams to gate agent configuration changes on eval results, and to review any change that drops the pass rate. It does not say how to prove that the eval ran, how many runs one decision needs, or how to pick the threshold. Take a fixed 90% bar and 20 to 50 tasks. A system with a true 92% pass rate fails it in one run of five at best, and in two of five at worst.
Do you let an AI agent merge its own pull requests?
It depends on the repo. On my personal site, an agent merges its own pull request after a separate review agent approves and CI is green on the exact commit. Then it watches the deploy until it finishes. New essays also need my approval of the content. On Kaiord and my other repos, I press merge myself.
How do I check that a CI gate actually runs?
Make every check report what it checked, as a count, so an empty run is visible. Treat a skipped, rate-limited or credential-less run as a failure, not a success. Compare the list of red jobs with the list of required checks. A job that is red and not required is either a missing requirement or noise to delete. Make alerts update one open issue instead of opening a new one on every run.

SOURCES & PROOF