---
title: "A gate is only as real as its run: three of mine that were not"
description: "An AI-native delivery loop is a chain of gates. Three of my own gates looked real and were not: one never ran, one ran and checked nothing, one failed and nobody listened."
date: 2026-10-24
canonical: https://pabloalbaladejo.com/blog/a-gate-is-only-as-real-as-its-run
---

# A gate is only as real as its run: three of mine that were not

A gate is a check that can stop a change before it reaches users. An AI-native delivery loop is mostly gates: hooks, evals, reviews, required checks, a merge, a deploy. In August, Anthropic published a playbook for that loop, and I agree with most of it. But this week I checked my own gates against one simple question: did this gate run, and could it block? Three of them failed that question. Below are the three, what the playbook leaves out, and the merge rule I use now.

## What the playbook gets right

[The AI-native SDLC playbook](https://claude.com/resources/articles/the-ai-native-sdlc-playbook) describes six stages, from plan to maintain. Every stage commits an artifact that the next stage reads. Hooks are deterministic guardrails. Autonomy is tiered by environment, and one line sets the limit: "The agent may act up to the production gate and cannot pass it." Monitoring uses fixed statistical rules, and an agent is called only when a band is breached. The playbook says it plainly: "Detection stays deterministic."

I run most of this already. Kaiord, my open-source training platform, is built by agents. Every non-trivial change starts as a written proposal, a design and a task list. Hooks block architecture violations before a commit. Required checks guard `main`. So when I read the playbook, I did not find a missing stage. I found a missing question.

## Gate one: the eval that never ran

For months, two of my essays quoted this line as a gate I shipped:

```ts
process.exit(report.passRate >= 90 ? 0 : 1);
```

The line is real. It is [line 65 of the eval runner](https://github.com/pablo-albaladejo/kaiord/blob/23c788cd843e6af38f383b295451e0b6ddbe944c/packages/ai/src/evals/run-evals.ts#L65), as it was. But the runner gets its model through a loader that throws without a provider API key. The project has no key. So the comparison never executed, not once. In [#1232](https://github.com/pablo-albaladejo/kaiord/pull/1232) I removed the threshold. The file header now says what is true: "A comparison that cannot run is not a gate, and one dressed as a gate is the failure this suite exists to remove."

This is the gate the playbook leans on most. It asks teams to run 20 to 50 real tasks whenever `CLAUDE.md`, skills or hooks change, and to review any change that drops the pass rate. It does not say how to prove the suite ran. It also does not say how many runs one decision needs. That matters, because one run is noisy. A drop in the pass rate can be pure chance. Take a fixed 90% bar as an example. With 20 to 50 tasks, a system with a true 92% pass rate falls under it in one run of every five at best, and in two of every five at worst. With Kaiord's 22 benchmarks, it is about one run in four. A rule that fires on any drop fires even more often. Good changes fail, people learn to re-run until green, and then the gate is noise. I wrote about what one aggregate number hides in [A 90% pass rate is not a gate](https://pabloalbaladejo.com/blog/a-pass-rate-is-not-a-gate). What I plan instead is k-of-n runs measured against an A/A baseline, where I first run the same version against itself to learn the noise. I have not shipped that yet.

## Gate two: the review that ran and checked nothing

Kaiord pull requests get an AI code review from CodeRabbit. On [#1333](https://github.com/pablo-albaladejo/kaiord/pull/1333), the CodeRabbit status check is green: `SUCCESS`. The CodeRabbit comment on the same pull request says "Review limit reached". The free review quota was empty. No review happened, and the check still reported success.

I merged that pull request. The other gates on it were real, so nothing broke. I knew the review had not run, because I asked for it again and it was rate limited again. But the checks list still showed CodeRabbit green, and a reader of that list sees a review that never happened. This is the quiet failure mode of AI review. **A reviewer that runs out of budget does not fail. It reports success with nothing inside.** The playbook puts an AI reviewer on every pull request. It should also say what that reviewer reports when it does not review.

## Gate three: the alarm that nobody hears

Kaiord has a [security audit workflow](https://github.com/pablo-albaladejo/kaiord/blob/main/.github/workflows/security.yml). It runs on every push to `main`. Until this month, when it found high or critical advisories, it failed the job and opened a new issue. From late September, it found at least two high advisories, and for one week a critical one too. So the job failed on every push: from 30 September to 17 October, 97 of 97 runs on `main` were red. And each red run on `main` opened one more issue. On 2026-10-17 there were [99 open issues](https://github.com/pablo-albaladejo/kaiord/issues?q=is%3Aissue+%22Security+Alert%22+in%3Atitle) whose title starts with "Security Alert".

The job was not in the list of required checks. So it failed, and merges continued. In April, I closed four older alerts by hand in a cleanup. The issues came back, and there were more of them each week. In [my essay on shipping with agents](https://pabloalbaladejo.com/blog/shipping-production-software-with-ai-agents), I wrote that any red job must be treated as red. My own repo broke that rule on 97 pushes in a row.

On 17 October I fixed both halves. [#1296](https://github.com/pablo-albaladejo/kaiord/pull/1296) updated the packages behind the advisories, so the audit on `main` is green again. [#1510](https://github.com/pablo-albaladejo/kaiord/pull/1510) changed the workflow. It now keeps one open alert issue, updates it on every red run, and closes it when the audit is clean. A report that cannot be read now fails the job, so an audit that did not run cannot look green. By 23 October, every old copy was closed. The job is still not a required check, and that decision is still open.

The playbook is right that detection should stay deterministic. This audit is deterministic. The problem is what happens after detection. A red check that blocks nothing and an issue queue that only grows are not a closed loop. They teach everyone to stop looking.

## The question the loop needs

All three gates had a status. None had proof. So I now ask three questions of every gate:

1. **Did it run?** On this change, with real inputs, not on a path that exits early.
2. **Did it check something?** It should report a count: tasks scored, files reviewed, advisories found. Zero is a result to see, not to hide.
3. **Can it block?** If it goes red, does something stop? If not, it is a log line, and it should be named like one.

This is also how I think about the merge now. The playbook's tiered autonomy is the right shape, so here is mine. On this personal site, an agent merges its own pull request. It may do that only after a separate review agent approves and CI is green on the exact commit. After the merge, it watches the deploy until it finishes. New essays, like this one, also need my approval of the content. On Kaiord and my other repos, I press merge. Before this month, the shipping essay said the merge was "the one thing I never delegate". That stopped being true, so [I corrected it](https://pabloalbaladejo.com/blog/shipping-production-software-with-ai-agents).

## What this does not fix

This essay has four limits. First, the eval gate I describe as the alternative is not shipped. Today Kaiord has no live eval gate at all, and I would rather say that than show a line that does not run.

Second, you cannot check my site's merge gate yourself. That repository is private. The Kaiord gates in this essay are public, and every link above opens the real file or pull request.

Third, there are probably more. While checking a draft of this essay, a review agent found a fourth. Kaiord has a Claude hook that should flag architecture violations in `packages/core`. It filters changed files with `includes("/packages/core/src/")`, but git prints those paths with no leading slash. So the hook runs and never matches a file. The layer rules still hold before every commit, because ESLint checks the architecture boundaries there. But the hook was one more check that passed with nothing inside. I fixed the filter in [#1516](https://github.com/pablo-albaladejo/kaiord/pull/1516), with tests for the path format git really prints.

Fourth, the audit trail is weaker than the playbook assumes. GitHub records every merge on my site as mine, because the agent uses my token. The record cannot tell who pressed the button. Today the proof that an independent review ran lives in pull request comments and in the session, not in a field that GitHub enforces.

## A checklist you can lift

- **Write down what every gate checked, as a number.** "22 tasks scored", "14 files reviewed". An empty run then looks empty.
- **Turn skips into failures.** A missing key, a rate limit or a timeout is red, never green.
- **Compare red jobs with required checks every week.** Each red job that is not required becomes a requirement or gets deleted.
- **Make alerts update one issue.** Find the open issue by a fixed title or label, update it, and close it when the count reaches zero.
- **Before you gate on a pass rate, measure the noise.** Run the same version against itself first. Then pick the threshold and the number of runs from that measurement.
- **Write your autonomy tiers down, per repo.** Put them in the file your agents read, and say what happens after the merge too.

A gate is a promise that a check ran and could have stopped you. Check the promise, not the color.

## Related writing

- [A 90% pass rate is not a gate](https://pabloalbaladejo.com/blog/a-pass-rate-is-not-a-gate): why one aggregate number cannot gate an LLM system, and the criteria table that can.
- [Shipping production software with AI agents](https://pabloalbaladejo.com/blog/shipping-production-software-with-ai-agents): the agent workflow these gates belong to, and the merge claim this essay corrects.
- [Evaluating LLM outputs in production](https://pabloalbaladejo.com/blog/evaluating-llm-outputs-in-production): the eval harness behind gate one, with the mechanics of each assertion.
- [Observability for LLM pipelines](https://pabloalbaladejo.com/blog/observability-for-llm-pipelines): what happens after deploy, where detection has to stay deterministic.
- [AI-DLC: the only gates that count are the ones code enforces](https://pabloalbaladejo.com/blog/ai-dlc-gates-only-count-when-code-enforces-them): the next check after this one, whether the change a gate checks can also lower the gate.
- [A discovery file is a promise](https://pabloalbaladejo.com/blog/a-discovery-file-is-a-promise): the same rule for agent discovery, where a card is a promise and a live round trip checks it.

## FAQ

### What does "a gate is only as real as its run" mean?

A gate protects you only if three things are true on every change. It executed, it checked something real, and its result can stop the merge. A gate that shows a green or red status without all three is decoration. In my open-source repo, Kaiord, I found three that failed the test. One never executed, one executed and checked nothing, and one failed on every push without blocking anything.

### What does Anthropic's AI-native SDLC playbook leave out?

The playbook gets the loop right: committed artifacts at every stage, hooks as deterministic guardrails, tiered autonomy, and deterministic detection. It asks teams to gate agent configuration changes on eval results, and to review any change that drops the pass rate. It does not say how to prove that the eval ran, how many runs one decision needs, or how to pick the threshold. Take a fixed 90% bar and 20 to 50 tasks. A system with a true 92% pass rate fails it in one run of five at best, and in two of five at worst.

### Do you let an AI agent merge its own pull requests?

It depends on the repo. On my personal site, an agent merges its own pull request after a separate review agent approves and CI is green on the exact commit. Then it watches the deploy until it finishes. New essays also need my approval of the content. On Kaiord and my other repos, I press merge myself.

### How do I check that a CI gate actually runs?

Make every check report what it checked, as a count, so an empty run is visible. Treat a skipped, rate-limited or credential-less run as a failure, not a success. Compare the list of red jobs with the list of required checks. A job that is red and not required is either a missing requirement or noise to delete. Make alerts update one open issue instead of opening a new one on every run.

## Sources

- [Anthropic: The AI-native SDLC playbook (the loop this essay responds to)](https://claude.com/resources/articles/the-ai-native-sdlc-playbook)
- [Kaiord run-evals.ts at 23c788cd: the 90% exit line, as it was before removal](https://github.com/pablo-albaladejo/kaiord/blob/23c788cd843e6af38f383b295451e0b6ddbe944c/packages/ai/src/evals/run-evals.ts#L65)
- [Kaiord #1232: removes the threshold and marks the runner INERT](https://github.com/pablo-albaladejo/kaiord/pull/1232)
- [Kaiord #1333: CodeRabbit check SUCCESS next to a "Review limit reached" comment](https://github.com/pablo-albaladejo/kaiord/pull/1333)
- [Kaiord security.yml before #1510: the audit job that opened a new issue on every red run on main](https://github.com/pablo-albaladejo/kaiord/blob/e7b6fbf4/.github/workflows/security.yml)
- [Kaiord #1510: one alert issue instead of one per red run, and no green for an audit that did not run](https://github.com/pablo-albaladejo/kaiord/pull/1510)