From prompt rules to failing tests: building sensors and gates for coding agents

From prompt rules to failing tests: building sensors and gates for coding agents

When you hand work off to a coding agent, what you really care about is whether the code that comes back still matches your intent: the architecture you chose, the boundaries you drew, and the conventions your team spent months agreeing on. The earlier you spot a change wandering off, the less you have to untangle later.

You have two basic levers to keep an agent on track. You can tell it the rule upfront in CLAUDE.md or AGENTS.md. Or you can check the code after the fact, with a test that fails when the rule is broken.

I originally started this experiment expecting to prove that instructions don’t work—that agents inevitably forget written rules under pressure, and that you need hard, failing checks to hold the line.

That’s not what happened.

Modern models are surprisingly good at following rules you explicitly state. For a lot of everyday cases, the fastest way to keep an agent on track is simply writing a clear sentence in its instructions.

So why bother building automated checks at all? Four practical reasons came out of my testing:

Rather than treating prompts and tests as competitors, they work best together as a cycle. Start by writing the rule down. Turn the boundaries that matter into automated checks. As your instruction file gets bloated, move the rare rules out into checks. And when a check keeps firing in practice, take the hint and add that rule back to your prompt instructions.

A quick distinction on terms: adherence is the model following instructions it was given. Conformance is whether the code actually meets the rule in the repository, whoever wrote it. Drift is the widening gap between what you intended and what the agent actually built over a session. Guidance buys adherence. Only a check confirms conformance.

It’s also worth being realistic about boundaries. An import check only verifies static imports. It won’t catch dynamic runtime calls or bad business logic, and it won’t stop an agent with shell access from deleting the config file.

Below is a walkthrough using a small Python service: watching an architecture check catch a real violation, feeding a clear diagnostic back to the agent, and seeing the clean repair. From there, I look at placing the check in three different spots (an agent stop hook, a pre-commit hook, and CI), each with different trade-offs. Then we’ll look at the data: a few hundred agent sessions across five models from two vendors, with and without checks, and what that data actually tells us about keeping agents on track.

Everything lives in the companion example: the service, the sensor, all three integrations, the experiment tooling, and the captured output quoted below.

Guidance, sensors, gates, and repair loops

This way of thinking comes out of Birgitta Böckeler’s articles on harness engineering and maintainability sensors for coding agents. The core idea is simple: an agent is a model plus its harness. You can’t retrain the underlying LLM to reliably respect your specific architectural boundaries, but you can build a harness around it to keep it on track.

In that framework, you manage an agent through two main mechanisms:

To make this practical, I borrow the split Dasith Wijesiriwardena drew in his API Days talk, Don’t Waste Your Backpressure: a sensor checks the agent’s work, and a gate turns its findings into backpressure the agent must resolve before it can proceed. Add guidance, and a repair loop (my addition, not Dasith’s), and you get four concrete pieces:

That fourth piece is the one teams most frequently overlook. A red CI job is a sensor and a gate. It only becomes a repair loop if something takes that failure, hands it back to the agent, and asks it to try again.

guidance onlyagentinstruction?done?sensor, gate, repair looprepair loopagent✓sensor✗ controller → repositorygate✓done✓123456
  1. Guidance only: the change finishes with “?”. Nothing checked it.
  2. The check (sensor) raises a flag: controller → repository. Nothing is blocked yet.
  3. The gate drops: the turn can’t finish.
  4. The repair loop: the diagnostic goes back to the agent.
  5. The agent makes a new attempt (the token changes shape).
  6. The check passes, the flag clears, the gate lifts: done ✓.
The same change on two tracks. With only an instruction, nobody knows whether the rule held. With a sensor, a gate and a repair loop, the problem is found, the turn is stopped and the agent tries again.

If an agent follows instructions most of the time, building automated checks might feel like overkill. But ‘most of the time’ is an invisible metric, and your prompt instructions are never complete. A deterministic check doesn’t make the model smarter; it simply removes your reliance on prompt adherence for that specific rule. You trade a few seconds of compute to save human review time, and compute scales much better than senior engineers.

The rule, adapted to Python

A classic example of an architectural boundary is keeping API controllers separated from the data layer. Controllers should decode HTTP requests, invoke application services, and format responses. Repositories handle persistence. When a controller starts querying a repository directly, you end up with query logic and business rules tangled into your HTTP layer.

The rule is Dasith’s worked example. He checks it in a layered .NET codebase with architecture fitness tests. Everything below is my adaptation of his rule to Python, not his code.

The companion Python service has three clean layers:

The sensor uses Import Linter with a forbidden contract:

[importlinter]
root_packages =
    orders

[importlinter:contract:controllers-not-repositories]
name = Controllers must not import infrastructure repositories directly
type = forbidden
source_modules =
    orders.controllers
forbidden_modules =
    orders.infrastructure
allow_indirect_imports = True

That last line is critical. Import Linter checks indirect imports by default. In this design, the valid path is intentionally indirect: controller → application service → repository. Setting allow_indirect_imports = True keeps that clean path legal. The contract catches exactly one thing: a direct import edge from anything under orders.controllers to anything under orders.infrastructure.

Let’s also be clear about what an import edge proves. It’s a proxy for architectural intent, and it has distinct blind spots:

The guarantee is narrow: no direct import edge from controllers to infrastructure inside the checked package at that specific snapshot. That narrowness is what makes it cheap and deterministic. Within that scope, it holds every time. Outside it, you are back to guidance.

A failing change, a diagnostic and a repair

The sensor runs via one command, integrations/check_architecture.py, and every integration placement runs this same command. On the baseline codebase, it passes cleanly:

$ .venv/bin/python -m pytest -q tests/unit/orders
22 passed
$ .venv/bin/python integrations/check_architecture.py
ARCHITECTURE CHECK PASSED (snapshot 470476b02ddd): Controllers must not import infrastructure repositories directly

Now consider the kind of change a coding agent might make when trying to be efficient. The list endpoint reads straight from the repository instead of going through the service. It’s shorter, and it behaves identically:

+from orders.infrastructure.repository import InMemoryOrderRepository
-    def __init__(self, service: OrderService) -> None:
+    def __init__(self, service: OrderService, repository: InMemoryOrderRepository) -> None:
+        self._repository = repository
-        orders = self._service.list_for_customer(customer_id)
+        orders = self._repository.list_for_customer(customer_id)

The unit tests still pass: 22 of 22. They test behaviour, so they can’t see the structural shortcut. The sensor can:

ARCHITECTURE VIOLATION
Rule broken: Controllers must not import infrastructure repositories directly
  orders.controllers.orders -> orders.infrastructure.repository (line 16)
How to fix: Controllers may decode requests, call orders.application services and map responses. Route the data access through the application service (add a service method if needed) and remove the controller's import of orders.infrastructure.
Snapshot: 3ee1fecf56ac
Repair the code; do not edit .importlinter, the gates package or tests, and do not move code out of the checked package to make this pass.

The diagnostic is tailored for an agent: it names the broken rule, the offending edge and line number, the expected fix, and explicit instructions not to tamper with the test suite or configuration to force a pass.

The repair routes the read back through the service. The sensor passes again, and the snapshot id returns to the baseline’s 470476b02ddd:

$ .venv/bin/python integrations/check_architecture.py
ARCHITECTURE CHECK PASSED (snapshot 470476b02ddd): Controllers must not import infrastructure repositories directly

Every command output above comes from the walkthrough transcript, recorded on 2026-11-19 with Python 3.12.15, Import Linter 2.17, and pre-commit 4.7.0. The README includes the copy-and-paste commands to test it yourself.

The sensor produces three distinct outcomes:

ExitResultMeaning
0passThe package was analysed and the rule held.
1violationThe rule was broken. The output names each offending import.
3checker errorThe rule was not checked: a syntax error, a missing tool, a broken or renamed contract.

Handling that third exit code is essential. lint-imports exits with code 1 for broken contracts as well as unexpected tool crashes, syntax errors, or missing config files. If your gate relies solely on exit codes, an agent that leaves a half-written module triggers a generic architecture violation, sending it hunting for an import that doesn’t exist. Worse, a loose wrapper might treat an unexpected exit code as a passing state. The sensor script parses the report directly, verifies the contract actually executed, and flags tool failures as checker errors.

What the import check can’t see

Here is that same controller, fully compliant with the import rule. It imports only the application service. But look at what it decides to do with order pricing:

def _priced_body(order: Order) -> dict[str, Any]:
    subtotal = sum(i.quantity * i.unit_price_cents for i in order.items)
    # Big orders get 10% off.
    discount = subtotal * 10 // 100 if subtotal >= 10_000 else 0
    return {
        **_order_body(order),
        "subtotal_cents": subtotal,
        "discount_cents": discount,
        "total_cents": subtotal - discount,
    }

Decoding requests and formatting responses belong in a controller. Calculating discounts and hard-coding pricing thresholds do not. The import contract passes this file (the semantic_leakage fixture) without hesitation because the imports look clean. The problem is purely behavioural: the controller is making business decisions.

The pragmatic answer is to split the rubric into two buckets. Structural facts like import boundaries become deterministic tests. Semantic judgments go to a smaller inferential check that evaluates only what is left.

The rubric is intentionally narrow. Controllers may validate request fields, call services, handle exceptions, format results, and pick status codes. They may not evaluate discount eligibility, calculate discounted totals, or hard-code pricing rates.

A few deliberate design choices keep this check grounded:

structural (import check)semantic (business logic)intenttime →distance from intentallowed bandwasted1234
  1. The furthest the structural line strays between checks.
  2. A check finds the structural line outside the band and sends the work back to intent.
  3. Structural checks can’t see the semantic line; it carries on drifting.
  4. The shaded area above the band is work a correction throws away. The bars total it: the later the catch, the more is wasted.
How far work strays from intent over a long session. A correction sends the work back to intent, so everything done above the allowed band (the shaded area, totalled in the bars) is wasted. The more often a check runs, the earlier it catches drift and the less is thrown away. A check only corrects what it measures: the import check sees structural drift, and it takes a semantic judge to catch the rest.

The diagram above illustrates the economics: the later a correction arrives, the more rework it creates. A deterministic check catches structural violations immediately. A semantic judge catches fuzzier leaks that slip past static linting. When I ran the experiments, the drift didn’t come from the rules I explicitly wrote down; it came from unwritten assumptions and things static checks couldn’t see.

Running Claude Haiku through the Claude Code CLI against this setup (claude_cli_judge.py and the README’s live judge section) confirmed the split. On my recorded test run, the judge correctly returned violation for the discount controller, citing lines 59–66. It returned pass for the baseline and pass for the runtime reach-through fixture, which is right: the rubric only evaluates pricing logic. In a subsequent run over a 12-case calibration set, its verdicts matched the reference labels across all 12 cases.

Where the check runs: hook, pre-commit or CI

Using the exact same sensor and snapshot fingerprint, you can place the check in three different spots. What changes is when the check runs, who receives the finding, and what actually gets blocked.

editturn endscommitpushCImergequeueagent hookagent’s own editsame session✗can't finish✗pre-commitedit outside the agentback to the committernot run✗commit rejected✗CIlocal checks bypassedperson or botunread on the PRbypassedbypassed✗merge blocked✗123
  1. Agent hook: the agent’s own edit is caught as its turn ends, and the finding goes straight back into the same session. The shortest loop, but it only sees work that agent does.
  2. Pre-commit: an edit made outside the agent never meets the hook. The commit is rejected and the failure goes to whoever ran it. It covers every local commit, unless someone skips it.
  3. CI: with the local checks bypassed, the change is pushed and queued before the check fails and blocks the merge. Nothing local can skip it, but the trip back is the longest and needs a person or bot to carry the failure.
The same check in three places, and the same violation arriving three ways. The earlier a check runs, the shorter the trip back, but the less it sees: the agent hook only covers the agent’s own turns, and pre-commit only covers commits that run it. CI sees everything that reaches the pull request, at the price of the slowest loop and a failure that nobody is handed automatically.
Claude Code Stop hookpre-commitCI (GitHub Actions)
TriggerThe agent says it has finished its turngit commitA pull request, or a push to main
Snapshot checkedThe working tree as the agent left itStaged changes with unstaged changes stashed. Untracked files remain presentThe committed tree in a clean checkout
Who gets the findingThe agent, as the reason it cannot finish yetWhoever ran git commit: the agent’s shell tool or a developerThe pull request status list. Nobody, unless a bot or person reads it
What is blockedThe agent finishing, up to a budget of 3 repair attemptsThat specific commitMerging the pull request, provided the check is marked required
Bypasses and failure modesUser interrupts or API failures bypass the hook. A crashed hook lets the agent stop. Exhausting the repair budget ends unresolvedgit commit --no-verify, SKIP=architecture-contracts, or not having the hook installedAdmins can bypass branch rules. A job skipped by an if: condition counts as passing a required check. A workflow skipped by a path or branch filter, or by a skip instruction in the commit message, leaves the check pending and blocks the merge
Operating costRuns on every stop. Fast feedback directly inside the turnRuns on every commit in the local environmentClean setup per run. Slowest feedback, but the only placement outside the local working copy

The agent hook

The Claude Code integration uses a Stop hook. When the agent attempts to conclude its turn, the hook runs the sensor. On a violation, it returns decision: "block" along with the diagnostic output. Claude Code feeds that reason directly back into the conversation, keeping the agent in its execution loop. This is the only placement where a closed repair loop exists out of the box.

That loop needs an explicit circuit breaker. Claude Code stops honouring stop-hook blocks after 8 consecutive turns, but that counter resets whenever the agent uses a tool. Because fixing a violation requires editing files, an agent stuck in a loop could easily cycle indefinitely. My adapter manages its own session budget: 3 repair attempts per user turn. That gives the model enough room to read the diagnostic and try a couple of fixes, while ensuring a stuck agent stops before burning tokens. If the budget runs out, the turn concludes with an explicit UNRESOLVED warning.

Pre-commit

The pre-commit integration runs locally on git commit:

- id: architecture-contracts
  name: architecture contracts (full package)
  entry: .venv/bin/python integrations/check_architecture.py
  language: unsupported
  pass_filenames: false
  always_run: true

Setting pass_filenames: false ensures the sensor checks the entire package rather than just staged files, since architectural boundaries depend on relationships between modules. always_run: true makes sure the check executes even during commits that only delete files.

Pre-commit checks the staged index while stashing unstaged changes. In my recorded runs, a staged violation paired with an unstaged fix was rejected, while a staged valid change paired with an unstaged violation passed. Untracked files stay in the working directory, so an untracked file with a violation will still cause a commit to fail.

CI

The CI integration runs as a GitHub Actions workflow on a clean checkout. It runs on every pull request without path filters, avoiding the GitHub issue where filtered workflows leave required checks hanging indefinitely.

A failing CI job is only a gate if repository rules make it one. The workflow reports a check status; whether that status blocks a merge depends on branch protection settings requiring architecture-contracts.

Furthermore, CI alone is not a repair loop. GitHub does not automatically reprompt an agent when a check fails. To close that loop, an external orchestration script has to fetch the failure output, feed it back to an agent, and manage retry limits. The CI guide includes illustrative pseudocode showing how that orchestration looks in practice.

Who controls the rules

An agent with standard filesystem access can bypass any local check. During my walkthrough, I verified three simple bypasses: using git commit --no-verify, adding a re-export module outside the checked package, and adding an ignore_imports line to .importlinter.

I closed the ignore-imports loophole by having the sensor treat ignored imports on required rules as explicit violations (capture). Local hooks give fast feedback to a cooperative agent. The true security boundary has to live in CI where the agent cannot tamper with the checks, paired with human review of any changes to contracts, tests, or workflow files.

What the testing showed

The mechanics work. Whether they’re worth wiring up depends on how often guidance actually slips, what a slip costs, and what the check costs when it has nothing to catch. I ran the example through a few hundred agent sessions to find out. Each session is a four-step feature in a small codebase where one legacy controller already breaks the rule, so the shortcut is always in view.

Current models keep a rule you state

I started by writing the import rule into CLAUDE.md and comparing guidance alone against guidance plus the gate: first ten single-task features with claude-sonnet-5-5, then 36 four-step sessions across Sonnet and claude-haiku-5-5.

Neither produced a single structural slip. With samples this small, zero means “rare”, not “never”. Throughout this section I give exact one-sided 95% bounds: zero in 10 still allows a slip rate of up to about 26%, and zero in 12 sessions per arm, about 22%. What these runs did measure was the price of a gate that never fires: about 3 to 4 seconds per turn for the hook, and around 6 seconds with the semantic judge.

So I went looking for pressure that would break a written rule. These were exploratory probes without a protocol fixed in advance, and the per-session results are in the repo. With guidance only and no gate, I tried:

Across 58 of those sessions, the agents added the forbidden import zero times. Pooled, that puts the upper bound at about 5%, but it pools very different conditions. Most conditions ran only twice, and zero in two rules out almost nothing (the bound is 78%). Asked for the shortcut outright, Haiku declined and cited CLAUDE.md. Told the application layer was frozen, it stopped and asked for the freeze to be lifted. The compaction summary I inspected listed the chat-only rule as its first item.

It’s a small sample across specific conditions, but the pattern was consistent. It wasn’t the result I expected to find, and it would be dishonest to pretend otherwise.

It also doesn’t take much to get that adherence. I froze a protocol with three levels of guidance (the full rule, a one-line “the code is layered, keep to the layering”, and nothing), with and without the gate, across both models and both scenarios. That’s 72 sessions, 12 per cell. Here is what the run produced:

Guidance in CLAUDE.mdGateSessions with a slipCaught and repairedViolation left in final code
Full ruleNo0/12 (≤22%)n/a0/12 (≤22%)
Full ruleYes0/12 (≤22%)n/a0/12 (≤22%)
One-line hintNo0/12 (≤22%)n/a0/12 (≤22%)
One-line hintYes0/12 (≤22%)n/a0/12 (≤22%)
NothingNo9/12 (47–93%)n/a9/12 (47–93%)
NothingYes10/12 (56–97%)10/10 (≥74%)0/12 (≤22%)

Brackets give exact one-sided 95% bounds on the underlying rate.

One line was enough to stop the import in all 24 sessions that had it, though zero in 24 only rules out a slip rate above about 12%. Precision still paid: with only the hint, the semantic judge flagged business logic in the new controller in 5 of 12 Haiku sessions, typically the controller deciding which orders count toward lifetime value and summing them itself. With the full rule written out, it flagged none in 24 sessions. Those are judge verdicts, not human labels, but the direction is clear. If a rule matters and comes up often, a precise sentence or two in the instruction file is the cheapest prevention you can buy.

You can’t see adherence

Look at the bottom two rows of that table. With nothing written down, Sonnet imported a repository into the new controller in 6 of 6 ungated sessions (a rate of at least 61%), and Haiku did so in 3 of 6 (anywhere from 15% to 85%). The layering was clear in the codebase, but so was a legacy controller that bypassed it—and that was the pattern the agents copied.

Every single one of those sessions passed its functional tests and generated a confident summary. Nothing about the pull request looked wrong. The only way to tell those nine broken sessions apart from the clean ones was the automated check.

That is the detection problem. Guidance influences what an agent tries to do, but it records nothing about what the agent actually produced. The 58 clean pressure probes above are clean because I ran a check over every one of them afterwards. Without that check, I would have had 58 confident summaries and zero proof.

With the gate enabled, the same slips happened just as often, but none survived. All ten were blocked at the end of the turn and fixed after a single repair request, and every session still passed all four acceptance tests. The diagnostic was the first time the agent had been told the rule, and that was enough.

Adherence belongs to the model

Everything so far comes from two current models. Maybe written rules hold across the board, or maybe these two models are just exceptionally good at keeping them. The oldest models my login could still run were a generation back, so I put claude-haiku-4-5 beside claude-haiku-5-5: 96 sessions planned, 92 run before the budget cap stopped the last four.

Sessions that added the forbidden importHaiku 4.5, no gateHaiku 4.5, gateHaiku 5.5, both arms
Rule stated, nothing pushing against it0/6 (≤39%)0/5 (≤45%)0/11 (≤24%)
One-line hint5/6 (42–99%)3/5 (19–92%)0/11 (≤24%)
Rule stated, request asks for a shortcut6/6 (≥61%)6/6 (≥61%)0/12 (≤22%)
Rule stated, a nested file contradicts it6/6 (≥61%)6/6 (≥61%)0/12 (≤22%)
Violation left in the final code17/24 (52–85%)0/22 (≤13%)0/46 (≤6%)

Brackets give exact one-sided 95% bounds. Five or six sessions per cell is enough to show a large difference, not to pin down a rate.

Left alone, the older model followed the written rule. But given only a vague hint, a user asking for a shortcut, or a contradictory instruction file, it broke the rule almost every time. The newer model held firm across the board: no slips in 46 sessions, which puts its rate below about 6% under these conditions. Same prompt, same tasks: whether the written instructions worked depended entirely on which model read them.

The gate didn’t care. It caught all 15 slips in its arm, each was repaired after a single request, and not one gated session left a violation in the repository.

This isn’t just an Anthropic story. I pushed the same tasks through OpenAI’s Codex CLI on two of its models: 40 exploratory sessions with the rule in AGENTS.md and a plain outer loop standing in for the hook. These were exploratory, with two sessions per cell, so treat them as a direction rather than a rate. gpt-6.2-sol never added the import in 20 sessions, even with no rule written down, which still allows a rate of up to about 14%. gpt-5.6-luna kept the written rule when nothing pushed against it, took the shortcut in 4 of 4 sessions when asked, and broke the unwritten rule in 3 of 4. The loop caught the three slips in its half, and each was fixed after one request. Adherence varies by model; the check doesn’t.

Those pressure conditions are conflicts by design: a person or a file asked for the shortcut. Whether an agent should follow a user’s direct instruction is a fair debate. The repository’s rule is what the check enforces, whoever asked.

The models I tested did not agree on what the same instruction file was worth. If the prompt is your only enforcement, every model you evaluate becomes a new adherence question you have to verify by hand. With a check in place, the rule holds whichever model you pick.

Catching it early keeps the fix small

The earlier drift diagram was a conceptual illustration, but the session data backs it up.

Across the 26 ungated sessions that slipped, the agents added forbidden imports across 42 separate turns. Once that first forbidden import landed in the controller, subsequent steps built directly on top of it. With the gate enabled, the 25 sessions that slipped were caught immediately: one slip, one repair request, and the rest of the session stayed inside the boundary.

The repair was cheap because it happened early. The agent had just touched the file, still had the context in memory, and received the exact file, line number, and repair instruction. For Sonnet with no rule written down, adding the gate raised median session time from 86 to 113 seconds, and mean reported cost from about US$0.26 to US$0.34 per session. Every gated session needed exactly one repair, so the difference is a rough estimate of what a repair costs: about US$0.08, or roughly 150,000 input tokens (mostly cached re-reads of the conversation) and 4,000 output tokens. That’s the gap between two groups of six sessions, not a measurement of the repair turn itself.

A note on every dollar figure in this article: they are the cost Claude Code reports for each run (total_cost_usd), meaning Anthropic’s published API list prices at the time of the runs (19–20 November 2026), including prompt-caching rates. My runs used a subscription login, so no per-token bill was involved. Read them as API-equivalent estimates for comparing arms, not as invoices.

I didn’t measure the alternative: finding the same violation during code review, after four dependent endpoints have already been built on top of it. I’d expect that to cost quite a bit more than thirty seconds.

Rules accumulate, and the prompt is paid for on every request

The obvious reaction to unwritten rules is to write more down. Large codebases do exactly that, and their instruction files bloat to match. I compared two prompts over 48 sessions, each with and without the gate: the one-line hint, and a generated 40,000-token engineering handbook with the complete architecture rule sitting 60% of the way down.

Per session, both arms pooledLean promptLarge handbook
Structural slips0/24 (≤12%)0/24 (≤12%)
Sonnet mean input tokens369,0001,326,000
Sonnet mean costUS$0.30US$0.63
Haiku mean costUS$0.02US$0.07
Flagged by the semantic judge5/240/24

Neither prompt produced a structural slip, so the gate had nothing to do in either arm. But the handbook roughly doubled the cost of every Sonnet session and tripled Haiku’s, because instruction files are re-read on every single turn whether their rules are relevant or not.

The handbook did buy something: the judge flagged business logic in the controller in 5 of 24 lean-prompt sessions and none with the handbook. But earlier sessions got that exact same clean result from a single architecture paragraph in a 30-line file, at the lean prompt’s price. The paragraph did the work; the other 39,000 tokens were pure overhead.

That asymmetry is the key economic trade-off. An instruction in your prompt costs a little bit on every session, including all the sessions where it’s completely irrelevant. A deterministic check makes no model calls while it passes. It still adds runtime, about 3 to 4 seconds per turn for the hook here, but model spend only arrives when the rule is broken and you pay for a repair turn. A semantic judge is different: it makes a model call on every check.

Here’s a rough, order-of-magnitude estimate of where the line falls. It rests on three assumptions. First, prompt cost scales linearly with length, so a rule of about 200 tokens costs a 200th of the handbook’s overhead: about US$0.002 per Sonnet session, every session. Second, a repair costs about US$0.08, the estimate above. Third, an unwritten rule gets broken in about three of every four relevant sessions, as it did in my runs. On those assumptions, stating the rule pays for itself if it’s relevant to more than about one session in thirty. If it comes up less often than that, it’s cheaper to leave it out of the prompt and let the check teach the agent when the situation arises.

One rule at US$0.002 is negligible. Two hundred of them is the handbook.

What the gate didn’t fix

On the older model, the repair loop had an unintended side effect. In four of the twelve gated Haiku 4.5 sessions under pressure, the agent fixed the import, passed that step, and then failed an acceptance test on a later step. Without the gate, all twelve passed.

Three of the four made the same mistake. Asked for handle_low_stock(threshold), the agent wrote a handler that took a request dictionary instead—matching the shape existing controllers used—and its own tests called it that way. That mistake never appeared in ungated sessions, so the repair seems to have pushed the agent to copy its neighbors more aggressively than the prompt instructions. With only three sessions, that’s an observation rather than a firm conclusion.

What’s clearer is why nothing caught it. The method signature existed in exactly one place: the prompt text of the request. Nothing in the repository defined what the interface had to look like, and the example ran without a static type checker. An interface checked by mypy or pyright would have failed the turn immediately, just like the import contract did. In Go or Rust, you would likely have written that interface anyway, and the compiler would have been your gate.

The problem wasn’t that gates failed; it’s that we were missing a gate for interfaces.

The semantic judge has limits of a different kind. In the four-step sessions, it flagged a cancelled-order filter in an existing, untouched reports module in 3 of 51 runs—a file it had passed earlier. The agents dutifully refactored legacy code nobody had asked them to touch. A judge’s verdict can shift based on surrounding context, which is why it needs tight scoping before you grant it blocking authority.

How the two fit together

I started this post believing the core argument for gates was that agents forget written rules. Based on this evidence, current models mostly don’t. That doesn’t weaken the case for checks; it clarifies what checks are actually for, and it suggests a practical order of operations:

Start with the instruction. For a rule that comes up frequently, a precise sentence in CLAUDE.md or AGENTS.md is the fastest route to clean code on current models. It takes a minute to write and prevents the slip upfront.

Add the check for what instructions cannot guarantee. Verifying that the rule actually held. Getting the same answer from another model, a different vendor, a colleague’s editor, or a dependency bot. Enforcing a boundary when someone asks for a shortcut. A check is written once and evaluates code identically, whoever or whatever wrote it.

When your instruction file gets heavy, prune rules into gates. Start with rules that are important but rarely relevant. They represent the worst value in a prompt—paid for on every single request and rarely used—and the best value in a gate. Just ensure the automated check is active before you delete the prompt line. An unwritten rule with no check is where bugs slip straight through to production.

Let the feedback loop run both ways. Moving rules from instructions into gates is only half the cycle. The rules that actually slipped in my testing were ones nobody had thought to write down, and the gate’s diagnostic was the first time the agent learned of them. A check that fires repeatedly tells you which rule earns a spot in your prompt instructions. A check that never fires tells you the prompt is doing its job, or that the rule is rare enough to leave to the check.

It’s not prompts versus tests. It’s a continuous loop: state the rule, check the result, and let what the check discovers dictate what belongs in the prompt next.

Start with one rule

You don’t need a sprawling harness to get value out of this pattern:

  1. Pick one structural rule your team keeps pointing out in code reviews. Especially one that isn’t written down anywhere. Write a lint contract or test for it, paired with a diagnostic message explaining how to fix it.
  2. Run it passively first. Run it against your repository history and recent agent branches to verify clean signals without false positives. This gives you your first real measurement of how well your existing instructions work.
  3. Write the rule down too, if it comes up often. A precise sentence or two in your instruction file is the cheapest prevention you can put in place.
  4. Turn it into a fast local feedback loop. Connect the check to your agent’s stop hook or pre-commit configuration so slips are caught while the agent still has full context.
  5. Make it a required check in CI. Branch protection is your true boundary, and the only one that reliably catches changes made outside the agent.
  6. Prune your instruction file based on check findings. Rare rules with reliable gates can leave the prompt. Rules whose gates fire repeatedly belong in the prompt.
  7. Add model-based evaluation only for the subjective remainder. Keep the rubric narrow, require explicit line citations, and treat unparseable output as non-passing.

Ask in the prompt. Know from the check. Let each one tell you what the other is missing.

Published:

Hi, I'm Glenn! 👋

I've spent most of my career working with or at startups. You'll usually find me working in Product leadership roles, on an advisory board, or maybe as an early investor.

I've been the VP of Product & GTM @ Ockam. I led the Terraform product team @ HashiCorp, where we launched Terraform 1.0, Terraform Cloud, and a whole host of amazing capabilities that set the stage for a successful IPO. Prior to that I was part of the Startup Team @ AWS, and earlier still an early employee @ Heroku. I've also invested in a couple of dozen early stage startups.