Small Models Are Prompt Linters: How a Model Eval Turned out as an Agent Contract Refinement Exercise
I ran a five-model comparison to find out which model could drive my planner agent of my Forge agentic pipeline. I got back a code review of my prompt.
My /forge pipeline starts with a planner agent. It interviews me, sends researchers into the codebase, and writes a plan.md that the coder agent implements without re-exploring anything. It had run on a frontier model since I built it, and I wanted to know whether a local model could take that seat. The planner is the most expensive stage, and it runs first, so every token it burns is spent before I know the task was worth doing.
So: a one-shot simple prompt, 4 models adn the pipeline. A frontier reference (Sonnet 5) plus three open-weight models I can host (qwen3.5-122b-a10b, qwen3.6-35b-a3b, qwen3.6-27b), producing plans, scored on contract compliance, grounding, hazard coverage, executability and consistency.
I got the ranking I wanted. I also got something I hadn’t gone looking for: a defect list for my own agent definition.
Every failure I’d filed under “the small model got this wrong” turned out to be a place where my prompt was ambiguous, contradictory or silent. The frontier model had been quietly patching over all of it for weeks. It never told me, because there’s nothing to tell; a model that guesses your intent correctly produces no error message.
A weaker model is a differential test of your agent. Where a frontier model and a small model diverge on process rather than on reasoning depth, the agent is at fault.
The agent contract that was never tested
v0 was nine sections, about 950 words. Against it, the frontier model produced a plan that did three things I never asked for:
- traced every constant it wanted to change back to every file that reads it;
- found an ordering obstacle in the calling code, named three resolutions, and priced each one;
- recorded that its first draft had been rejected, and why.
The plan scored top marks, so I concluded my contract was good. What had actually happened is that the contract was never tested. The model supplied the missing half on every run, for free.
agent v1: writing down what the model did unasked
If a model reliably does something valuable that your agent doesn’t request, promote it from behaviour to requirement. v1 did that for item 1 on the list above, at 1,388 words:
- A
## Reachabilitytable. Every symbol the plan changes gets a row: the symbol, everyfile:linethat reads it, and whether that file is in scope. A row marked NO means the plan is incomplete. - A tooling constraint. The planner has no Grep and no Bash. Every line number comes from a researcher dispatch, quoted in the handoff log. The planner can’t check its own coordinates, so it isn’t allowed to invent them.
[BLOCKING]and[VERIFY]tags, so “the coder can’t proceed” and “the coder should confirm this” stop being one sentence.
That’s the version published in the forge repo. Then I pointed four models at it.
Defect 1: the optional-section trap
The plan format lives in the agent definition as a fenced block. The model reads the block, fills it in, writes the result.
One section didn’t live in the block. ## Exploration findings sat below the fence, under this preamble:
Optional section, only when
## Parallel explorationran — place it after## Context
The frontier model handled it: read the prose, understood the condition, put the section where it belonged. Most of the other runs omitted it entirely, and it was the only part of the format positioned outside the template.
The hazard is structural. The fenced block is what the model copies. Prose around the block is advisory, it competes for attention with everything else in the system prompt, and it degrades first under load. The fix was to move the section inside the fence with its condition inline.
If you want it in the output, put it in the block.
Defect 2: the self-referential check
v1’s failures pushed me to add a pre-handoff self-check. It’s called ## Forbidden in plan.md. It was listing banned phrases, and I meant no string from that list. Read literally and it says the characters: ## Forbidden in plan.md must not appear. Which is true of every plan ever written! The check passed unconditionally.
This never showed up as a failure. It showed up as a check that never fired. The fix was renaming, not rewording: the section became ## Banned in plan.md, and the check became “no banned string appears outside ## Open questions“.
Never name a rule with a phrase that could be quoted as data. If your heading can appear inside the artifact the section governs, the check can’t tell the rule from an instance of it.
Defect 3: the stop condition nobody wrote down
One run produced a good planning document and then kept going, straight into implementation. Writing code, in the planner stage, before the approval gate that exists so I read the plan first.
The stop wasn’t in the planner agent at all. It lived in the orchestrator slash command, and in that run the orchestrator’s instructions had been pushed out of context by the time the agent finished.
The frontier model stopped every time. Not every model will; attention decays over long inputs and gets crowded out by raw data tokens. The answer is a block you can’t read past, in the agent definition itself:
**CRITICAL: STOP AFTER WRITING THE PLAN.** Your job ends when `plan.md` and the handoff log are written. Do NOT: - Begin implementation (write code, edit source files or configs) - Write or run tests - Generate changelogs
Two lessons here. Behaviour implied by an agent’s role is not specified behaviour; it’s a bet on the model’s priors, and the bet gets worse as the model gets smaller. And the constraint belongs to the agent that must honour it, not only to the caller. Orchestrator instructions get diluted over a long run. The agent’s own system prompt doesn’t. Put it in both.
Defect 4: adding words is the reflex, reorganising is the fix
This one is about me, not the models.
After the first bad run against v1, my reaction was to add. v2 gained a role preamble, the banned-phrase section, a wait-for-researchers rule and the self-check. 1,388 to 1,878 words, +35%.
Compliance didn’t improve by 35%. Of three runs against v2, two still failed the gate, and one of those failed on format, which v2 had said nothing new about.
v3 barely appended anything. It reorganised:
## Toolsmoved above## Input modes, so the “no Grep” constraint appears before the workflow that depends on it.## Parallel explorationmerged into## Researcher dispatch; one topic that had been split across two sections.## Escalation conditionsbecame## Interview vs escalate, keyed on what’s missing rather than on severity. Missing a preference means interview; missing an unobtainable fact means escalate; missing a fact a grep answers means dispatch a researcher. That third branch existed in v1 as a throwaway line, and models kept escalating things one grep would have closed. The information was there, the decision structure wasn’t.- The stop rule added.
1,878 to 1,476 words, −21%, while gaining the stop rule and the decision table.
| Version | Words | What changed | Prompted by |
|---|---|---|---|
| v0 | 955 | baseline: nine sections, no reachability, no tagging | |
| v1 | 1,388 | Reachability table, no-Grep rule, [BLOCKING]/[VERIFY] | writing down what the frontier model did unasked |
| v2 | 1,878 | role preamble, banned phrases, wait-for-researchers, self-check | one bad run and a reflex to add |
| v3 | 1,476 | reorganised; stop rule; interview-vs-escalate | four models’ failure modes |
| v4 | 1,662 | ## Hazards table, boundary rule, discard record, one deletion | the two things the frontier model did in v0 that v1 never wrote down |
One of the best run in the whole comparison came from the smallest model in the batch, running against v3.
v4: the other two things the frontier model did unasked
Back to that v0 list. v1 promoted item 1 into ## Reachability. I left items 2 and 3 as behaviour, which is a bet on priors, which is exactly what Defect 3 warns about. I’d written the lesson down and not applied it.
## Hazards, a table inside the fence. Reachability answers what else reads this. It doesn’t answer why is the obvious implementation wrong. The second question is the one the frontier model kept answering unprompted while smaller models walked straight past it. Five columns: obstacle, where it lives, why the naive approach fails, the resolution, what that resolution costs.
One rule matters more than the table. Write none if the researchers found no obstacle, and never leave the section out. An empty section is a claim: I looked, and there was nothing. A missing section is silence, and silence can’t be reviewed.
A boundary rule, because the new section collided with an old one. ## Multiple solutions already said never pick silently, present the options and get a choice. A hazard with three possible resolutions looks exactly like that, and without a boundary the agent stops at every obstacle and asks. So a hazard with one defensible resolution is a row: resolve it, price it, keep going. It only becomes an options block when the resolution is itself a real fork.
I’d have missed that a version ago. Adding a mechanism to a prompt is not additive. Every new section has to be reconciled against the ones that could claim the same input, or what you’ve shipped is an ambiguity rather than a rule.
The discard record, one line in the handoff log: if you drafted an approach and discarded it, say what it was and why it lost. Twelve words for item 3. A rejected approach is the cheapest signal there is about the shape of the search space, and without it every reviewer re-proposes the design that already lost.
What the prompt couldn’t fix
One finding survived every rewrite, and it’s why this article isn’t simply “I improved my prompt”. It needed the harness: a validator pass before the approval gate. A subagent checks the plan against the required sections, flags unresolved reachability rows, and confirms a researcher dispatch was actually recorded. It catches format and completeness failures mechanically instead of relying on me to notice them.
Prompts state intent, gates enforce it. Defects 1 to 4 were agent prompt bugs and the agent prompt fixed them. This one isn’t a prompt bug, and no wording will touch it. The corollary runs the other way too: relaxing a prompt-level constraint is safe once a validator can enforce the same property, and not before.
Takeaways
- Run your agent definitions against a weaker model on purpose. A frontier model repairs sloppy instructions silently and gives you no error for the repair. A smaller model runs what you actually wrote. That divergence is the cheapest lint pass a prompt will get, and you can run it locally.
- The template is the output. Anything outside the fenced block is advisory prose that degrades first under context pressure, and anything duplicated inside it gets duplicated in the artifact.
- Never name a rule with a string that could be quoted as data. A check that can’t tell its own name from an instance of what it checks is a check that never fires.
- Behaviour implied by an agent’s role is not specified behaviour. It’s a bet on the model’s priors, and the bet gets worse as the model gets smaller. Put the stop condition in the agent, not only in the orchestrator/caller.
- Adding words is the reflex, reorganising is the fix. v2 grew 35% and improved little; v3 shrank 21% below it and was stronger. Structure carries more instruction-following weight than emphasis does, and word count is an output rather than a lever.
- Adding a mechanism is not additive. Reconcile every new section against the existing ones that could claim the same input, or you’ve shipped an ambiguity.
The pipeline is at github@PazsitZ/forge. The published planner is the v1 lineage above: the contract, before four models finished reviewing it.