Small Models Are Prompt Linters: How a Model Eval Turned out as an Agent Contract Refinement Exercise
I ran a five-model comparison to find out which model could drive my planner agent of my Forge agentic pipeline. I got back a code review of my prompt.
My /forge pipeline starts with a planner agent. It interviews me, sends researchers into the codebase, and writes a plan.md that the coder agent implements without re-exploring anything. It had run on a frontier model since I built it, and I wanted to know whether a local model could take that seat. The planner is the most expensive stage, and it runs first, so every token it burns is spent before I know the task was worth doing.
So: a one-shot simple prompt, 4 models adn the pipeline. A frontier reference (Sonnet 5) plus three open-weight models I can host (qwen3.5-122b-a10b, qwen3.6-35b-a3b, qwen3.6-27b), producing plans, scored on contract compliance, grounding, hazard coverage, executability and consistency.
I got the ranking I wanted. I also got something I hadn’t gone looking for: a defect list for my own agent definition.
Continue reading How a Model Eval Turned out Agent Contract Refinement