L2 — Agent harness shootout

Same model (DWQ-4bit), same 10 polyglot Python tasks, test-blind (the hidden test is withheld until grading), identical prompt + caps, tokens tallied by a measurement proxy. ← [[Benchmarking a whole local-LLM stack on an M4 Pro — and auditing my own mistakes|back to overview]]

harness	pass	median tokens/task	median s/task	notes
goose	0/10	15.5k	16	leanest + fastest
aider	0/10	16.5k	48	lean, one-shot
crush	1/10	250k	199	agentic, spirals
claude-code	2/10	260k	900	most passes, via iteration; never self-terminates
opencode	0/10	696k	900	spirals to 1.5–1.7M tok; flaky headless

type: bar
title: Median tokens per task (log-scale spread ≈ 45×)
x: [goose, aider, crush, claude-code, opencode]
series:
  - name: tokens/task
    data: [15500, 16500, 250500, 259934, 696064]

[!NOTE] Absolute pass rates are low (0–2/10) by design — these are the harder polyglot tasks driven by a local 30B-4bit, consistent with public data that small local models score poorly on polyglot. Not a harness defect; it's the model ceiling. The signal here is efficiency, and the spread is enormous.

What it means

Agentic iteration buys a little, at huge cost. claude-code's loop got 2/10 and crush 1/10 where the one-shot/light harnesses got 0 — but at ~16× the tokens and ~20× the wall-clock. claude-code never self-terminates on a local model (runs to the 900 s cap every task).
Recommendation: aider or goose for token-efficient daily local use. Reach for claude-code/crush only when you need the extra completions and can pay the 15–45× cost.
Reliability: claude-code & opencode hit the 900 s cap on most tasks; opencode also intermittently hangs with 0 requests headless. (opencode works once configured with tool_call:true, but is the heaviest + flakiest for batch use.)
claude-code reaches the local model through a claude-code-router Anthropic→OpenAI shim; its token-heaviness is its own multi-turn context growth (~11 turns/task), not the shim.