Methodology & the self-audit

The most reusable part of this project. ← [[Benchmarking a whole local-LLM stack on an M4 Pro — and auditing my own mistakes|back to overview]]

:::quote "It ran and produced a number" is not the same as "the number is true." :::

The gate

No result counts as trusted until both hold:

1. Config verified — the engine/model/harness is at its best config and that config is proven to have taken effect, from log evidence (full GPU offload, the intended batching/flags actually active), not assumed.
2. Anchor-checked — the number is plausible against an external anchor: decode tok/s vs the box's 273 GB/s memory-bandwidth ceiling; coding pass@1 vs a published value; TTFT vs prompt÷prefill-rate.

A result that fails either is marked suspect, not published.

The bugs this caught — in my own first-pass results

Every one of these was present in the first run and would have shipped as a confident finding. They were found by an independent adversarial audit (a fresh pass whose only job was to break the findings).

Reported 0.14 s time-to-first-token for an 18,669-token prompt — while the same row's prefill rate (749 t/s) implies ~25 s. Three identical prefill reps were fired; reps 2–3 hit the KV prefix cache → near-zero TTFT → the median was the cached lie. Fix: unique prefix per request so the cache can't hit.

A headline conclusion — and false. Every engine had been run with concurrency disabled by default (mlx --decode-concurrency 1, ollama NUM_PARALLEL 1, llama.cpp auto --parallel 4). MLX batches fine; it just doesn't gain aggregate throughput from it. Catch: if "it doesn't scale" — did you enable scaling?

A Mistral 24B and a Qwen 30B both scored exactly 0.823/0.793 on HumanEval+. Generating at concurrency 4 against a slow model, requests queued past the client timeout → empty completions → depressed, coincidentally-equal scores. Catch: identical floats across different architectures is a red flag. Fix: serial generation + a known-good model must hit its published number.

L2 copied the exercise's hidden test file into the working directory — agentic harnesses could see the assertions; one-shot ones couldn't. Apples-to-oranges. Fix: withhold the test until grading; every harness solves blind.

• MLX served 4-bit-DWQ (~3.5 bpw) vs GGUF Q4_K_M (~4.5 bpw) — fixed by matching bit-budget.
• EvalPlus's sandbox setrlimits fail on macOS → every solution scored 0; fixed by grading in a native arm64-Linux container (and it caches results — a stale file masks retries).
• L2 harnesses had different output caps + extra per-harness prompt scaffolding; fixed to identical context, caps, and instruction text.

Why it matters

None of these were model facts — they were configuration and methodology errors, each producing a plausible, publishable, wrong number. The only defense that worked: hold each contender at a config you can prove, anchor-check every number, and get independent eyes to try to break your findings.

Full per-issue tracker with evidence: FIXES.md · docs/pitfalls.md.