L1 — Model × quant
Fixed engine (MLX), 8 models incl. a 4/6/8-bit + DWQ sweep of Qwen3-Coder-30B-A3B. Coding re-run at concurrency 1 with saved generations — all scores independent and distinct (earlier cross-architecture ties were a generation-timeout artifact). Anchor: qwen2.5-coder-32b = 0.902 ≈ its published ~0.90. ← [[Benchmarking a whole local-LLM stack on an M4 Pro — and auditing my own mistakes|back to overview]]
Coding (EvalPlus pass@1)
| model | decode t/s | wired GB | HumanEval+ b/p | MBPP+ b/p |
|---|---|---|---|---|
| coder-next-mxfp4 (80B-A3B) | 67 | 45–51 | 0.909 / 0.872 | 0.913 / 0.783 |
| 30b-a3b-4bit | 89 | 20–25 | 0.860 / 0.811 | 0.847 / 0.714 |
| 30b-a3b-4bit-DWQ | 90 | 20–25 | 0.933 / 0.902 | 0.905 / 0.770 |
| 30b-a3b-6bit | 69 | 28–33 | 0.927 / 0.896 | 0.907 / 0.786 |
| 30b-a3b-8bit | 59 | 35–40 | 0.927 / 0.902 | 0.910 / 0.783 |
| gpt-oss-20b | 74 | 15–17 | 0.890 / 0.884 | 0.913 / 0.775 |
| devstral-2507-8bit (dense 24B) | 13 | 23–31 | 0.823 / 0.793 | 0.812 / 0.709 |
| qwen2.5-coder-32b-8bit (dense 32B) | 7 | 38–51 | 0.902 / 0.866 | 0.881 / 0.767 |
Non-coding (tool-calling / instruction-following / long-context)
Served by the engine that parses the model's tool format: mlx for the Qwen3-Coder family, llama.cpp --jinja for the rest.
| model | tool (bfcl) | ifeval | longctx 32k | engine |
|---|---|---|---|---|
| coder-next | 1.00 | 0.92 | check | mlx |
| 30b-a3b-4bit | 0.91 | 0.83 | check | mlx |
| 30b-a3b-4bit-DWQ | 1.00 | 0.75 | check | mlx |
| 30b-a3b-6bit | 1.00 | 0.75 | check | mlx |
| 30b-a3b-8bit | 1.00 | 0.75 | check | mlx |
| gpt-oss-20b | 0.64 | 0.75 | cross | llama.cpp |
| devstral-2507 | 0.64 | 0.75 | check | llama.cpp |
| qwen2.5-coder-32b | n/a | 0.92 | check | llama.cpp |
Conclusions
[!TIP] Winner:
30b-a3b-4bit-DWQ. Best HumanEval+ (0.933/0.902 — the single best score, beating the 80B flagship), reliable tools, full 32k retention, 90 t/s, 20 GB. DWQ is decisively worth it: plain 4-bit drops to 0.860; DWQ recovers (and slightly exceeds) the 6/8-bit quants at the same 4-bit speed and memory.
- Quant saturates by 4-bit-DWQ — 6/8-bit cost 1.3–1.8× memory and halve speed (90→59 t/s) for no quality gain.
- The 80B flagship isn't worth it here — Coder-Next scores below DWQ-4bit while using 2× the memory at 67 t/s.
- MoE crushes dense on speed — A3B 59–90 t/s vs dense 7–13 t/s. qwen2.5-32b codes well but 7 t/s rules it out for interactive use.
[!CAUTION] The gpt-oss / devstral / qwen2.5 tool & long-context cells are serving-path results, not model verdicts — each verified from raw responses: parallel tool-calls fail via llama.cpp (a single-call-per-response limit), qwen2.5 emits no tool_calls via either engine (template/parse failure), and gpt-oss returns empty/leaked content on long-context. For agentic tool use locally, the Qwen3-Coder family on mlx is the only reliable path.