Benchmarking a whole local-LLM stack on an M4 Pro — and auditing my own mistakes

[!NOTE] Tooling, raw results & reproducible methods: github.com/zcag/local-llm-bench. This is the narrative + findings hub; detailed per-layer pages are nested below.

I run models locally on a Mac mini (M4 Pro, 64 GB). Most "local LLM benchmarks" measure one model on one engine and quote a tokens/sec number with no config and no checks. I wanted the answer to a bigger question: what's the best whole stack — inference engine × model/quant × agent harness × embeddings — and how do I know the numbers are real?

So I built a layered benchmark, ran it end-to-end, and — the part most benchmarks skip — had it independently audited, caught several of my own headline findings being wrong, and fixed them in the open.

The recommended stack (this box, single user)

:::stats

Engine

mlx_lm.server

fastest single-user; llama.cpp for multi-user

Model

Qwen3-Coder-30B-A3B 4bit-DWQ

HumanEval+ 0.93 · 90 t/s · 20 GB

Agent harness

aider / goose

lean ~16k tokens/task

Embeddings

qwen3-embedding 0.6b

recall@1 = 1.0 :::

👉 [[Choosing your stack — pros, cons & a decision guide]] — pick the right component for your situation (pros/cons + scenario-based picks).

How it's structured — four isolatable layers

graph LR
  L0["L0 · Engine<br/>mlx · llama.cpp · ollama · LM Studio"] --> L1["L1 · Model × quant<br/>8 models · 4/6/8-bit + DWQ"]
  L1 --> L2["L2 · Agent harness<br/>aider · goose · crush · claude-code · opencode"]
  L2 --> L3["L3 · Embed / RAG"]

Each layer varies one thing and holds the rest at its best, verified configuration. Pages:

Seven things that surprised me

  1. A distilled 4-bit model matches 8-bit quality at 4-bit cost — and beats the 80B flagship (Qwen3-Coder-Next) on HumanEval+. Quant saturates by 4-bit-DWQ; more bits just halve your speed for nothing.
  2. MLX batches, but gains no aggregate throughput from it (flat ~80 t/s under load) while llama.cpp scales to 134. My first pass wrongly concluded "MLX can't batch" — I'd run every engine with concurrency disabled. A config bug.
  3. Tool-calling outside the Qwen3-Coder family is a serving-format minefield — gpt-oss "harmony" channels and Mistral [TOOL_CALLS] aren't parsed by mlx_lm.
  4. Dense models are 5-13× slower than MoE (qwen2.5-coder-32B at 7 tok/s vs the 30B-A3B's 90).
  5. Agent harnesses span ~45× in token cost for identical work (goose ~16k vs opencode up to 700k per task).
  6. Speculative decoding is net-negative for A3B MoE (0.64× — slower).
  7. MLC-LLM doesn't run on this box at all (its runtime crashes on import); SWE-bench's eval infra does, but a full local-model run is impractical.

The part I actually care about: not trusting my own numbers

[!WARNING] Left to a single pass, this benchmark would have shipped with fake TTFT numbers (prefix-cache hits — 0.14 s reported where the real value was 25 s), un-batched concurrency (the false "MLX can't batch"), and two different models scoring identically (a generation-timeout artifact). All plausible. All wrong.

What caught them: an independent adversarial audit plus a hard rule — no result counts until its config is proven to have taken effect (from logs) and its number passes an anchor check (does it match physics, or a published value?). The traps and fixes are in [[Methodology & the self-audit]] and the repo's FIXES.md.

If you benchmark a local stack, assume you have some of these bugs right now. "It ran and produced a number" is not the same as "the number is true."

Take it and run it

The harness is small, dependency-light Python, built to be reused — engine adapters, a measurement proxy for uniform token accounting, transparent eval suites (tool-calling, instruction-following, long-context, EvalPlus, embed/RAG), and agent-harness adapters. Everything, with the raw runs.jsonl and reproduction steps, is at github.com/zcag/local-llm-bench.