L3, breadth & the contenders that hit walls
← [[Benchmarking a whole local-LLM stack on an M4 Pro — and auditing my own mistakes|back to overview]]
L3 — Embedding / RAG
qwen3-embedding:0.6b (the live embed model): 1024-dim, 53.4 embeddings/sec (batched), recall@1 = recall@3 = 1.00 on a 20-doc / 12-query labelled set (topic-clustered: networking, biology, finance, algorithms, history, geography). Perfect retrieval here — the 0.6 B embedder is more than adequate for small/medium local RAG; you'd need a larger, noisier corpus to find its ceiling.
Contenders that hit real walls
The contract was: never silently drop a contender — make it work, or prove the wall with evidence.
| contender | outcome |
|---|---|
| MLC-LLM (engine) | Wall. import tvm crashes on this macOS/arm64 box (tvm::ffi::Error); nightly wheels are version-mismatched, and even past that MLC needs per-model compilation and doesn't support the Qwen3-Next-MoE architecture. |
| ollama-MLX backend | n/a. ollama 0.23 has no flag to select an MLX backend for an arbitrary GGUF; it only applies to specific registry models. The ollama row is its real serving path here. |
| SWE-bench | Infra works, full run impractical. Probed one Verified instance with the gold patch through the Docker harness on arm64 → resolved, zero errors. Not a wall. But generating patches needs an agentic harness driving the local 30B over 500 instances (~900 s/task, ~0 expected) — a scope call, not a capability limit. |
| cline / continue / roo | GUI/IDE-extension harnesses; not driven in this headless batch (out of scope, documented). |
Speculative decoding (engine add-on)
Net-negative for the A3B MoE — see the detail in [[L0 — Engine bake-off]]: a Qwen3-0.6B draft slowed DWQ-30B to 0.64× because the MoE's ~3 B active params make base decode already fast, so draft+verify overhead dominates.