Do "auto-document my repo" tools actually capture the codebase? A local bake-off

I tested whether "auto-document my repo" tools actually capture the codebase

The real question isn't whether you can chat with a repo. It's whether the documentation a tool generates actually contains the answers. So I built a question set, generated docs with each tool — locally, one model — and checked: can you answer these questions from the output alone?

[!TIP] TL;DR — The pipeline under test is repo → docs → answers. On known/small repos (a Spring sample, a small Python lib) the best generators produce complete docs — every question answerable. On a real, novel 750-file repo, the best tool still documents only ~⅓ of the entry-point surface — it nails the main flow and skips the operational long tail. Two tools are worth running: DeepWiki-Open (richest, most-cited docs) and deepwiki-rs / Litho (most robust on a weak model). Everything else either produced no docs at all on a local model, or is closed/paywalled.

:::stats

Tools that produced gradeable docs

2 of 13

DeepWiki-Open · Litho

One brain for all

Qwen3-30B

local, on an M4 Pro — no cloud

Known-repo coverage

100%

every question answerable from docs

Hard-repo input coverage

≈⅓

10 of mira's 29 entry points

Questions per repo

6

architecture · flow · inputs · outputs · e2e

Closed / never-produced docs

11

paywalled, or stalled on local :::

The actual goal: repo → docs → answers

I'm not testing a live chatbot. I'm testing documentation generators — tools that read a repository and emit a wiki / architecture doc. The value chain is:

graph LR
  R[(Repo)] -->|tool ingests| D[Generated docs]
  D -->|reader / model| A[Answers]
  style R fill:#f4f4f5,stroke:#d4d4d8,color:#18181b
  style D fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
  style A fill:#dcfce7,stroke:#22c55e,color:#14532d

The question set exists to interrogate the middle box. For each tool's generated docs, the test is simple: given only this output, is each question answerable? Answering straight from the source (agentic file-reading, raw RAG over code) is a different category — it measures a Q&A system, not whether the docs captured the knowledge. So the real contenders here are the doc generators.

The lab: one local model (Qwen3-30B + nomic embeddings) for every tool; three repos — tinydb (small Python lib), spring-petclinic (known Java framework), mira (my own ~750-file Go + Astro app, the hard one). The one rule: setup difficulty never counts against a tool — I pushed each through every install hurdle until it ran properly.

Why most tools never even produced docs

Of 13 tools, only two generated complete documentation on a local model. Understanding why is the most useful finding — and it comes down to method.

[!IMPORTANT] The tell: CodeWiki detected 132 components and 15 modules, then wrote zero files — its writer is a SWE-agent str-replace editor the model wouldn't drive. OpenDeepWiki reached its "catalog" step and produced nothing. Meanwhile Litho, asked to do the same job on the same model, produced 3,800 lines of clean C4 docs — because it just asks for prose and validates the Mermaid afterward. ==On local hardware, the elegant machinery is the liability.== With a hosted frontier model the agentic tools would likely shine; locally they're the weak link.

The real test: can you answer the questions from the docs?

This is the measurement that matters — checked objectively against a hand-built ground-truth key (every route, flag, env var, output the repo actually has).

On the known and small repos, the docs are complete — every question is answerable straight from the output. That's the generators doing exactly their job.

[!WARNING] The mira gap. DeepWiki's wiki documents the render → store → serve heart of mira beautifully — but ask its docs "what are all the entry points?" and you get only 10 of 29. It covers /v1/render, /og, /p, /r, /mcp, /admin, /spec, /api, /v1/spec.md. It misses /v1/renders, /v1/feedback, /v1/assets, /asset, the agent-discovery routes (llms.txt, .well-known/*, ai-plugin.json), /docs, robots.txt, sitemap.xml, all /static/*, every startup flag, and most env vars. On a real, novel codebase the generated docs answer the story but are only ~⅓ complete on the exhaustive surface. That's the honest ceiling of repo → docs → answers today.

The verdict: DeepWiki-Open vs Litho

:::tabs

🥇 DeepWiki-Open — the default pick

Richest, most answerable docs. petclinic: 36 pages, 54 diagrams, 197 inline file:line citations — you can trace any claim. Matched or beat Litho on every measured repo (tinydb 6/6 vs 4/6; petclinic a tie). Browsable wiki format. The cost: it needed context-window tuning + an embedding rebuild to handle the 750-file repo at all.

🛡️ Litho (deepwiki-rs) — the robust pick

It just works — completed cleanly on the local model where DeepWiki needed babysitting and two other generators failed outright. Output is a tight C4 architecture model (Context → Container → Component → Code): a better mental model, if less exhaustive page count. The pick if your model is small or you want zero fuss. :::

[!NOTE] Today I'd ship DeepWiki-Open as the default — denser, more cited, more answerable. Litho wins on robustness. The one open question is the mira head-to-head: does Litho's methodical per-directory C4 pass capture more of the hard repo's 29-route surface than DeepWiki's 10? That run is finishing now, and it's the single piece of evidence that could flip the call.

The wider landscape

For completeness — the other nine tools, and why they're not in the running for this goal:

[!CAUTION] Honesty box — Every product was fully stood up before grading; failures are capability or commercial, not "couldn't install." Coverage numbers are an objective text-match of each tool's generated docs against a ground-truth key (routes, flags, env vars, outputs). "Free" means free: a paywalled or cloud-only layer is that product's ceiling. One model, one box, three repos — a frontier model would reshuffle the agentic tier substantially. The deciding mira comparison (Litho) is still generating; this verdict is on the evidence in hand.

The test was never "can it chat." It was "does the documentation it leaves behind actually capture the codebase." On known code, yes. On real, novel code — only partly, even at its best.