How to build "ask your docs": a RAG pipeline, end to end

A minimal "ask your docs" demo is easy: chunk pages, embed them, retrieve the nearest few, put them in a prompt. The version that holds up on a real corpus needs more, because the common failure modes are all in the pipeline, not the model:

This is the full pipeline behind Ask in tela, an open-source team wiki, documented step by step. It has three stages: preparing documents (index-time), retrieving for a question (query-time), and grounding the generation. The whole thing on one diagram:

flowchart TD
    subgraph PREP["Stage 1 · Prepare documents (index-time, on every save)"]
        direction TB
        SRC["Page body · attachment · sheet"]
        NORM["Normalize to text<br/>sheet to prose · extract PDF/text · strip drawings"]
        CHUNK["Chunk on structure<br/>heading-aware, ~1500-1700 chars"]
        CTX["Contextualize<br/>prepend page title + heading breadcrumb"]
        EMB["Embed<br/>cache by hash(model + text) · stamp model"]
        STORE[("Store in pgvector<br/>beside content · no ACL on the chunk")]
        SRC --> NORM --> CHUNK --> CTX --> EMB --> STORE
        HEAL["Self-heal loop<br/>debounce · retry-backoff · stale sweep"]
        HEAL -.re-index.-> NORM
    end

    Q(["User question"])

    subgraph RET["Stage 2 · Retrieve (query-time)"]
        direction TB
        KW["Keyword search<br/>Postgres FTS · OR-recall"]
        VEC["Vector search<br/>embed query to cosine distance"]
        RRF["Fuse · Reciprocal Rank Fusion (k=60)"]
        RERANK["Rerank — optional<br/>cross-encoder on top ~50 · best-effort, timed out"]
        EXPAND["Expand to parent page<br/>dedup to pages · full body of hubs · budgeted"]
        KW --> RRF
        VEC --> RRF
        RRF --> RERANK --> EXPAND
    end

    subgraph GEN["Stage 3 · Ground and generate"]
        direction TB
        GROUND["Build the LLM input<br/>numbered cited excerpts [n]<br/>+ Space › path location labels<br/>+ known cross-doc conflicts<br/>+ answer-only / be-exhaustive brief"]
        LLM(["LLM · streamed, detached job"])
        OUT(["Answer + cited sources<br/>+ low-confidence flag + follow-ups"])
        GROUND --> LLM --> OUT
    end

    STORE -. access-scoped join .-> KW
    STORE -. access-scoped join .-> VEC
    Q --> KW
    Q --> VEC
    EXPAND --> GROUND

The rest of this post walks each stage and says what every step is for.

Stage 1: prepare documents

Runs in the background when a document is saved.

Step 0 — normalize to text

Not all content is prose. Embedding a spreadsheet's raw markup buries the data under syntax; embedding only page bodies misses attachments. Normalize each source to plain text first:

Step 1 — chunk on structure

Whole pages are too long to embed well; embeddings blur as text grows, and retrieval should return the relevant section. Split on the document's own headings rather than fixed-size windows: each chunk is a coherent section (~1500–1700 characters) that never cuts mid-idea.

Step 2 — contextualize each chunk

A section ripped out of its page loses meaning: a chunk titled "Limits" under a "Rate limiting" page matches any query about limits and answers none. Before embedding, prepend each chunk with its page title and heading breadcrumb (Rate limiting › Limits › <section>). The embedded text is self-contained; the original text is kept for display. This is a lightweight form of contextual retrieval.

Step 3 — embed and cache

Run each contextualized chunk through an embedding model (tela uses a 1024-dim model). Two practices matter from the start:

Step 4 — store beside the source; authorize through the source

Store vectors in pgvector. A dedicated vector database is unnecessary when the app already runs Postgres: vectors sit next to content and permissions, and the lexical and vector indexes share one query engine.

Treat the chunk table as a disposable, derived cache carrying no permissions of its own. At query time, authorize through the live source row in the same SQL: join chunks → pages → access. Never copy an ACL onto the chunk. A chunk that can only be seen by joining to its live parent can never out-live or out-scope that parent — so moving a page to a private space cannot leak its old chunks.

Step 5 — keep it fresh automatically

Three mechanisms keep the index from drifting:

Stage 2: retrieve

Step 1 — hybrid retrieval

Vector search matches by meaning ("how do I publish a space" finds the right page without the word "publish") but misses exact terms — error strings, identifiers, product names. Keyword search (Postgres full-text) matches those and misses paraphrase. They fail in opposite directions, so run both.

On the keyword side, use OR-recall rather than AND: a conversational question carries filler words a relevant document lacks, and requiring all terms zeroes out good documents. Match any term for candidacy; ranking sorts the rest.

Step 2 — fuse with RRF

Combine the two ranked lists with Reciprocal Rank Fusion: each list contributes 1 / (k + rank) (k = 60), so chunks ranked highly by both rise to the top. RRF needs no score calibration between the incomparable cosine and text-rank scales.

Step 3 — rerank (optional)

RRF gives good recall; the top few results may still be similar-looking rather than answer-bearing, because neither keyword overlap nor embedding distance reads the query and passage together. A cross-encoder reranker re-scores the top ~50 candidates by jointly reading query and passage — the largest precision lever above hybrid+RRF. Make it best-effort (fall back to the fused order on failure) and time-bounded (a slow reranker must not stall the request).

Step 4 — expand to the parent document

Chunk retrieval finds the right neighbourhood, but some answers span a whole page (a "services using X" registry, a split table) and cannot be reconstructed from one fragment. After retrieval, dedup chunks back to their source pages and expand the pages that matter — the top few by rank, plus any page the query drew many chunks from (a dense hub, often an overview page) — to their full body before grounding. Bound it with a per-page and a total character budget; the long tail degrades to chunk text.

Stage 3: ground and generate

Step 1 — build the LLM input

Assemble the retrieved material into a numbered, cited context block with a strict brief:

Step 2 — production concerns

Measure retrieval and generation separately

There are two distinct things to measure, and they fail independently:

What to skip until measured

At wiki scale (tens of thousands of chunks) an exact vector scan is sub-millisecond with perfect recall, so an ANN index (HNSW) only trades recall for unneeded speed. GraphRAG and multi-step agentic retrieval add fragility at this size. Add machinery when the eval set shows the simpler approach is the bottleneck — not before.

Summary

  1. Normalize everything to text (sheets, PDFs, not just prose).
  2. Chunk on structure, then contextualize each chunk before embedding.
  3. Cache embeddings by content+model hash; stamp the model.
  4. Store beside the content; authorize through the live source row.
  5. Self-heal the index (debounce · retry · stale sweep).
  6. Retrieve hybrid, fuse with RRF, rerank for precision.
  7. Expand chunks to their parent page for whole-page answers.
  8. Ground strictly — cite, label locations, surface conflicts, don't invent.
  9. Stream and detach generation; rate-limit; flag low confidence.
  10. Measure retrieval and generation separately.

The whole pipeline is open source in tela — a self-hostable, markdown-native team wiki with this Ask pipeline built in (pgvector, hybrid + RRF + optional cross-encoder rerank). Code: github.com/zcag/tela. Try Ask on the free cloud at telawiki.com.