Building AI Search for Your Docs: A 2026 Reference Architecture
Hybrid retrieval, reranking, freshness, and the four evals that separate a good docs bot from a legal liability.
"Ask our docs" bots are the most common AI feature we ship. They are also the easiest to get wrong — one bad answer to a legal or pricing question and trust is gone.
The reference stack
- Ingest: chunk on semantic boundaries, not fixed tokens. Keep headings as metadata.
- Store: Postgres + pgvector for <5M chunks. Dedicated vector DB above that.
- Retrieve: hybrid (BM25 + dense) with reciprocal rank fusion.
- Rerank: cross-encoder on the top 30 → keep top 5.
- Answer: instruct the model to cite by chunk id and refuse if score is low.
The four evals that matter
- Answerability: does the answer live in the retrieved context?
- Faithfulness: does the answer only use retrieved context?
- Citation accuracy: does every cited chunk actually support the claim?
- Refusal rate: does the bot say "I do not know" when it should?
Freshness is a feature
Stale docs are the number one source of wrong answers. Reindex on doc merge, not on cron. Attach a "last verified" timestamp to every chunk and show it in the UI.
Where teams overspend
Fancy chunking strategies, custom embeddings, and giant context windows rarely move the needle. Good retrieval and a real reranker do.
FAQ
Frequently asked questions
Do I need a vector database?+
Not under 5M chunks. Postgres with pgvector is faster to ship and cheaper to run.
Should I fine-tune?+
Almost never for docs search. Improve retrieval and prompts first.
Building something similar?
Let's talk in 30 minutes.