Skip to content
SQ

RAG18 September 20266 min

Grounding before cleverness

Most LLM features fail at retrieval, not at reasoning. What that changes about how you build them.

When a retrieval-augmented feature gives a wrong answer, the instinct is to reach for a bigger model or a better prompt. In the systems I have built — a store assistant inside Fidras, document grounding for the Mariam voice agent, RFQ understanding in Chandler OS — that instinct has been wrong almost every time. The model was reasoning correctly over the wrong context.

The failure is upstream

A retrieval pipeline has several places to lose the answer before generation starts: the document was never ingested, it was chunked across the boundary that mattered, the embedding did not capture the query's vocabulary, or the top-k cut it off. All four look identical from the outside — a confident, wrong paragraph.

  • Log the retrieved context with every answer. If you cannot see what the model read, you are debugging blind.
  • Chunk on document structure, not character count. A policy clause split in half retrieves as noise.
  • Evaluate retrieval separately from generation. Recall@k on real questions tells you more than any end-to-end vibe check.
  • Make citation mandatory in the output schema. An answer that cannot point at a source is a guess with good grammar.

Why this matters more in voice

In a text chat, a wrong answer is embarrassing. On a phone call, it is a commitment — the caller heard a business make a statement. That is why the voice agent's retrieval is scoped tightly to the client's own documents, and why anything outside that scope escalates to a human rather than improvising. The architectural decision that most improved answer quality was not a model upgrade; it was narrowing what the agent was allowed to answer.

The cost argument

Grounding also happens to be the cheap path. Retrieval over a small, well-structured corpus with a mid-tier model beats a frontier model reasoning from a vague prompt, at a fraction of the token spend. In production that difference compounds every call, every chat, every day — and an AI feature that quietly burns tokens is a liability, not a feature.