Production RAG you can trust
A production-grade RAG system returns grounded, citable answers from your own documents — and refuses to guess when the sources aren't there. The difference between that and a demo is not the model: it's retrieval quality, grounding enforcement, and an evaluation harness that measures accuracy against your real questions. That evaluation-first approach is how NorthSight Technologies builds RAG for enterprises and public-sector organizations in Canada.
- RAG lets an LLM answer from your knowledge base, with citations back to source documents.
- Most RAG failures are quiet retrieval failures — the model papers over the gap with a confident guess.
- The fix is engineering, not prompting: better retrieval, grounding enforcement, refusal behaviour, and a golden-task eval set built from real user questions.
- With evals in place, every change is scored before it ships, and drift is caught before users see it.
What RAG is — and what it's for
Retrieval-augmented generation (RAG) is an architecture where the model answers using passages retrieved from your document store — policies, contracts, manuals, tickets, records — instead of relying on what it memorized during training. It's the standard pattern when answers must be current, private to your organization, and attributable to a source. For government and regulated industries, that last property is the whole point: an answer you cannot trace to a document is an answer you cannot stand behind.
Why RAG systems hallucinate
In our audits, RAG hallucination is almost never a "model problem." The typical chain looks like this:
- The right passage exists but wasn't retrieved — poor chunking, weak embeddings for the domain's vocabulary, or a question phrased differently than the document.
- Retrieval returned something, so the pipeline proceeded — no one checked whether the passages actually contain the answer.
- The model, given thin context and asked to be helpful, produced a fluent, plausible, wrong answer.
Each link is fixable, but only if you can see it happening — which is why measurement comes first.
The reliability layer we build
- Retrieval engineered for your corpus. Chunking that follows document structure, hybrid search (semantic + keyword), and reranking — tuned against measured retrieval hit rates, not defaults.
- Grounding enforcement. The answer may only claim what retrieved sources support, with inline citations users can click and verify.
- Honest refusal. When sources are insufficient, the system says so and routes to a human — in production, "I don't know" is a feature.
- Golden-task evaluation set. Real user questions with verified answers and sources, scored automatically on retrieval hit rate, answer correctness, groundedness and citation accuracy. Every change runs against it; regressions are caught before deployment.
- Drift detection. As documents and usage evolve, scheduled eval runs catch quality decay early — before users lose trust in the system.
Case study
A genericized example from our work: a knowledge-base assistant where we treated quality as an evaluation problem from day one — grounding answers in retrieval, then building a golden-task eval set from real user questions with scoring rubrics and regression checks. Result: measured accuracy instead of anecdotes, drift caught early, and answers that are grounded rather than guessed. We're happy to walk through the real architecture and trade-offs on a call.
Frequently asked questions
What is RAG (retrieval-augmented generation)?
An architecture where the model answers questions using documents retrieved from your own knowledge base, with citations back to sources — instead of relying only on training memory. It's the standard approach for private, current, attributable answers.
Why does our RAG prototype give wrong answers so confidently?
Almost always: retrieval failed quietly and the model filled the gap. Without grounding enforcement and refusal behaviour, an LLM will produce its best guess in fluent prose. The fix is retrieval engineering plus a rule that answers may only claim what sources support.
How do you measure RAG quality?
A golden-task eval set: real questions, verified answers, verified sources — scored on retrieval hit rate, correctness, groundedness and citation accuracy. It runs on every change and on a schedule, so quality is a number you can audit and trend.
Can RAG run under Canadian data residency requirements?
Yes — embeddings, vector store and inference can all run in Canadian cloud regions or on self-hosted models. See AI for government in Canada for how we handle residency and Protected-B.
We already built a RAG system. Can you fix it rather than rebuild?
Usually, yes. We start with an audit — tracing real failures to root cause — then add the reliability layer to what you have. Rebuilds are the exception, not the default recommendation.