RAG development: retrieval systems built for correctness
Most retrieval-augmented generation systems that disappoint in production do so for the same reason — the retrieval half was treated as a solved problem. This page describes how we actually build them: chunking, hybrid search, reranking, grounded generation, and the evaluation harness that tells you whether any change helped.
The short version
A RAG system is a search engine with a language model attached to the end. The model is the part everyone discusses and the part least likely to be your bottleneck. If the correct passage is not in the retrieved set, no prompt engineering will recover the answer — so the engineering effort belongs in ingestion, chunking, retrieval and ranking.
Two decisions separate systems that hold up from ones that quietly regress: running keyword search alongside vector search, and building an evaluation set before tuning anything. Everything below follows from those.
Have a retrieval system that needs building — or rescuing?
Book a free 30-minute discovery callThe pipeline, stage by stage
Each stage below is a place where quality is won or lost. We instrument them separately so a regression can be attributed rather than guessed at.
- 1Stage 1
Ingestion and normalisation
Source documents arrive as PDFs, HTML, spreadsheets, ticket threads and database rows, and each needs different handling. Layout-aware extraction matters here: a naive PDF-to-text pass destroys table structure and silently corrupts every downstream answer. We normalise to structured text with metadata preserved — source, section, author, timestamp, and the access-control identifiers we will filter on later.
- 2Stage 2
Chunking strategy
Fixed-size splitting is the most common cause of poor retrieval quality. We chunk on document structure — heading boundaries, clauses, logical sections — with modest overlap, and attach the parent heading trail to each chunk so an isolated paragraph retains its context. Chunk size is tuned per corpus: dense contracts and conversational support tickets do not want the same window.
- 3Stage 3
Embeddings and indexing
Chunks are embedded and stored with their metadata. For most workloads pgvector alongside your existing PostgreSQL is the right answer — one fewer system to operate, and transactional consistency between documents and their vectors. A dedicated store such as Pinecone earns its place at large corpus sizes or when you need index features Postgres does not provide. We keep the embedding model swappable, because re-embedding a corpus is routine maintenance, not a crisis.
- 4Stage 4
Hybrid retrieval
Vector search alone reliably fails on exact identifiers — part numbers, error codes, proper nouns, policy references — because semantic similarity is not lexical matching. We run dense vector search alongside keyword search (BM25 or Postgres full-text) and fuse the result sets. Metadata filters are applied inside the query, not after it, so tenant and permission boundaries are enforced by the retrieval layer rather than trusted to the prompt.
- 5Stage 5
Reranking
First-stage retrieval optimises for recall, so we over-fetch and then rerank with a cross-encoder to order candidates by actual relevance to the question. This is usually the highest-leverage single addition to a mediocre RAG system: it lets you pass fewer, better passages to the model, which improves answer quality and reduces token cost at the same time.
- 6Stage 6
Grounded generation with citations
The model answers strictly from retrieved context and cites the passages it used, with instructions to decline rather than speculate when the context does not contain the answer. Citations are not decoration — they are the mechanism that lets a user verify a claim, and they make the system debuggable when an answer is wrong.
- 7Stage 7
Evaluation, then continuous monitoring
We build a labelled question set from your real queries and score retrieval and generation separately. Separating them is essential: if the correct passage was never retrieved, no amount of prompt work will fix the answer. In production we log queries, retrieved passages, latency and cost, and review low-confidence and thumbs-down cases to feed the next iteration.
Want an architecture review of your current retrieval setup?
Book a free 30-minute discovery callRAG, fine-tuning, or neither
These are often presented as competing options. They solve different problems, and a fair number of projects need neither.
| Approach | Solves | Does not solve |
|---|---|---|
| RAG | The model lacks your facts. Data changes often. Answers must cite a verifiable source. | Output style and formatting. Adds retrieval infrastructure you then have to operate. |
| Fine-tuning | Consistent format, tone or domain-specific response behaviour that prompting keeps missing. | Keeping knowledge current. Requires a curated training set and revisiting when the base model changes. |
| Both together | You need house style and live facts — a fine-tuned model retrieving from a current index. | Justifiable only once a RAG baseline is measurably good and style is the remaining gap. |
| Neither | A good search index, a rules engine, or a fixed data model. Often faster, cheaper and fully deterministic. | Open-ended synthesis across many documents — the case where RAG genuinely earns its complexity. |
Our honest default: if your users are asking questions whose answers live in your documents, build RAG and measure it. Consider fine-tuning only once retrieval is demonstrably good and the remaining complaint is about form rather than accuracy. And if a well-built search page would satisfy the actual user need, we will recommend that instead — it is less work to run and easier to trust.
Failure modes we design against from the start
Exact identifiers are never found
Vector search misses part numbers, error codes and policy references because they are lexical, not semantic. Hybrid retrieval with keyword search fixes this, and it is the most common single defect we find in existing systems.
Chunks that lost their meaning
Fixed-size splitting cuts tables in half and severs clauses from their headings. Structure-aware chunking with a preserved heading trail keeps passages interpretable in isolation.
Stale and deleted content resurfacing
Without deletion and re-index handling, removed documents keep answering questions. Index freshness is a pipeline requirement, not a maintenance afterthought.
Permission leakage across tenants
Filtering after retrieval, or asking the model to withhold, both fail eventually. Access filters belong inside the retrieval query so unauthorised passages are never candidates.
Context stuffed to the limit
Passing every candidate passage raises cost and latency and measurably degrades answers. Reranking to fewer, better passages improves quality and spend simultaneously.
No way to tell if a change helped
Without a scored evaluation set, prompt and model changes are decided by anecdote, and regressions ship unnoticed. This is why we build the harness before tuning.
Technology we work with
Core Technology Stack
Component choices are deliberate per project. We keep the embedding model, vector store and generation model behind interfaces, because all three change faster than your product does and none of them should require a rewrite to replace.
Where we are with published RAG work
We do not have a published, named RAG case study we can point you to, and we would rather say that than dress up an unrelated project as one. Our shipped portfolio is application engineering — mobile and web platforms with live video, payments and substantial content models — which you can inspect on our app projects and web projects pages.
What we can offer instead is specificity: bring your corpus and your failure cases to a technical call, and we will walk through the retrieval architecture we would build, the evaluation we would set up first, and where we think your current system is losing answers. If reference work in this exact area is a requirement for you, that is a reasonable position and we will tell you so rather than talk you round it.