Architecting RAG Pipelines for Production

Moving beyond basic vector search. How we build reliable Retrieval-Augmented Generation systems that don't hallucinate.

Building a prototype Retrieval-Augmented Generation (RAG) system is simple: load a PDF, generate vector embeddings, run a cosine similarity query, and send the results to OpenAI. However, moving RAG systems to production reveals significant challenges: high token costs, hallucinations, retrieval noise, and the 'lost-in-the-middle' context window issue.

1. The Semantic Chunking Pipeline

Standard chunking splits text by raw character limits (e.g. 500 characters). This often slices sentences in half, discarding valuable context. We solve this by implementing semantic chunking, which determines boundaries based on semantic shifts between adjacent sentences.

Once chunks are created, we generate both raw embeddings and parent-child linkages. If a query matches a small child chunk, the system retrieves the larger parent block to ensure the model receives complete surrounding context.

2. Hybrid Keyword and Vector Search Routing

Vector embeddings excel at semantic matching, but struggle with precise product codes, client IDs, or specialized terminology. We build hybrid search pipelines that combine vector retrieval (e.g. Pinecone) with traditional BM25 keyword matching, then re-rank results using models like Cohere Rerank.

// Fetch candidates from vector database and adjust rankings via Cohere API
export async function rerankCandidates(query: string, candidates: Candidate[]) {
  const response = await fetch("https://api.cohere.ai/v1/rerank", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.COHERE_API_KEY}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      model: "rerank-english-v3.0",
      query,
      documents: candidates.map(c => c.text),
      top_n: 3
    })
  });
  
  const data = await response.json();
  return data.results.map((r: any) => ({
    text: candidates[r.index].text,
    score: r.relevance_score
  }));
}

3. Handling Context Window Re-ordering

LLMs can fail to locate information hidden in the middle of long prompts. This behavior is known as the 'lost-in-the-middle' effect. If a long system prompt contains many relevant data blocks, the model usually pays attention to items placed near the beginning or the end of the text.

To address this, our retrieval system re-orders the extracted context blocks before sending the prompt to the model. We place the highest-scored results at the top and bottom of the prompt, and the secondary results in the middle. This layout ensures the model makes use of the retrieved data.

4. Best Practices for RAG Pipelines

• Address Lost-in-the-Middle: Reorder retrieved items so that the most relevant documents are positioned at the beginning and the end of the context window, leaving the middle for secondary details.
• Implement Strict Schema Formatting: Utilize Zod libraries to validate LLM outputs, preventing JSON syntax issues.
• Run Continuous Evaluation: Track retrieval quality using Ragas metrics (Faithfulness, Answer Relevance, Context Recall) in staging environments.
• Implement Semantic Caching: Cache common queries and responses in Redis to reduce API costs and latency.

Transitioning RAG architectures from basic concepts to production-grade systems requires structuring data and evaluating output quality. The result is a robust, reliable AI system that scales efficiently.