Building a RAG (Retrieval-Augmented Generation) demo takes an afternoon. Building a RAG system that doesn’t hallucinate or miss obvious answers takes months of tuning.
We have all been there: You spin up a vector database, dump in your documentation, and hook it up to an LLM. It works great for “Hello World” questions. But when a user asks something specific, the system retrieves the wrong chunk, and the LLM confidently answers with nonsense.
The problem isn’t usually the LLM (Generation); it’s the Retrieval.
In this engineering guide, based on real-world production data from a massive Help Desk deployment, we are going to dissect the three variables that actually move the needle on RAG accuracy: Data Cleansing, Chunking Strategy, and Embedding Model Selection.
We will look at why “Semantic Chunking” might actually hurt your performance, and why “Hierarchical Chunking” is the secret weapon for complex documentation.
The Architecture: The High-Accuracy Pipeline
Before we tune the knobs, let’s look at the stack. We are building a serverless RAG pipeline using AWS Bedrock Knowledge Bases. The goal is to ingest diverse data (Q&A logs, PDF manuals, JSON exports) and make them searchable.
Optimization 1: Data Cleansing (The Hidden Hero)
Most developers skip this. They dump raw HTML or messy CSV exports directly into the vector store. This is a fatal error.
Embedding models are sensitive to noise. If your text contains
The Experiment
We tested raw data vs. cleansed data.
- Raw Data: Direct export from CRM/Salesforce.
- Cleansed Data: Removed HTML tags, standardized terminology (e.g., “FAQ” vs “F.A.Q.”), and stripped headers/footers.
The Result:
- Search Accuracy improved by ~30%.
- In specific technical domains, accuracy jumped from 59% to 77%.
The Code: A Simple Cleaning Pipeline
Don’t overcomplicate it. A simple Python pre-processor is often enough.
import re
from bs4 import BeautifulSoup
def clean_text_for_rag(text):
# 1. Remove HTML tags
text = BeautifulSoup(text, "html.parser").get_text()
# 2. Remove noisy separators (e.g., "-------")
text = re.sub(r'-{3,}', ' ', text)
# 3. Standardize terminology (Domain Specific)
text = text.replace("Help Desk", "Helpdesk")
text = text.replace("F.A.Q.", "FAQ")
# 4. Remove extra whitespace
text = re.sub(r's+', ' ', text).strip()
return text
raw_data = "<div><h1>System Error</h1><br>-------<br>Please contact the Help Desk.</div>"
print(clean_text_for_rag(raw_data))
# Output: "System Error Please contact the Helpdesk."
Optimization 2: The Chunking Battle
How you cut your text determines what the LLM sees. We compared three strategies:
- Fixed-Size Chunking: Split text every 500 tokens. (The baseline).
- Semantic Chunking: Split text based on meaning shifts (using embedding similarity).
- Hierarchical Chunking: Retrieve small chunks for search, but feed the “Parent” chunk to the LLM for context.
The Surprise Failure: Semantic Chunking
We expected Semantic Chunking to win. **It lost.
In a Q&A dataset, the “Question” and the “Answer” often have different semantic meanings. Semantic chunking would sometimes split the Question into Chunk A and the Answer into Chunk B.
- Result: The system found the Question but lost the Answer. Accuracy dropped by 10-18% compared to Fixed Chunking.
The Winner: Hierarchical Chunking
Hierarchical chunking solved the context problem. By indexing smaller child chunks (for precise search) but retrieving the larger parent chunk (for context), we achieved the highest accuracy, particularly for long technical documents.
-
Business Domain Accuracy: 94.4% (vs 88.9% for Fixed).
Optimization 3: Embedding Model Selection
Not all vectors are created equal. We compared Amazon Titan Text v2 against Cohere Embed (Multilingual).
The Findings
- Short Q&A (Science/Technical):
- Cohere Embed outperformed Titan. It is highly optimized for short, semantic matching and multilingual nuances.
- Accuracy: 77.3% (Cohere) vs 54.5% (Titan).
- Long Documents (Business/Manuals):
- Titan Text v2 won. It supports a larger token window (up to 8k), allowing it to capture the full context of long policies or manuals.
- Accuracy: 94.4% (Titan) vs 88% (Cohere).
Developer Takeaway: Do not default to OpenAI text-embedding-3. If your data is short/FAQ-style, look for models optimized for dense retrieval (like Cohere). If your data is long-form documentation, look for models with large context windows (like Titan).
The Final Verdict: How to Build It
Based on our production deployment which reduced support ticket escalation by 75%, here is the blueprint for a high-accuracy RAG system:
1. Know Your Data Type
- Is it Q&A / Support Logs?
- Use Fixed-Size Chunking. (Don’t let Semantic chunking split your Q from your A).
- Use an embedding model optimized for short text (e.g., Cohere).
- Is it Manuals / Long Docs?
- Use Hierarchical Chunking.
- Use an embedding model with a large context window (e.g., Titan v2).
2. Clean Aggressively
Garbage in, Garbage out. A simple RegEx script to strip HTML and standardize terms is the highest ROI activity you can do.
3. Don’t Trust Smart Defaults
Semantic Chunking sounds advanced, but for structured data like FAQs, it can actively harm performance. Test your chunking strategy against a ground-truth dataset before deploying.
RAG is not magic. It is an engineering problem. Treat your text like data, optimize your retrieval path, and the “Magic” will follow.
