Peeling the Onion of Vector Search: From FAISS to from the Scratch Algorithmic Implementation
A few weeks ago, I found myself staring at FAISS, LangChain, and other vector search tools, thinking… “How does this actually work?”
At first glance, they’re magic. You feed in some documents, ask a question, and poof — relevant passages appear. But I wanted to understand the mechanics, layer by layer, like peeling an onion.
And what started as curiosity quickly turned into a small experiment.
Experiment
I wanted to build a mini research assistant for distributed systems. The goal? Ask questions like:
- What are the tradeoffs of consensus algorithms?
- How do distributed databases maintain consistency?
- What are the design principles behind scalable systems?
…and instantly retrieve passages from 50–100 seminal research papers and books.
It wasn’t about building a perfect product. It was about understanding the magic underneath.
The First Working Pipeline
Before reinventing anything, I built a simple pipeline using existing tools. The architecture looked like this:
Surprisingly, this entire pipeline can be implemented in less than 100 lines of code. But understanding what happens at each step reveals how much complexity hides underneath.
Turning PDFs Into Text
PDFs might look like text, but they’re really instructions for drawing pages: blocks, fonts, coordinates. Copying and pasting often yields gibberish.
Libraries like PyMuPDF or PyPDF2 reconstruct readable text. Why does this matter? Because messy text leads to messy embeddings, and messy embeddings break search. In short: garbage in → garbage out.
Choosing the Right Embedding Model and Dimension
Next: embeddings.
I asked myself:
- Will this model understand the concepts I care about?
- How many dimensions can it produce?
- How much memory and latency will it consume?
I compared text-embedding-3-small (1536 dimensions) and all-MiniLM-L6-v2 (384 dimensions). For a local, cost-effective setup, all-MiniLM-L6-v2 won.
Why dimensions matter: higher dimensions capture nuance but slow everything down and bloat memory. Rough rule of thumb:
Memory ≈ number_of_vectors × dimension × 4 bytes
Millions of vectors? You quickly see why startups obsess over embedding size.
Chunking the Text
Embedding an entire research paper as a single vector? Too coarse. We’d lose nuance.
Solution: break it into chunks, with some overlap to preserve context:
Chunk 1: sentences 1–10
Chunk 2: sentences 9–18
Tiny chunks → higher precision but less context. Large chunks → more context but lower precision.
Many practitioners say: vector search quality often depends more on chunking than on the algorithm itself. Pick poorly, and your “magic” vanishes.
Generating Embeddings
Each chunk becomes a point in a high-dimensional space. Suddenly, your corpus isn’t just text—it’s a cloud of vectors. A universe where each idea occupies its own coordinates.
Indexing With FAISS
Now, brute force would be slow. Imagine computing distances to millions of vectors for every query. Enter FAISS, Meta’s library for fast similarity search.
FAISS builds indexes optimized for speed, using clever data structures and hardware acceleration.
At this point, the system works: you ask a question, it embeds the query, searches for nearest vectors, and returns the most relevant chunks.
At this stage, I realized: I had built the retrieval component of a RAG pipeline.
Peeling the Onion Further: Brute-Force Search
But I wanted to understand why FAISS works so well. So I went back to basics: brute-force nearest neighbor search.
Algorithm:
- Compute the distance between the query vector and every document vector.
- Sort by similarity.
- Return the top K results.
for vector in dataset:
score = similarity(query, vector)
return top_k(scores)
Complexity: O(N × d) (N = vectors, d = dimensions). Slow—but provides ground truth for benchmarking smarter algorithms later.
Before diving into FAISS, I wanted to get my hands dirty and see how vector search works at the most basic level. So, I implemented L1, L2, and cosine similarity indexing from scratch. Check out the full implementation on my GitHub
Peeking Under the Hood of FAISS
At its core, FAISS builds indexes—specialized data structures that let you skip huge swaths of vectors and zoom straight to the most promising candidates. Flat indexes are exact but scan everything. IVF clusters the space into neighborhoods, so you only visit a few. HNSW creates a navigable graph where vectors point to their nearest neighbors, letting queries hop quickly through the most relevant regions.
Think of it like this: instead of wandering every aisle in a library, FAISS gives you a map, a guided path, and sometimes even a speedboat across oceans of vectors. Each index type is a different way of trading speed, memory, and accuracy, and choosing the right one is like picking the right tool for your adventure.
The Next Question: How Do We Evaluate Retrieval?
Once the pipeline works, how do we know if it’s any good?
- Precision — Of the top K results, how many are actually relevant?
- Recall — Of all relevant documents, how many did we find?
- Mean Reciprocal Rank (MRR) — How far down the list does the first relevant result appear?
- Query Latency – How fast do I get the search results?
Conclusion
We now have a working pipeline: PDFs → chunks → embeddings → FAISS → query.
Next, in Part 2: the curse of dimensionality, approximate nearest neighbor algorithms like HNSW, IVF, PQ, and the hidden trade-offs that make vector search feel like magic at scale.
