A basic RAG prototype takes an afternoon; semantic search that survives production takes months. Failure follows a pattern: fixed-size chunks slicing sentences in half, stale documents ranking forever, mediocre retrieval capping answer quality. A working pipeline assembles five pieces.

Semantic chunking before fixed windows

A fixed 512-token window splits ideas mid-thought. Structure-aware chunking respects headers and paragraphs, with 10-15% overlap between neighbors so context crosses the boundary. Metadata rides along on every chunk: source, section, date, title. That metadata enables tenant filters and citations with links in the final answer.

Embeddings: benchmark on your own corpus

OpenAI's text-embedding-3-small against open-source bge-m3 and e5-large: the winner emerges on your data, in your language, in your domain. Third-party leaderboards lie. Dimension count drives storage cost (1536 dims cost 3x more than 512 in any vector store). Run recall@k on a query set of your own before signing anything.

pgvector covers most cases

An HNSW index inside Postgres, SQL joins between vectors and relational rows, one less system to operate. The graduation trigger to a dedicated vector database arrives near 50M+ vectors or with strict latency SLAs (~30ms p95). Before that point, extra complexity costs more than it gives back.

Hybrid search with reranking

Vector-only retrieval misses proper names, product codes, and acronyms; BM25 misses the rest. Combine both with Reciprocal Rank Fusion, then place a cross-encoder on top (cohere-rerank-3.5 or bge-reranker-v2-m3) to reorder the top-50 candidates into the final top-k. Tune top_k per query type: a FAQ holds at 5, legal analysis wants 15.

An evaluation loop from day one

Build a golden set of 50-200 question-answer pairs from real queries, stored next to the prompts. Measure hit-rate@k before and after every pipeline change. Swapped the embedding model? Everything requires re-embedding anyway, so rerun the full eval suite in the same pass.

Structure-based chunking, embeddings benchmarked on your own data, hybrid retrieval with reranking, continuous evaluation: four pieces separate demos from products.