Your RAG Pipeline Doesn’t Need a Vector Database, It Needs a Cache Invalidation Strategy

The dominant framing around retrieval-augmented generation is that quality lives in the retrieval layer: pick the right embedding model, tune your chunk size, add a reranker, maybe fine-tune on your domain. Teams pour weeks into this and ship pipelines that demo beautifully. Then the source documents change, and the system keeps confidently answering from context that no longer exists. Nobody notices for a while, because a wrong answer that reads fluently looks exactly like a right one.

Here is my position, stated plainly: in production, RAG failures are almost never embedding failures. They are invalidation failures. A vector database is a cache — a derived, denormalized copy of your source of truth optimized for similarity lookup. And like every cache, its hard problem is not how fast you can read from it. It’s knowing when what you read is a lie. If you cannot answer what happens to answers already generated when the source document changes, you don’t have a retrieval system. You have a very expensive way to serve stale context with high confidence.

Why the embedding layer gets all the attention

The embedding pipeline is attractive to build because it is legible and finite. You can measure recall@k. You can A/B two chunking strategies. You can swap text-embedding-3-small for something larger and watch a number move. There is a satisfying loop of tuning and validation, and it produces artifacts — dashboards, eval sets, notebooks — that look like progress to everyone in the standup.

Cache invalidation offers none of that comfort. It’s distributed-systems plumbing. It touches your ingestion path, your document lifecycle, your permissions model, and your generation logs all at once. It has no clean metric that goes up and to the right. And it only fails in ways that are hard to attribute, because the symptom — a wrong answer — is indistinguishable from a hundred other causes. So it gets deferred. The retrieval layer gets polished to a mirror finish while the reconciliation layer stays at “we reindex nightly and hope.”

This is the same misdirection that has haunted caching for forty years. Phil Karlton’s line that there are only two hard things in computer science — cache invalidation and naming things — is a joke, but it endures because the invalidation half is genuinely where systems rot. RAG rediscovered the read path and forgot the write path came with it.

The failure isn’t retrieval, it’s reconciliation

Walk through what a vector store actually is. You take a document, split it, embed each chunk, and store the vectors alongside the chunk text and some metadata. That store is now a projection of your source data at a point in time. The moment the source changes — an edit, a deletion, a policy update, a new version of a contract — your projection is wrong until you rebuild it. Between the change and the rebuild, every query that hits the affected region returns confidently outdated context.

The interesting failure modes aren’t the obvious “document deleted, chunk still served.” They’re subtler and worse:

Contradiction across versions. Suppose a knowledge base article about refund policy gets edited from “30 days” to “14 days.” If your ingestion appends rather than replaces — which naive pipelines do, because upsert-by-identity is more work — both chunks now live in the index. A query for the refund window retrieves both. The model sees two authoritative-looking statements that disagree and picks one, or worse, hedges by mentioning both. You’ve built a system that manufactures ambiguity out of a source that was unambiguous.

Partial reindex skew. You update a document that spans twelve chunks. Reindexing fails on chunk seven due to a rate limit on the embedding API. Now eleven chunks reflect the new version and one reflects the old. Retrieval stitches them together and the model reasons over a document that never existed in that form. Nothing errored. Nothing alerted.

Provenance rot. An answer gets generated, cached, and surfaced in a UI. Three days later the source is retracted for legal reasons. The generated answer is still sitting in a support macro, a Slack thread, a cached response. The vector store’s invalidation, even if perfect going forward, does nothing about the answers already emitted into the world.

None of these are solved by a better embedding model. A perfect embedding of stale text retrieves stale text perfectly.

The question that exposes an unserious pipeline

When I want to know whether a RAG architecture is production-grade, I ask one thing: when a source document changes, what is the propagation path to every place its content already lives? The answer reveals everything.

A serious team can trace it: the change fires an event, the affected chunks are identified by stable document ID and version, old vectors are removed or superseded, new ones are written atomically, and any downstream caches keyed on those chunks are invalidated. They can also tell you the staleness window — the bounded time between a source edit and the index reflecting it — and they treat that window as an SLO, not an accident.

An unserious team says “we run a nightly job.” Which means the answer to “what happens when the source changes at 9 a.m.?” is: nothing, for up to twenty-four hours, and the system will argue with the new reality the entire time. That might be completely fine for a corpus of academic papers that never change. It is a live incident waiting to happen for anything transactional — pricing, policy, inventory, legal terms, on-call runbooks.

The point of the question isn’t to shame nightly reindexing. Batch reindexing is a legitimate strategy for slow-moving corpora. The point is that most teams have never chosen their staleness window. They inherited it from whatever the ingestion script happened to do, and they’ve never mapped it against how fast their source of truth actually moves. That mismatch is where RAG production failures cluster.

Versioning is the actual product surface

Once you accept that the vector store is a cache over a source of truth, the design problem reframes cleanly: you are building a materialized view that must stay reconciled with its base tables. The tools for that are old and well understood. We just have to admit that’s what we’re doing.

Give every chunk a stable identity and a version. Not a random UUID assigned at ingest — a deterministic key derived from (document_id, chunk_position) or a content-addressed hash, plus the source document’s version or content hash. This is the single highest-leverage decision in the whole pipeline. With stable identity, an edit becomes an upsert that replaces the old chunk in place. Without it, every reindex is an append, and your store slowly fills with contradictory ghosts. Content-addressed chunk IDs also make dedup free and let you skip re-embedding unchanged chunks, which is where most of your embedding-API cost hides.

Make invalidation event-driven where the corpus is live. If your documents live in a system that emits change events — a CMS webhook, a database CDC stream via something like Debezium, an object-storage notification — subscribe to it. The change event carries the document ID; you re-derive its chunks, diff against what’s indexed, and upsert only the delta. This collapses the staleness window from hours to seconds and turns reindexing from a blunt full-corpus sweep into a targeted patch.

Treat deletion and retraction as first-class. Soft-delete markers, tombstones, an is_active flag filtered at query time. The system must be able to say “this content existed and is now gone” without waiting for a rebuild. Hard-deleting from a vector index mid-query is its own consistency hazard; a filtered tombstone you sweep later is usually the calmer path.

Carry version metadata into generation and log it. When you retrieve a chunk, record which document version it came from alongside the generated answer. Now, when a source is retracted, you can query your generation log for every answer that depended on it and decide what to do — invalidate a cache, flag a review, re-run the answer. This is the only mechanism that addresses the provenance-rot problem, and it costs almost nothing to add at build time and is nearly impossible to retrofit under pressure.

I’ll grant the strongest counterargument here directly: for a genuinely static corpus — regulatory archives, a fixed set of published papers, product manuals for discontinued hardware — all of this is over-engineering. If the source truly never changes, the cache never goes stale, and a one-time index is correct forever. Fair. But be honest about how rare that is. Most corpora people call “static” are static only because nobody’s tracking the edits. The FAQ gets updated. The onboarding doc gets a new step. The “final” contract template gets a clause added. Assumed-static-but-actually-drifting is the default state, and it’s the most dangerous one, because the pipeline was designed as if reconciliation would never be needed.

Embedding drift is the same problem on a longer clock

There’s a slower, sneakier version of this that deserves its own name: embedding drift in production. Your vectors are only comparable if they were produced by the same model. The day you upgrade your embedding model — and you will, because the vendors deprecate them and better ones ship — every vector generated by the old model is now in a subtly different space than every vector generated by the new one. Cosine similarity across that boundary is meaningless. Retrieval quality degrades in a way that looks like model regression but is actually a versioning failure in disguise.

The same discipline solves it. Tag every vector with the embedding model and version that produced it. Never mix versions in a live index; either maintain the model version as part of your index partition or run a full backfill before cutting over. Treat an embedding-model upgrade as a schema migration, because that’s structurally what it is: you’re changing the encoding of every row and you need a migration plan, a backfill, and a cutover, not a config change you push on a Friday.

This is why I distrust pipelines that can’t tell you which model embedded which chunk. It means an embedding upgrade will silently corrupt retrieval, and the team will spend a sprint blaming the reranker.

The localized corpus makes all of this concrete

Multilingual RAG is where the invalidation problem stops being abstract, because you have not one source of truth but a graph of them. A source document in English, its translations into eight languages, and embeddings for each. Edit the English source and you’ve invalidated far more than one chunk: every translation is now stale, and every embedding derived from every translation is stale behind it. If your pipeline re-embeds the English delta but leaves the translations pointing at the old version, your German-speaking users get answers grounded in last month’s policy while your English-speaking users get this month’s.

This is a big part of why translation belongs in a system that tracks source-to-target state rather than in a folder of exported strings. In my product LingoHub, the value for a RAG context is precisely that a change to a source segment marks its translations as outdated and pending review — you have an explicit, queryable signal that a downstream artifact is stale. That signal is exactly the invalidation event a multilingual retrieval pipeline needs to key off. Without something tracking that relationship, “reconcile the translated corpus after a source edit” becomes a manual guess, and the guess is usually “we’ll catch it in the next full reindex,” which is the nightly-job answer wearing a different hat.

What I’d build, and what I’d cut

If I were standing up a production RAG system tomorrow, I’d spend the first design conversation not on embedding models but on the source of truth and its change semantics. How does content enter the system? How does it change? Does it emit events, or must I poll? What’s the acceptable staleness window per content type — seconds for pricing, hours for docs, effectively infinite for the archive? Those answers determine the architecture. The embedding model is a swappable component I can upgrade later precisely because I versioned it.

Concretely, the non-negotiables: deterministic content-addressed chunk IDs, upsert-by-identity so edits replace rather than append, event-driven invalidation wherever the corpus moves faster than my tolerable staleness window, tombstones for deletion, embedding-model version tags on every vector, and version metadata threaded from retrieval into the generation log. That last one is what lets me answer the question that started this — what happens to answers already generated when the source changes — with something better than a shrug.

What I’d cut, or at least defer: elaborate chunking heuristics, exotic rerankers, fine-tuned embeddings. Not because they’re worthless — they move retrieval quality at the margin — but because they optimize the read path of a cache whose write path is broken. A 3% recall improvement is invisible next to a system that serves a contradicted refund policy for eighteen hours. Fix the reconciliation first. The vector database is the easy part; it was always the easy part. The hard part is the one every caching system has fought since before RAG had a name, and pretending this generation of AI infrastructure is exempt is how calm systems turn into 2 a.m. pages.

References

• Phil Karlton, “There are only two hard things in Computer Science: cache invalidation and naming things.” Widely attributed; discussed at martinfowler.com/bliki/TwoHardThings.html
• Martin Kleppmann, Designing Data-Intensive Applications (O’Reilly, 2017) — materialized views, derived data, and change data capture as the model for keeping caches reconciled with a source of truth
• Debezium documentation — change data capture patterns for event-driven propagation of source changes (debezium.io/documentation)
• OpenAI embeddings guide — model versioning and the requirement that compared vectors originate from the same model (platform.openai.com/docs/guides/embeddings)
• Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020), arXiv:2005.11401 — the original RAG formulation, useful for grounding what the retrieval component is and is not responsible for