I replayed 13 months of real churn on a RAG index - 90% of live chunks were stale, orphaned or duplicated.

A reproducible teardown of what happens to a vector index after a year of real document churn, across pgvector, Qdrant and Chroma.

Why I built this

I build, consult and teach in the GenAI domain, and I'm an active AWS Community Builder in AI Engineering category. One of my friends came to me with some version of "Why retrieval quality of our in-house RAG degrades over time?" - and I didn't have a satisfying answer other than "let me look". The root cause was removal of a bunch of docs belonging to another track, and this situation inspired me to write this post.

I built a version of the worst case on purpose: took a real corpus with real edit history, replayed a year of it the way real ingestion pipelines actually behave (including their bugs), and then measured exactly what's wrong with the result. Engine by engine, against a ground-truth ledger that knows what should be true at every point in time.

The corpus is the GitLab Handbook - a real company wiki, thousands of Markdown pages, years of git history, CC BY-SA 4.0 licensed (per GitLab's own "Our Handbook is open source: here's why", not the repo's boilerplate LICENSE file, which is silent on content). It reads exactly like the internal wiki or Confluence space most RAG buyers actually index. On top of it: 100 clearly-labeled synthetic employee records (Faker, seed=42, not real people) for a GDPR erasure demo, since a real company handbook by itself has no personal data to erase.

Method

• T0 → HEAD: picked a commit ~13 months before today (d9dfa4eb7a, 2025-06-02) as the starting index, then walked 4,742 real first-parent commits forward to HEAD (6a7e263deb, 2026-07-10, the day I started the experiment) - actual edits, deletes, and renames from a real repository's history, not simulated churn.
• One embedding pass, three engines: every chunk is embedded exactly once (BGE-M3, fp32, HF revision 5617a9f61b, sentence-transformers pinned to 5.6.0) and the identical vectors are upserted into pgvector (HNSW, cosine), Qdrant (cosine), and Chroma (cosine) - all on documented defaults, no engine given the other's tuning. Docker image digests, client library versions, and the master seed (42) are all pinned and published in the repo.
• The ingestion bugs are on purpose. Production RAG post-mortems consistently blame ingestion and change-detection, not the model, for index rot. I modeled four canonical failure modes on labeled subsets of the real churn:
  • (A) Update-without-delete - re-embed and upsert on edit, but never delete the old chunks. The single most common real bug.
  • (B) Async/eventually-consistent deletes - issue the delete call but don't wait for it to actually take effect before considering the job done.
  • (C) Full re-ingest duplication - one mid-timeline "someone re-ran the whole pipeline" event that re-upserts everything under new ids.
  • (D) Unhandled renames - index the new path as a new doc, leave the old path's chunks orphaned.
  • Plus a GDPR erasure demo: Article-17-style "delete this person" requests for 40 of the 100 synthetic employees, issued via each engine's normal delete API, with no manual VACUUM/optimize run afterward - because that's what real teams do: call the delete API and assume it's handled.

Every single mutation - real or injected-bug - is logged to a SQLite ledger with the expected index state per engine. That ledger is what makes every number below checkable.

Findings

Full scorecard →

The headline: across all three engines, 89.73% of what's still live in the index fails at least one check - stale, orphaned or a duplicate. All three engines agree almost exactly with each other, because staleness and orphan status are pure ledger facts, not engine-dependent behavior. That figure is a real set union (chunk_id set operations against the ledger, not a sum of the rows below. Those three categories overlap heavily, so adding their individual percentages would double-count and isn't how this number was computed):

• 116,798 live chunks are stale. It's 69.96% of the whole live index, or 76.59% of the 152,504 chunks whose source document still exists. Their source document has since been edited and the index still serves the old version. Identical in pgvector, Qdrant and Chroma.
• 131,241 chunks (78.61%) are orphaned - their source doc is deleted, renamed or superseded by a newer version that was never cleaned up. Breakdown: 4,723 from outright-deleted docs, 9,720 from renames and 116,798 from ordinary "the doc was edited and the old chunks never got removed" (failure mode A, the dominant driver). Every stale chunk is also an orphan - staleness's predicate is literally the "superseded_live_doc" cause of orphan status, not an independent category, so these two rows are not additive with each other.
• ~76% of live chunks are duplicates (exact plus near-duplicate combined). 111,758 (66.9%) are exact byte-identical re-embeds of already-indexed text - this is not just the deliberate mid-timeline reingest event. The bigger driver is mundane: editing one paragraph in a multi-chunk doc reprocesses the whole doc into new chunk_ids, so every unchanged chunk becomes a byte-identical duplicate of itself once re-upserted. If that update landed in the update-without-delete bucket, both copies persist forever. A further cosine-ANN pass (>=0.98 similarity, no ledger ground truth) on the 18,428-chunk remainder (166,947 live chunks minus 111,758 excess exact duplicates minus 36,761 kept canonical originals of those duplicate clusters) found 15,303–15,740 more near-duplicates, engine-dependent (real ANN variance, not a bug). Unlike stale/orphaned, the duplicate set is mostly not covered by the other two checks, which is why the true union (89.73%, or 91.30–91.33% including the cosine-ANN layer) comes out higher than any individual row above, not lower. Full per-engine union, 7-region breakdown, and the complement (chunks passing every check: 17,147/10.27% conservative, 14,468–14,529/8.67–8.70% including cosine-ANN) are committed in audit file as chunk_set_union_analysis.

The good news, and it's a real finding, not an absence of testing: deletion itself held up. I queried for the 637 real git-deleted docs and the 40 GDPR-erased synthetic employees using their own original text as the worst-case query - the query most likely to surface a leak - at both top-5 and top-10, in all three engines. Zero leaked. I also checked storage-layer persistence directly (not just "does the API return it") - also zero. For pgvector specifically, the storage layer was dirty first - 254,060 dead tuples sat on disk right after the churn replay and got reclaimed by PostgreSQL's own background autovacuum (no manual VACUUM, ever) before this audit read the heap. The persistence window was real; it just closed on its own. This distinction matters: EDPB Guidelines 05/2019 are explicit that erasure has to be verifiable and irreversible - "the API stopped returning it" alone doesn't satisfy that and you have to check the storage layer too, which is what "Ghost Vectors" (https://arxiv.org/abs/2606.18497) found soft-deleted HNSW vectors failing on elsewhere. Here, both checks came back clean, in all three engines.

I don't want to overclaim a clean result any more than I'd want to overclaim a dirty one, so I cross-validated the detection mechanism itself. I ran the identical query-and-check pipeline against 200 chunks per engine that are known to still be live, confirming each finds itself. Pass rate: 97.5% (pgvector), 98.0% (Qdrant), 98.5% (Chroma) - the small residual gap is ordinary approximate-HNSW recall loss under untuned defaults, isolated by excluding chunks with exact-duplicate siblings (duplicate-tie crowding was initially misleading a naive version of this check down to 86%). If the detection pipeline can find a chunk that's definitely there 97–98% of the time, "zero found" for erased content is a real signal, not a broken probe.

Ground-truth verification

Every check above is scored against the ledger, not just eyeballed:

• Staleness detection: 1.0 precision / 1.0 recall against the independently-re-derived ledger predicate.
• Orphan detection: 1.0 precision / 1.0 recall.
• Exact-hash duplicate detection: 1.0 precision / 1.0 recall (true by construction - hash equality is unambiguous). The cosine-ANN layer has no ledger ground truth for "near but not identical" content, so it's reported as an upper bound, not a confirmed count.

The methodology behind this post is the same one the paid audit runs on a customer's real index. "Trust me" wouldn't be good enough there, so it doesn't get a pass here either.

Honest limitations

• One corpus. This is the GitLab Handbook's specific edit patterns - a different corpus with different churn shape could look better or worse.
• Defaults, not tuned, and I'm saying so. All three engines ran their documented default deletion/index behavior. That's deliberate - most teams run defaults - but it means these are not the best numbers each engine could produce with tuning.
• Not a speed benchmark. No cross-engine timing claims anywhere in this post or the raw results.
• Synthetic PII is clearly labeled. It was generated with a fixed Faker seed, kept in its own namespace, never mixed with real handbook attribution.
• HNSW build-order nondeterminism is mitigated (chunk_id-sorted insertion, fixed seed) but not eliminated - rerun variance on the order of single-digit vectors is possible.
• Qdrant's server-side deleted-vector count isn't directly measurable in this server version - I'm reporting the optimizer settings that gate physical reclaim (deleted_threshold=0.2, vacuum_min_vector_number=1000) rather than inventing a number that doesn't exist.
• This is a self-built experiment, not a client engagement. No customer data, no client's actual index.
• I sell an audit built on this exact methodology, so I have a stake in these numbers being credible - same reason Supabase disclosed their Postgres bias in the pgvector-vs-Qdrant thread. It's also why every number here traces to a checkable ledger and a committed JSON file.

Reproduce it

Please go to the experiment repo.

Why not just use X

The obvious question. Four adjacent categories exist, and none of them does this.

RAG eval tools

(Ragas, Arize Phoenix, LangSmith, DeepEval, TruLens, Future AGI) score answer quality (faithfulness, context precision, groundedness) against a sampled question set. Their own vendors' framing is that these metrics assume the retrieval index is trustworthy. That assumption is exactly what breaks here: a pipeline can score 0.95 faithfulness while serving a 13-month-old chunk, because the eval was built against the same stale corpus. Evals sample the questions you thought to ask; this is a census of what's actually in there. Phoenix's embedding-cluster view gets closest to index introspection, but it's a debugging aid, not an inventory.

Embedding PII scanners

(IronCore VectorLens, BigID) answer "is there sensitive data in these vectors?" - a different axis from "should this vector still be here at all?". VectorLens is the closest structural analogue to what I do: self-hosted, read-only, CLI, scans your own exported embeddings. It has no staleness, orphan, or duplicate detection so is a complementary tool.

Deletion-verification libraries

(mnemo) overlap the erasure check specifically, and genuinely so: as of mnemo's v1.9.9 release (July 17, 2026), mnemo's ErasureAuditor re-attempts recovery of an erased subject across registered stores using verbatim scans and NN-inversion, with soft-delete residue probes naming pgvector, Qdrant and Chroma by name. The difference is scope and wiring - it verifies deletion propagation for a named subject across stores you register in your application, which is a real and useful thing to have in-app. It doesn't census an external index for staleness, orphans or duplicates, and it needs adapters written into your codebase.

The vector databases themselves

are the subject of the audit, not the auditor. All of them soft-delete by design: pgvector accumulates dead tuples until VACUUM, Qdrant holds tombstones until a segment crosses deleted_threshold, Chroma's HNSW grows unbounded. None of them will tell you that a deleted source document is still in your top-k - which, to be clear, is precisely the check that came back clean here.

...and the one that isn't a product at all:

"Just write a SQL cron job comparing timestamps."

That is genuinely the mechanism, and for teams with a disciplined document_id→chunk→vector mapping, several of these checks do reduce to a query. The part that doesn't reduce is knowing whether the cron caught everything - which is what the ledger reconciliation above is for, and why the precision/recall numbers are in this post at all.

If you want this on your own index