RAG Content Ops: Fix Stale, Duplicate and Conflicting Source Docs

It gives confident, wrong answers that quote your own documents. The bot is not making things up; it is faithfully repeating an old page that nobody deleted. RAG content ops is the work that stops this, by fixing the documents instead of the bot.

Picture a common setup. An online electronics shop runs a support bot on its help-centre wiki, and the return window recently dropped from 30 days to 10 days. The setup is made up, but this kind of conflict is what the research below studies.

The support team updated the main returns page. Nobody touched the 2024 version still sitting in an old folder. The sales team had also copied that old page into their own space back in 2024.

Now a customer asks how many days they have to return a phone. The bot finds three pages that look almost the same. Two say 30 days and one says 10. It answers 30, and the customer misses the real deadline.

The engineers try the usual fixes: fetch fewer documents, fetch more, rank newer pages higher. Each change fixes one question but leaves another wrong. This guide explains why, and what to do instead.

• Source documents: the pages, PDFs and help articles your RAG system is allowed to answer from. People often call this set the knowledge base.
• Retriever: the part that searches the source documents and picks the pieces most similar to the question.
• Chunk: a small piece of a document, a few sentences to a few paragraphs, that the retriever stores and returns.
• Index: the searchable store of chunks the retriever looks things up in.
• Top-k: how many chunks the retriever hands to the model, for example the top 3.
• Stale document: a page whose facts were true once but are not now.
• Near-duplicate: two pages that share most of their wording, such as a copy with a few edits.
• Contradiction: two pages that state different values for the same fact, such as 30 days and 10 days.
• Owner: the team responsible for keeping a page correct, and for archiving it when it is no longer needed.

RAG content ops is the set of checks, metadata and habits that keep your source documents true, single-copy and owned. Metadata means small labels stored with each page, such as its owner and last review date.

It matters because a RAG system can only be as right as the pages it reads. The paper that introduced RAG models described their document store as a searchable index of Wikipedia. It treats that store as the system's memory, kept separate from the model's own learned knowledge.

If the memory holds two answers to one question, the model has to pick. Research on this, covered below, shows models do not pick reliably. So the cheapest place to fix the answer is the page.

How it differs from keeping the index fresh

Keeping an index fresh means making the index match the source documents. Edited pages get re-embedded, turned again into the number lists the retriever searches, and deleted pages get removed. Our guide on RAG index freshness covers that sync job.

Content ops sits one step earlier. A perfectly fresh index of a wrong page still serves a wrong page. Freshness asks whether the index matches the wiki; content ops asks whether the wiki matches reality.

Because a retriever ranks pages by how well they match the question, not by whether they are true. A stale page and a current page about returns often match a returns question almost equally.

Think of a library that kept every old edition of a rule book on the same shelf as the new one. A faster librarian finds rule books quicker. The librarian still hands you the edition whose cover best matches your request, not the one that is in force.

Mapping it back: the shelf is your index, the librarian is the retriever, and the editions are stale and current pages. Tuning makes the librarian faster or pickier. It does not remove the old editions.

Where the picture breaks: a human librarian can read the date on the cover. A retriever only sees dates if you store them as metadata and use them, and our toy run shows dates can mislead too.

What research says models do with conflicting pages

A 2024 survey of knowledge conflicts calls disagreement between retrieved passages an inter-context conflict. It names outdated information and misinformation as two causes.

The same survey reports studies where models favour answers backed by more documents, or repeated more often in the context. That is the pattern a stale page plus its copy produce: two votes against one.

A 2023 study found models can be highly receptive to outside evidence that conflicts with what they learned, if the evidence is coherent. A well-written stale page is coherent, so it is likely to be believed.

A RAG benchmark paper tested four abilities and found models struggle with information integration and with false information in retrieved text. A 2025 paper on conflicting search results reached a similar view: models often fail to resolve conflicts between sources.

An experience report on three RAG case studies listed seven failure points. One is missing content: the answer is not in the documents. Another is a model failing to extract the answer when the context holds noise or contradictory information.

None of the 24 settings fixed all three questions. In our toy run on 10 made-up pages, the best setting gave only the correct fact for 2 of 3 questions. The worst managed none.

With no tuning and top-3, every question pulled in mixed facts. Fetching only the top 1 did not help. For returns, the top match was a shipping FAQ with no return window, because it shared the question words how and do.

A recency boost lowers a page's score for each year since its review. With top-3 or fewer, a mild boost fixed returns, and stronger boosts also fixed delivery. No boost fixed cash on delivery, where two pages say Rs 5,000 and Rs 10,000.

Both pages were reviewed within the last year. In our made-up truth, Rs 10,000 is right and the newer checkout page copied an old number. The strongest boost served only Rs 5,000, confidently wrong.

Then we left the retriever alone and fixed the sources. Archiving the three stale pages gave 2 of 3 in all 24 settings. Correcting the one wrong line on the checkout page gave 3 of 3 in all 24.

Our retriever was TF-IDF, a simple word-match score, not an embedding model, so treat the exact numbers as a toy. The lesson is about ranking in general: no score can prefer the truth when the truth is not what it measures.

Give every page a last-reviewed date and a review interval, then flag pages whose last review is older than the interval. A review interval is how often the page must be checked, such as every 90 days.

Use the review date, not the last-edit date. A typo fix updates the edit date without anyone checking the facts. A review means an owner read the page and confirmed it is still true.

Our audit script reads four fields from the top of each page: title, owner, last reviewed and review interval. In our toy run on 29 October 2026, it flagged 6 of 10 pages.

• Two pages had no owner, including the 2024 returns page.
• Five pages were past their review interval, from 16 days to 984 days overdue.
• One page, a team offsite plan, had no review interval at all. It should not be in a support bot's sources.

Treat the flag as a prompt to review, not a verdict. The delivery-times page was 16 days overdue but correct. The checkout page passed the date check yet held the wrong cash-on-delivery limit.

Pick intervals by how fast the facts change. Our suggestion, not a standard: 90 days for prices, limits and delivery times, 180 days for policies, 365 days for stable how-to pages.

Beginners can stop here: you now know why bad sources beat tuning, and the first check to run. The rest covers duplicates, contradictions, ownership and the full process.

Compare pages by the short word sequences they share, and flag pairs that share most of them. Exact duplicates are easy to catch with a hash, a short fingerprint of the text; the costly ones are copies with small edits.

Shingles, Jaccard and MinHash in plain words

A shingle is a run of a few words in a row; we used three. Jaccard similarity is the number of shingles two pages share, divided by the number of distinct shingles across both.

Comparing every pair of pages gets slow as the wiki grows. MinHash shrinks each page into a short signature whose overlap estimates Jaccard similarity. LSH, locality-sensitive hashing, then buckets signatures so only likely pairs get compared.

The datasketch library documents both. Its LSH page warns that results are approximate, with false positives and false negatives, and suggests verifying candidates when the threshold must be exact. Researchers have used MinHash to deduplicate large training sets of text.

import re
from datasketch import MinHash, MinHashLSH

def shingles(text, k=3):
    words = re.findall(r"[a-z0-9]+", text.lower())
    return {" ".join(words[i:i + k]) for i in range(len(words) - k + 1)}

def signature(text):
    m = MinHash(num_perm=128, seed=1)
    for s in shingles(text):
        m.update(s.encode("utf8"))
    return m

docs = {
    "new": "You can return most items within 10 days of delivery. Start the return from the Orders page.",
    "old": "You can return most items within 30 days of delivery. Start the return from the Orders page or call the support line.",
    "cod": "Cash on delivery is available for orders up to Rs 10,000.",
}
lsh = MinHashLSH(threshold=0.5, num_perm=128)
sigs = {name: signature(text) for name, text in docs.items()}
for name, sig in sigs.items():
    lsh.insert(name, sig)

pairs = set()
for name, sig in sigs.items():
    for other in lsh.query(sig):
        if other != name:
            pairs.add(tuple(sorted((name, other))))

for x, y in sorted(pairs):
    a, b = shingles(docs[x]), shingles(docs[y])
    print(x, y, "estimate", round(sigs[x].jaccard(sigs[y]), 2),
          "exact", round(len(a & b) / len(a | b), 2))

As of 29 October 2026, datasketch 2.0.0 is the newest release. Run with it on Python 3.13, this prints one line: new old estimate 0.52 exact 0.52. The cash-on-delivery text shares no three-word shingle with the others, so it was not paired.

What our toy run found

On the full toy set, LSH with a 0.5 threshold and 128 permutations, or hash functions, returned three pairs. They were the current returns page, the 2024 page and the sales copy.

The second row matters most. The current page and its stale version are near-duplicates that differ mainly in one number, 10 against 30. A dedup step that keeps "one of them" at random can keep the wrong one.

So never auto-delete a duplicate group. Send the group to the owner, who marks one page as the source of truth and archives or redirects the rest.

What near-duplicate detection misses

The 2023 shipping FAQ said delivery takes 5 to 7 working days; the current page says 3 to 5. It was rewritten rather than copied, so their exact Jaccard was only 0.068. No similarity check catches that. The fact check below does.

MinHash also depends on a seed, the starting number for its random hash functions. Across 50 different seeds, 43 returned exactly the three true pairs. Five missed at least one true pair, and two added a false pair with an exact Jaccard of only 0.19. Verify every candidate pair with exact Jaccard, as the docs suggest.

List the facts your bot must never get wrong, write a pattern that pulls each fact's value out of a page, and flag any fact with more than one value. We call that list a fact registry.

Start with numbers: limits, prices, deadlines, durations and version numbers. They are easy to extract and costly to get wrong. Our toy registry had four facts, each a regular expression, a text pattern matcher.

The warranty row shows why a human must triage. A pattern sees two numbers; only the owner knows that one is a scoped exception. Fix it by making the exception explicit on the general page, so the bot sees both rules together.

Another option is asking a model whether two passages contradict each other. That can catch claims without numbers, but we did not test it here. Keep the regular-expression registry for facts you cannot afford to miss.

A team, not a person, should own every page, because people change roles and their pages go stale quietly. The owner field is what turns an audit flag into a ticket someone must close.

Archive instead of deleting when you can. Archived pages should leave the index but stay readable to staff, which keeps history for audits and stops the same page being recreated.

Copies are one way stale facts multiply, as the sales copy showed. Ask teams to link to the main page rather than paste it. If a team needs its own wording, it should take ownership of that page and its review date.

Start small: one source collection, one audit script and one weekly triage meeting. The steps below take a team from a messy wiki to a routine.

1. 1

   List what the bot reads

   Export every source path the index pulls from. Remove whole folders that should never be answered from, such as event plans and meeting notes.

2. 2

   Add the metadata fields

   Give each page an owner, a last-reviewed date and a review interval. Pages with no willing owner go to an archive folder.

3. 3

   Run the audit

   Run the stale check, the near-duplicate check and the fact registry. Expect a long first list; our toy set flagged 6 of 10 pages.

4. 4

   Triage with owners

   For each flag choose correct, merge, archive or keep. Duplicate groups name one source-of-truth page, the one everyone else links to; the rest are archived or redirected.

5. 5

   Re-sync the index

   Make sure edits are re-embedded and archived pages are removed from the index. Our guide on RAG index freshness covers this sync job.

6. 6

   Check with an evaluation set

   Keep fixed test questions whose right answers come from the fact registry. Run them after each clean-up and after each retriever change.

7. 7

   Gate new pages

   Require the metadata fields before a new page joins the sources, and run the audit on a schedule, such as weekly.

To check that it worked, re-run the audit and the evaluation questions. In our toy run, after archiving three pages and fixing one line, the fact check showed one value per fact except the warranty exception. All three questions were then right in all 24 settings.

Look at the chunks the retriever returned for a wrong answer. If they contain the wrong fact, the source is the problem; if the right page exists but was not returned, retrieval or chunking is.

Our guide on RAG evaluation metrics covers measuring retrieval and answer quality. Our guide on RAG chunking strategies covers the fourth row.

The experience report on RAG failure points also argues that a RAG system's robustness evolves during operation rather than being designed in at the start. Content ops is how that work reaches the documents.

Treat the documents as a data source with quality rules, not as a static input. Interviewers want to hear who keeps the data correct and how you would know it went wrong.

1. 1

   Name the failure

   Say that a wrong answer can quote a real page that is stale, duplicated or conflicting, and that ranking cannot judge truth.

2. 2

   Add ownership metadata

   Every source page carries an owner team, a last-reviewed date, a review interval and a status of current or archived.

3. 3

   Describe the audit job

   A scheduled job flags overdue reviews, near-duplicate groups with MinHash and LSH, and fact conflicts from a fact registry.

4. 4

   Route flags to owners

   Flags become tickets for owner teams. Owners correct, merge or archive; nothing is deleted automatically.

5. 5

   Close the loop

   Re-sync the index, then run a fixed evaluation set whose answers come from the fact registry. Track wrong answers by layer.

6. 6

   State the trade-offs

   Review work costs people time, and LSH misses some pairs. Mention verifying candidates with exact similarity and starting with the facts that cause complaints.