RAG citation checks tested: why the same query retrieves an expired rule

What the retrieval fixture contains

SampleNote is a fictional service written for this fixture. Standard retention changes from thirty to fourteen to seven days; the seven-day version starts in 2027. Invoices have a separate policy, and a similar rule for another product acts as a distractor. Every record is published in protocol.json; none describes real customers or an internal company policy.

Each of the eight queries supplies a product and an as-of date. Record type and role are also fixture inputs, rather than conditions inferred from natural language. Document intervals include the start date and exclude the end date. The fixture therefore makes the expected historical and future source explicit.

Why an unfiltered query can return an old version

The control splits the English query and passage into tokens, ranks by token overlap, and preserves file order for ties. It uses neither vectors nor policy-validity checks. It returns the old thirty-day policy for the current standard-retention query, while correctly matching the historical query. Simply deleting every old version would also remove evidence needed by historical queries.

The filtered variant removes ineligible records before applying the same token ranking. Standard records do not inherit the invoice exception, and a public query cannot retrieve the staff-only batch limit. The table reports actual outputs for these eight cases. The input order deliberately preserves a failure condition; three out of eight is not an estimate of average performance on real knowledge bases.

A source identifier alone does not validate a citation

We also supplied six candidate citations to a validator. It accepted only the case with an eligible current source, a literal passage and matching controlled fields. Expired sources, altered numbers and nonexistent sources produced explicit rejection reasons. The complete JSON retains both expected and actual decisions; displaying a source link alone does not establish that the content has been checked.

claim_key and claim_value are fields explicitly defined in this fixture. The validator detects changing fourteen days to forty-one and checks whether a quoted substring literally exists, but it does not understand arbitrary paraphrases. Microsoft distinguishes retrieval-process evaluation from generated-answer evaluation. These results cover retrieval and limited citation-field checks.

How to reproduce these checks

Download and extract the complete package, enter rag-citation, and run the command below. The script uses only Python standard-library code and makes no network request. It reruns the eight queries and six citations and saves individual outcomes. Expected control failures do not fail the run; a mismatch in the filtered or citation checks produces a nonzero exit code.

Execution timestamps change when rerun, and Python versions may differ. Compare each expected/actual row rather than demanding an identical result-file hash. The protocol and runner have separate SHA-256 values. Keep a new version when modifying materials, so a new experiment is not mistaken for a repeat of the original fixture.

python3 run_rag_checks.py --output reproduced-results.json

What this small fixture does not establish

A role is only an input string in this program; no identity system is connected. A deployed knowledge base must not let a user declare themselves staff and treat that string as authorization. Microsoft documents identity-aware access checks at query time. This fixture reproduces only filtering on a supplied permission field.

This program does not test synonym retrieval or Chinese tokenization. Whether a model omits the invoice exception or generates an answer without evidence requires a separate test with the actual model. The fixture helps inspect candidate sources and locate a fault, but eight passing retrieval cases do not establish readiness for real business data.