{"id":"c7c7689b-db79-45ed-8d78-135fd5fe3d8f","arxiv_id":"2608.09393","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Version-conditioned retrieval over a 32,436-version French tax code corpus reaches 98.3% strict accuracy on 209 temporal-reasoning questions where LLM-only and static RAG score about 3%.","lead":"A new benchmark measures how often legal question-answering systems cite the current version of a French tax article when an older version is the correct answer. Across eleven models, static retrieval never found the right version, while a date-aware versioned retriever answered 98.3% of the scored questions correctly.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Most load-bearing concern: Appendix D's contamination claim for Cprod is asserted but not released; 28 scored questions on four articles overlap the encoder's fine-tuning pairs, so the 98.3% strict headline depends on an unaudited check that no gold values leaked.","rationale":"I read the paper as a careful, internally consistent benchmark contribution. The deterministic nugget scoring, the explicit disclosure that Conditions A and B are partly construction checks, the cluster-aware statistics, and the honest statement that Cprod is proprietary all speak to good methodology. The strongest empirical claim is the 98.3% end-to-end result, and the least secure support for that claim is the unaudited contamination assertion in Appendix D. The reader identified exactly this as the weakest assumption, and I agree: the check is the single condition that, if wrong, would change the interpretation of the headline from 'version-conditioned retrieval works' to 'a retriever with benchmark overlap retrieves memorized articles.' Because the paper itself flags the overlap and states the check is only on request, this is not manufactured; it is the most load-bearing soft spot. My concrete test would settle it: release the check and ablate the 28 affected questions. If the result survives, no revision is needed; if it does not, the paper should report the uncontaminated subset as the primary number. The current conditional verdict remains appropriate, so I recommend no change.","tokens_in":18014,"tokens_out":4445,"duration_ms":49688,"concrete_test":"Run two checks. (1) Release the re-runnable contamination script from Appendix D, together with fine-tuning corpus hashes, and confirm for each of the 28 scored question IDs on the four overlapping articles that the gold value string does not appear in any training passage. (2) Ablate the benchmark: recompute pooled and per-model Cprod strict and provenance on the 181 questions that do not touch the four overlapping articles, and compare against the full-set 98.3% with cluster-bootstrap CIs. If the uncontaminated subset stays within about one point (e.g., strict above 97% with overlapping CI), the headline holds; if it drops materially below 95%, the reported 98.3% is partly a training-overlap artifact and the paper should report both numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 7.2's headline result (Cprod 98.3% mean strict, provenance 99%) rests on the end-to-end retriever locating both article and date-applicable version without an oracle. The only disclosed overlap between the retriever's training data and the benchmark is in Appendix D: the encoder's fine-tuning pairs overlap the benchmark on five released articles (art. 150 U is flagged out of scoring scope), leaving 28 affected scored questions on four of the 33 scored articles. The paper asserts that zero of those questions' gold values appear in any training passage because the pairs hold current-version texts, but the check and its re-runnable script are 'available from the authors on request,' not in the released repository. If that assertion is wrong, or if the current-version pairs still teach article-level associations for those four articles, then part of the 98.3% could reflect memorized retrieval rather than a general version-conditioning method. The size is material: 28 of 209 scored questions (13.4%) sit on the overlapping articles, and the Cor-to-Cprod gap is only 0.8 points, so even a modest contamination effect on those clusters could close the only gap that demonstrates first-stage recall is the bottleneck. This is not an internal contradiction; the paper is unusually explicit about its construction checks. But the central claim's empirical content is not independently auditable until the check is released or the affected questions are ablated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies temporal misgrounding in legal RAG: retrieval and citation of the currently in-force version of a legal article when the question requires an earlier or future version. It introduces FiscalQA Pro, a versioned corpus of 32,436 CGI/LPF article-versions spanning 1938-2031, and an R3 temporal-reasoning benchmark of 209 scored, expert-reviewed, all-model-hard questions across 33 CGI articles, with deterministic nugget-based scoring. A controlled three-condition experiment across eleven models reports 3.0% mean strict accuracy for LLM-only (Condition A), 2.7% for static-corpus RAG (Condition B), 99.1% for oracle version selection (Cor), and 98.3% for the end-to-end versioned retriever Cprod, with the residual gap attributed to first-stage article recall rather than version selection. The paper releases the corpus, benchmark, model responses, and pipeline code; the Cprod retriever itself and the encoder contamination check are not released.","tokens_in":18336,"tokens_out":5307,"duration_ms":55735,"significance":"If the results hold, this is a valuable contribution: it names and characterizes a real failure mode, provides a reproducible versioned legal corpus and benchmark, avoids LLM-as-judge scoring through deterministic nuggets, and uses an oracle-article ablation to isolate version selection from article retrieval. The internal controls are notably careful: value-only coverage tracks strict accuracy, the divergence and all-model-hard filters are explicitly labeled as construction checks, cluster bootstrap by article is appropriate, and the paper is unusually transparent about what is and is not released. The central empirical claim, however, depends on an unaudited contamination assertion for the Cprod encoder, and the end-to-end retriever is not independently reproducible from the released artifacts.","major_comments":[{"comment":"The 98.3% mean strict result for Cprod is not independently auditable as reported. Appendix D discloses that the encoder's fine-tuning pairs overlap the benchmark on four scored articles (28 of 209 scored questions, 13.4%) and asserts that zero gold values appear in any training passage because the pairs hold current-version texts, but the check and its re-runnable script are only 'available from the authors on request.' Because the Cor-to-Cprod gap is only 0.8 percentage points, even a small number of leaked gold values on those clusters could erase the only evidence that first-stage recall, rather than version selection, is the residual bottleneck. Please release the value-level contamination check or a reconstruction script, and additionally report the Cprod results after ablating the 28 affected scored questions.","section":"Appendix D; §7.2, Table 1"},{"comment":"The paper states that Cprod scores are independently re-verifiable with the deterministic scorer, but the strict score can be rechecked only from model responses and nuggets, while the provenance score (gold version in retrieved top-5) requires the retrieved top-5 lists, which are not listed among the released artifacts. The provenance claim of 99% and the conclusion that the residual gap is a recall@5 ceiling therefore cannot be re-derived from the repository. Please release the per-question retrieved top-5 lists (or the retriever's index and inference code) so that both strict and provenance scores are reproducible from the released materials.","section":"Appendix D; §5.2, §7.2"}],"minor_comments":[{"comment":"The counting of frontier models is confusing: the abstract says 'five frontier closed-API systems plus Gemini 2.5 Pro as a substitute entry,' and the total is eleven, but Table 1 lists six rows under the 'Frontier (closed API)' block, including Gemini; please make the block labels and the counting convention consistent.","section":"Abstract; Table 1 caption; §6.1"},{"comment":"When reporting the sole provenance miss, the paper says 'art. 1417 (2 questions...)' without stating the denominator; please state that this is 2 of 209 scored questions, which is what makes the provenance figure 99%.","section":"§7.2"},{"comment":"The divergence filter is described as a 'regex check' for whether the gold value appears 'verbatim' in the current text, but numeric nuggets are scored with normalization and tolerance; please specify the exact normalization used by the filter's regex and state whether the 208/209 count is robust under the same normalization used for scoring.","section":"§5.3"},{"comment":"The canary GUID is a good practice, but the reproducibility section would be strengthened by documenting the construction of the encoder's fine-tuning pairs (number of pairs, sampling procedure, and the exact overlap list of articles and questions) so that future contamination audits can be conducted without emailing the authors.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"This is a well-designed benchmark paper with strong internal controls and unusually honest construction-check labeling. The gating issue is auditability of the Cprod result: the contamination check for the 28 overlapping scored questions is central because the Cor-to-Cprod gap is tiny, and the check is not released. I would support acceptance after the authors release the check or an ablation of the affected questions, and after the per-question provenance artifacts are released. The proprietary retriever is a concern for a benchmark paper, but it would be mitigated by releasing the retrieved top-5 lists and the contamination evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is the first deterministic, versioned statutory QA benchmark for a civil-law jurisdiction, and its core claim holds up on internal evidence: static RAG retrieves the date-applicable version 0% of the time, while version-conditioned retrieval reaches 98.3% strict. The benchmark construction is careful—nugget scoring, cluster-aware statistics, explicit construction checks. Credit where due: the all-model-hard filter, the current-version divergence filter, the oracle/oracle-article ablations, and the candid Appendix D disclosure are all good practice.\n\nThe soft spots are the ones you'd expect from a benchmark paper with a proprietary retriever. The end-to-end Cprod retriever is not released; only its responses are. More importantly, the contamination check that underwrites the headline 98.3% is 'available from the authors on request,' not in the repo. Four of the 33 scored articles overlap the encoder's fine-tuning pairs, affecting 28 of 209 questions. The paper asserts none of those gold values appear in training passages, but that assertion is not auditable. If it's wrong, a meaningful chunk of the Cprod score could be memorization. This is the single load-bearing external assumption. The paper is upfront that the A/B low baselines are partly by construction: the divergence filter screens out questions whose gold value still appears in the current text, and the all-model-hard filter removes questions any model answers parametrically. That makes the headline gap a conditional measurement, not an average-over-all-legal-QA number. They say this in the text, so it's a scope limitation rather than a hidden flaw.\n\nI don't see an internal contradiction. The central claim—version conditioning closes the temporal misgrounding gap—is supported by the oracle ablations and the control condition on the single non-drifted question. The weak point is external auditability. For peer review, I'd send it out; the benchmark is a real contribution and the methodology is mostly sound. But I'd make acceptance conditional on releasing the contamination check (or ablating the 28 overlapping questions) and on a clearer statement that Cprod is a black-box result until the retriever ships.","headline":"A solid, honestly-scoped benchmark that deserves referee time; the only load-bearing unverified spot is the contamination check for the proprietary retriever.","tokens_in":18845,"tokens_out":1817,"would_cite":true,"duration_ms":18221,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that legal question answering breaks when law changes over time, and that explicit version indexing plus date-conditioned retrieval lifts strict accuracy on temporal tax-law questions from about 3 percent (parametric…","keywords":["temporal misgrounding","legal RAG","versioned corpus","date-conditioned retrieval","French tax law","nugget scoring","temporal reasoning","all-model-hard benchmark"],"falsifier":"Request the encoder's fine-tuning pairs from the authors (Appendix D says they are available on request) and check each of the 28 scored questions on the four overlapping articles for whether any training passage contains that question's gold value; if even one does, the reported 98.3% strict figure is inflated by memorization rather than date-conditioned retrieval.","tokens_in":17787,"feed_emoji":"⚖️","tokens_out":6327,"duration_ms":55778,"temperature":0.7,"pith_summary":"This paper identifies and measures a failure mode it calls temporal misgrounding: legal question-answering systems retrieve and cite the currently in-force version of a statute even when the question asks about an earlier or future version. The authors argue that legal QA is not a static corpus problem but a temporally-indexed retrieval problem, because stable article numbers persist while their content changes yearly. On a new benchmark of 209 expert-reviewed French tax-law questions, no evaluated model answered correctly from parametric knowledge (3.0% mean strict), and RAG over a static current-version corpus did no better (2.7%) while retrieving the date-applicable version 0% of the time. A retriever over a versioned corpus of 32,436 article-versions, conditioning on the query date with no oracle, reached 98.3% mean strict, with an oracle-article ablation at 99.1%. If right, the paper shows the fix is structural, namely version indexing and date-conditioned retrieval, not larger or more recent models.","feed_headline":"Version-aware retrieval lifts legal QA from 3% to 98%","feed_subtitle":"Static RAG retrieves the right version 0% of the time; version indexing reaches 98.3% strict accuracy.","key_machinery":"The load-bearing mechanism is the versioned index with date-conditioned retrieval. Each article-version carries explicit date debut, date fin, and etat fields; the retriever builds a multi-version index of three representative chunks per article identifier (first, median, and last by date debut), fuses dense and BM25 channels by reciprocal-rank fusion, and hands the top-5 articles to a version layer that resolves the version applicable at the query's date before prompting the LLM. Scoring is equally deterministic: atomic nuggets (article regex, numeric-with-tolerance value) are checked by pattern match, never by an LLM judge, which would inherit the temporal bias it is meant to score.","core_discovery":"The paper's central claim is that temporal validity is a first-order axis of legal grounding, and that explicitly conditioning retrieval on the temporal context of the query closes most of the gap between LLM-only performance and version-conditioned performance, even with an end-to-end retriever rather than an oracle. To establish this, the authors construct FiscalQA Pro: a versioned corpus of 32,436 article-versions of the French tax code (CGI and LPF) spanning 1938–2031, and an R3 temporal-reasoning track of 209 scored, expert-reviewed, all-model-hard questions across 33 CGI articles, scored deterministically by atomic ground-truth nuggets (regex article identifiers, numeric values with tolerance) rather than an LLM judge. In a controlled three-condition experiment across eleven models, parametric knowledge yields 3.0% mean strict accuracy, RAG over a static current-version corpus 2.7% (retrieving the date-applicable version 0% of the time and confidently grounding on a real but inapplicable version), while the end-to-end retriever over the multi-version index, with no oracle for article or version, reaches 98.3% mean strict; an oracle-article ablation reaches 99.1%, locating the residual gap in first-stage article recall rather than version selection.","pith_inferences":["Because the article-number nugget fires from the question text itself for 83.7% of the scored questions, the 98.3% figure partly measures value extraction from the right version; queries that omit the article number would likely score lower in practice, a limitation the paper acknowledges only for the retrieval side.","The unreleased contamination check leaves the headline result non-reproducible from the released artifacts; an independent re-run with a retriever trained on data disjoint from the benchmark would settle whether any portion of the 98.3% is memorization rather than date-conditioned retrieval.","The paper's own taxonomy predicts three additional failure modes (future-law leakage, wrong-amendment resolution, multi-version confusion) that the single-anchor R3 set does not exercise; a benchmark including them could show whether the version-layer fix generalizes beyond current-law substitution.","The same architecture could apply to any amended-in-place corpus, such as financial regulation, administrative rulebooks, or product terms, where the correct answer depends on the version in force at a given date, not just to statutory tax law."],"forward_implications":["Deployed legal RAG systems that index only the current version of statutes will systematically give confidently wrong answers on any question whose answer depends on an earlier or future version, and the error is invisible to users because the cited text is real.","Versioning the corpus and conditioning retrieval on the query date is sufficient to lift strict accuracy above 95% for all eleven models tested, making model choice and scale secondary once the right version is served.","Because the failure is structural, larger or more recent LLMs will not fix it parametrically; the data show no within-family generation gain closes the gap.","After versioning, the residual bottleneck is first-stage article recall (the sole provenance miss is art. 1417, two questions), not date resolution, so further gains require better article-level retrieval rather than better reranking.","The same version-layer fix should transfer to other civil-law jurisdictions that expose versioned statutory APIs, such as Swiss Fedlex or German Gesetze-im-Internet, since the corpus properties that cause the failure are structural."],"supporting_citations":[{"why":"Supplies the enterprise grounded-reasoning methodology this paper extends, including deterministic scoring and the zero-knowledge parametric filter.","marker":"(Opsahl-Ong et al., 2026)"},{"why":"Source of the nugget-decomposition scoring concept that FiscalQA Pro adapts into deterministic, judge-free checks.","marker":"(Databricks AI Research, 2026)"},{"why":"LegalBench provides the benchmark context and scale reference the R3 track is deliberately compared against.","marker":"(Guha et al., 2023)"},{"why":"TimeQA evidences that LLMs default to the most recent fact seen in training, supporting the recency-bias root cause.","marker":"(Chen et al., 2021)"},{"why":"TempLAMA documents the same recency-default behavior in temporal question answering, grounding cause (1) of temporal misgrounding.","marker":"(Dhingra et al., 2022)"},{"why":"A concurrent German statutory QA study finding the same temporal failure is cited as independent evidence of generalization beyond French tax law.","marker":"(Prior et al., 2026)"},{"why":"SAT-Graph RAG is the contrasted point-in-time legal query approach that requires an ontology, whereas this paper's method needs only version indexing.","marker":"(de Martim, 2025)"}],"fun_headline_variants":["Legal RAG jumps from 3% to 98% with time-aware retrieval","Static RAG never finds the right law; version indexing fixes that","Temporal misgrounding: legal RAG fails when laws change","FiscalQA Pro: versioned tax benchmark exposes static RAG's blind spot","From 2.7% to 98.3%: legal QA demands time-aware retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline 98.3% accuracy rests on an unreleased contamination check asserting that none of the 28 affected scored questions' gold values appear in the retriever's training data; if that assertion is wrong, a share of the reported accuracy could be memorization rather than genuinely date-conditioned retrieval.","fun_headline_variants_meta":{"raw":{"variants":["Legal RAG jumps from 3% to 98% with time-aware retrieval","Static RAG never finds the right law; version indexing fixes that","Temporal misgrounding: legal RAG fails when laws change","FiscalQA Pro: versioned tax benchmark exposes static RAG's blind spot","From 2.7% to 98.3%: legal QA demands time-aware retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000574,"raw_usage":{"total_tokens":2829,"prompt_tokens":1181,"completion_tokens":1648,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":797,"completion_tokens_details":{"reasoning_tokens":1545}},"tokens_in":797,"tokens_out":1648,"duration_ms":14117,"temperature":1.0,"reasoning_tokens":1545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:05:54.003483+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Request the encoder's fine-tuning pairs from the authors (Appendix D says they are available on request) and check each of the 28 scored questions on the four overlapping articles for whether any training passage contains that question's gold value; if even one does, the reported 98.3% strict figure is inflated by memorization rather than date-conditioned retrieval.","supporting_citations":[{"cited_title":"E., R \\'e , C., et al","cited_arxiv_id":null,"evidence_quote":"LegalBench provides the benchmark context and scale reference the R3 track is deliberately compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TimeQA evidences that LLMs default to the most recent fact seen in training, supporting the recency-bias root cause."},{"cited_title":"R., Eisenschlos, J","cited_arxiv_id":null,"evidence_quote":"TempLAMA documents the same recency-default behavior in temporal question answering, grounding cause (1) of temporal misgrounding."},{"cited_title":"Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering","cited_arxiv_id":"2605.23497","evidence_quote":"A concurrent German statutory QA study finding the same temporal failure is cited as independent evidence of generalization beyond French tax law."}],"review_version":1}