REVIEW 2 major objections 4 minor 36 references
Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law
T0 review · 2 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper shows that legal question answering breaks when law changes over time, and that explicit version indexing plus date-conditioned retrieval lifts strict accuracy on temporal tax-law questions from about 3 percent (parametric…
desk verdict A solid, honestly-scoped benchmark that deserves referee time; the only load-bearing unverified spot is the contamination check for the proprietary retriever. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the versioned index with date-conditioned retrieval. Each article-version carries explicit date debut, date fin, and etat fields; the retriever builds a multi-version index of three representative chunks per article identifier (first, median, and last by date debut), fuses dense and BM25 channels by reciprocal-rank fusion, and hands the top-5 articles to a version layer that resolves the version applicable at the query's date before prompting the LLM. Scoring is equally deterministic: atomic nuggets (article regex, numeric-with-tolerance value) are checked by pattern match, never by an LLM judge, which would inherit the temporal bias it is meant to score.
What would settle it
Request the encoder's fine-tuning pairs from the authors (Appendix D says they are available on request) and check each of the 28 scored questions on the four overlapping articles for whether any training passage contains that question's gold value; if even one does, the reported 98.3% strict figure is inflated by memorization rather than date-conditioned retrieval.
Extended reading notes
Core claim
The paper's central claim is that temporal validity is a first-order axis of legal grounding, and that explicitly conditioning retrieval on the temporal context of the query closes most of the gap between LLM-only performance and version-conditioned performance, even with an end-to-end retriever rather than an oracle. To establish this, the authors construct FiscalQA Pro: a versioned corpus of 32,436 article-versions of the French tax code (CGI and LPF) spanning 1938–2031, and an R3 temporal-reasoning track of 209 scored, expert-reviewed, all-model-hard questions across 33 CGI articles, scored deterministically by atomic ground-truth nuggets (regex article identifiers, numeric values with tolerance) rather than an LLM judge. In a controlled three-condition experiment across eleven models, parametric knowledge yields 3.0% mean strict accuracy, RAG over a static current-version corpus 2.7% (retrieving the date-applicable version 0% of the time and confidently grounding on a real but inapplicable version), while the end-to-end retriever over the multi-version index, with no oracle for article or version, reaches 98.3% mean strict; an oracle-article ablation reaches 99.1%, locating the residual gap in first-stage article recall rather than version selection.
Load-bearing premise
The headline 98.3% accuracy rests on an unreleased contamination check asserting that none of the 28 affected scored questions' gold values appear in the retriever's training data; if that assertion is wrong, a share of the reported accuracy could be memorization rather than genuinely date-conditioned retrieval.
Editorial extensions
If this is right
- Deployed legal RAG systems that index only the current version of statutes will systematically give confidently wrong answers on any question whose answer depends on an earlier or future version, and the error is invisible to users because the cited text is real.
- Versioning the corpus and conditioning retrieval on the query date is sufficient to lift strict accuracy above 95% for all eleven models tested, making model choice and scale secondary once the right version is served.
- Because the failure is structural, larger or more recent LLMs will not fix it parametrically; the data show no within-family generation gain closes the gap.
- After versioning, the residual bottleneck is first-stage article recall (the sole provenance miss is art. 1417, two questions), not date resolution, so further gains require better article-level retrieval rather than better reranking.
- The same version-layer fix should transfer to other civil-law jurisdictions that expose versioned statutory APIs, such as Swiss Fedlex or German Gesetze-im-Internet, since the corpus properties that cause the failure are structural.
Reading between the lines
- Because the article-number nugget fires from the question text itself for 83.7% of the scored questions, the 98.3% figure partly measures value extraction from the right version; queries that omit the article number would likely score lower in practice, a limitation the paper acknowledges only for the retrieval side.
- The unreleased contamination check leaves the headline result non-reproducible from the released artifacts; an independent re-run with a retriever trained on data disjoint from the benchmark would settle whether any portion of the 98.3% is memorization rather than date-conditioned retrieval.
- The paper's own taxonomy predicts three additional failure modes (future-law leakage, wrong-amendment resolution, multi-version confusion) that the single-anchor R3 set does not exercise; a benchmark including them could show whether the version-layer fix generalizes beyond current-law substitution.
- The same architecture could apply to any amended-in-place corpus, such as financial regulation, administrative rulebooks, or product terms, where the correct answer depends on the version in force at a given date, not just to statutory tax law.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper identifies temporal misgrounding in legal RAG: retrieval and citation of the currently in-force version of a legal article when the question requires an earlier or future version. It introduces FiscalQA Pro, a versioned corpus of 32,436 CGI/LPF article-versions spanning 1938-2031, and an R3 temporal-reasoning benchmark of 209 scored, expert-reviewed, all-model-hard questions across 33 CGI articles, with deterministic nugget-based scoring. A controlled three-condition experiment across eleven models reports 3.0% mean strict accuracy for LLM-only (Condition A), 2.7% for static-corpus RAG (Condition B), 99.1% for oracle version selection (Cor), and 98.3% for the end-to-end versioned retriever Cprod, with the residual gap attributed to first-stage article recall rather than version selection. The paper releases the corpus, benchmark, model responses, and pipeline code; the Cprod retriever itself and the encoder contamination check are not released.
Significance. If the results hold, this is a valuable contribution: it names and characterizes a real failure mode, provides a reproducible versioned legal corpus and benchmark, avoids LLM-as-judge scoring through deterministic nuggets, and uses an oracle-article ablation to isolate version selection from article retrieval. The internal controls are notably careful: value-only coverage tracks strict accuracy, the divergence and all-model-hard filters are explicitly labeled as construction checks, cluster bootstrap by article is appropriate, and the paper is unusually transparent about what is and is not released. The central empirical claim, however, depends on an unaudited contamination assertion for the Cprod encoder, and the end-to-end retriever is not independently reproducible from the released artifacts.
major comments (2)
- [Appendix D; §7.2, Table 1] The 98.3% mean strict result for Cprod is not independently auditable as reported. Appendix D discloses that the encoder's fine-tuning pairs overlap the benchmark on four scored articles (28 of 209 scored questions, 13.4%) and asserts that zero gold values appear in any training passage because the pairs hold current-version texts, but the check and its re-runnable script are only 'available from the authors on request.' Because the Cor-to-Cprod gap is only 0.8 percentage points, even a small number of leaked gold values on those clusters could erase the only evidence that first-stage recall, rather than version selection, is the residual bottleneck. Please release the value-level contamination check or a reconstruction script, and additionally report the Cprod results after ablating the 28 affected scored questions.
- [Appendix D; §5.2, §7.2] The paper states that Cprod scores are independently re-verifiable with the deterministic scorer, but the strict score can be rechecked only from model responses and nuggets, while the provenance score (gold version in retrieved top-5) requires the retrieved top-5 lists, which are not listed among the released artifacts. The provenance claim of 99% and the conclusion that the residual gap is a recall@5 ceiling therefore cannot be re-derived from the repository. Please release the per-question retrieved top-5 lists (or the retriever's index and inference code) so that both strict and provenance scores are reproducible from the released materials.
minor comments (4)
- [Abstract; Table 1 caption; §6.1] The counting of frontier models is confusing: the abstract says 'five frontier closed-API systems plus Gemini 2.5 Pro as a substitute entry,' and the total is eleven, but Table 1 lists six rows under the 'Frontier (closed API)' block, including Gemini; please make the block labels and the counting convention consistent.
- [§7.2] When reporting the sole provenance miss, the paper says 'art. 1417 (2 questions...)' without stating the denominator; please state that this is 2 of 209 scored questions, which is what makes the provenance figure 99%.
- [§5.3] The divergence filter is described as a 'regex check' for whether the gold value appears 'verbatim' in the current text, but numeric nuggets are scored with normalization and tolerance; please specify the exact normalization used by the filter's regex and state whether the 208/209 count is robust under the same normalization used for scoring.
- [Appendix D] The canary GUID is a good practice, but the reproducibility section would be strengthened by documenting the construction of the encoder's fine-tuning pairs (number of pairs, sampling procedure, and the exact overlap list of articles and questions) so that future contamination audits can be conducted without emailing the authors.
Circularity Check
The low A and B accuracy numbers are partly selected into existence by the paper's own filters; the Cprod result is not shown to be circular, but its headline rests on an unreleased contamination check.
-
self definitional
[§5.3 (Parametric Knowledge Filter) and §7.2 (Controlled Experiment Results)]
"each candidate question is put to every evaluated LLM without corpus or web access, and any question whose gold value is stated in at least one of four sampling draws by any model is dropped... On the k=209 all-model-hard subset, parametric knowledge (A) is uniformly low across all eleven models (3.0% mean strict...), by construction of the filter"
The retained R3 set is defined by the filter: a question survives only if no evaluated model produced its gold value in any of four closed-book draws. Condition A's near-zero strict score is therefore a property of the inclusion rule, not an independent measurement on a random sample; the paper's own §7.2 calls it 'by construction of the filter.' The abstract's 'parametric knowledge yields 3.0% mean strict accuracy' thus restates the selection criterion (plus evaluation-time sampling drift), rather than testing the temporal-misgrounding hypothesis against an unfiltered question distribution.
-
self definitional
[§5.3 (Current-version divergence filter), §6 H2, and §7.2]
"a regex check tests whether the gold date-anchored value still appears verbatim in the currently in-force version of the article, and candidates whose value is unchanged are dropped... We state the consequence explicitly: Condition B fails partly by construction: the retained questions are screened so that the current-version text does not contain the gold value."
Condition B is evaluated on a subset from which every question whose gold value survives in the current-version text has been removed. Its strict accuracy is therefore forced to be near zero by construction, with only the single well-formedness control able to pass. The abstract's 'RAG over a static current-version corpus 2.7%' is thus partly an artifact of the divergence filter. The paper honestly reformulates H2 so the falsifiable content is the 0% provenance and the <10% ceiling, but the low strict score qua prediction reduces to the screening rule.
full rationale
The paper is unusually transparent: it repeatedly labels the low Condition A and Condition B scores as construction checks. Even so, two headline results are partly selected into existence by the benchmark's own filters. The all-model-hard filter drops any question any evaluated model answered closed-book in four sampling draws, so the 3.0% parametric-knowledge score is a consequence of the inclusion rule; the current-version divergence filter drops any question whose gold value still appears in the in-force text, so the 2.7% static-RAG strict score is partly forced. Those are the two circular steps counted here. The 0% provenance for static RAG is not circular, because a current-only index structurally cannot contain a historical version, and the Cprod result is not shown to be circular: the end-to-end retriever must locate the article without an oracle, the version layer then resolves the date-applicable version, and the paper reports internal controls (the Cor oracle-article ablation, leave-one-article-out, and cluster bootstrap CIs) that do not depend on the A/B filters. The genuine weak point is Appendix D: the encoder's fine-tuning pairs overlap four scored articles (28 scored questions), and the assertion that zero gold values leaked into training is only 'available from the authors on request', not released. That is an un-audited input and a contamination/correctness risk, not a demonstrated circularity, because the paper asserts the pairs hold current-version texts. Combined with the disclosed article-nugget leakage (the article regex fires on the prompt for 175 of 209 questions), the 98.3% headline should be read as conditional on that unreleased check. Score 6: one or more headline predictions reduce by construction, while the central versioned-retrieval derivation retains substantial independent content.
Assumptions & free parameters
free parameters (3)
- numeric tolerance for value nuggets =
not stated in paper, set by authors
- top-5 retrieval k for Cprod provenance =
5
- 8,000-char query-windowed budget =
8000 characters
assumptions (4)
- domain assumption The Légifrance PISTE API version history is complete and authoritative for all CGI/LPF article versions.
- ad hoc to paper The 209 scored questions and their gold nuggets are correct and representative of temporal drift in French tax law.
- domain assumption Regex and numeric-with-tolerance matching capture legal correctness without an LLM judge.
- ad hoc to paper The all-model-hard filter and divergence filter accurately identify questions that are not answerable from parametric knowledge or current-version text.
Cite this review
Pith. "Pith review of Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law." pith.science (2026). https://pith.science/paper/GQ6HIQ2Q
@misc{pith2026260809393,
author = {Pith},
title = {Pith review of: Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law},
year = {2026},
howpublished = {\url{https://pith.science/paper/GQ6HIQ2Q}},
note = {Machine review of arXiv:2608.09393}
}
read the original abstract
We identify and quantify temporal misgrounding: the systematic retrieval and citation of the currently in-force version of a legal article when the applicable version is an earlier or future one. Standard legal RAG treats the corpus as static; we argue legal question answering is a temporally-indexed retrieval problem. We introduce FiscalQA Pro, pairing a versioned corpus of 32,436 article-versions of the French tax code (93 years, 1938-2031) with an all-model-hard temporal-reasoning track: 209 scored, expert-reviewed questions across 33 CGI articles (221 released; twelve flagged out of the answerable scope). At selection time, no evaluated model recovered its date-applicable answer closed-book in any of four sampling draws, and the currently in-force text lacks the gold value for all but one of the scored questions. Answers are scored deterministically via atomic ground-truth "nuggets" (regex and numeric-with-tolerance), never LLM-as-judge: an LLM judge would inherit the temporal bias it is meant to score. Across eleven models (five frontier closed-API systems plus Gemini 2.5 Pro as a substitute entry, and five open-weight), parametric knowledge yields 3.0% mean strict accuracy and RAG over a static current-version corpus 2.7%. Static RAG retrieves the date-applicable version 0% of the time, confidently citing a real but inapplicable version. Our end-to-end retriever over a multi-version index, with no oracle, reaches 98.3% mean strict; an oracle-article ablation reaches 99.1%, locating the residual gap in first-stage recall, not version selection. We additionally release a version-aware jurisprudence dataset of 69,208 citation links, together with the corpus, benchmark, model responses, and pipeline code.
Figures
Reference graph
Works this paper leans on
-
[1]
LexTime : A benchmark for temporal ordering of legal events
Barale, C. LexTime : A benchmark for temporal ordering of legal events. In Findings of EMNLP, 2025. URL https://arxiv.org/abs/2506.04041
-
[2]
Justel : Consolidated Belgian legislation
Belgian Federal Public Service Justice . Justel : Consolidated Belgian legislation. URL https://www.ejustice.just.fgov.be/. Accessed July 2026
work page 2026
-
[3]
Chen, W., Wang, X., and Wang, W. Y. TimeQA : A benchmark for time-sensitive question answering. In NeurIPS Datasets and Benchmarks Track, 2021
work page 2021
-
[4]
Fine-grained Claim-level RAG Benchmark for Law
Das, S., Abualhaija, S., and Bianculli, D. Fine-grained claim-level RAG benchmark for law. arXiv preprint arXiv:2605.21071, 2026
work page Pith review arXiv 2026
-
[7]
Dhingra, B., Cole, J. R., Eisenschlos, J. M., Gillick, D., Eisenstein, J., and Cohen, W. W. TempLAMA : Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 2022
work page 2022
-
[8]
JuriBERT : A masked-language model adaptation for French legal text
Douka, S., Abdine, H., Vazirgiannis, M., El Hamdani, R., and Restrepo Amariles, D. JuriBERT : A masked-language model adaptation for French legal text. In Proceedings of the Natural Legal Language Processing Workshop, 2021
work page 2021
-
[9]
Fan, W., Zhou, Y., Zhang, M., Weng, Y., Hu, Y., Zheng, T., Xu, B., Li, C., Yang, J., Li, H., and Song, Y. Can LLMs time travel? Enhancing temporal consistency in legal agentic search through reinforcement learning. arXiv preprint arXiv:2605.25920, 2026 a
work page Pith review arXiv 2026
-
[10]
LEXam : Benchmarking legal reasoning on 340 law exams
Fan, Y., Ni, J., Huang, Y., Tian, Y., Stammbach, D., Ash, E., Engel, C., et al. LEXam : Benchmarking legal reasoning on 340 law exams. In International Conference on Learning Representations (ICLR), 2026 b . URL https://arxiv.org/abs/2505.12864
arXiv 2026
Show all 36 references
-
[11]
Gesetze im Internet : German federal law online
German Federal Ministry of Justice . Gesetze im Internet : German federal law online. URL https://www.gesetze-im-internet.de/. Accessed July 2026
2026
-
[12]
L\'egilux : The official journal of the Grand Duchy of Luxembourg
Government of Luxembourg . L\'egilux : The official journal of the Grand Duchy of Luxembourg . URL https://legilux.public.lu/. Accessed July 2026
2026
-
[13]
E., R \'e , C., et al
Guha, N., Nyarko, J., Ho, D. E., R \'e , C., et al. LegalBench : A collaboratively built benchmark for measuring legal reasoning in large language models. In NeurIPS Datasets and Benchmarks Track, 2023
2023
-
[14]
LEXTREME : A multi-lingual and multi-task benchmark for the legal domain
Niklaus, J., Matoshi, V., Rani, P., Galassi, A., St \"u rmer, M., and Chalkidis, I. LEXTREME : A multi-lingual and multi-task benchmark for the legal domain. In Findings of EMNLP, 2023
2023
-
[15]
OfficeQA Pro : An enterprise benchmark for end-to-end grounded reasoning
Opsahl-Ong, K., Singhvi, A., Collins, J., Zhou, I., Wang, C., Baheti, A., Oertell, O., Portes, J., Havens, S., Elsen, E., Bendersky, M., Zaharia, M., and Chen, X. OfficeQA Pro : An enterprise benchmark for end-to-end grounded reasoning. arXiv preprint arXiv:2603.08655, 2026. U...
2026
-
[17]
Asking for an old friend: Diagnosing and mitigating temporal failure modes in LLM -based statutory question answering
Prior, M., Schultz, A., and Grabmair, M. Asking for an old friend: Diagnosing and mitigating temporal failure modes in LLM -based statutory question answering. arXiv preprint arXiv:2605.23497, 2026
2026 arXiv
-
[18]
Fedlex : The publication platform for Swiss federal law
Swiss Federal Chancellery . Fedlex : The publication platform for Swiss federal law. URL https://www.fedlex.admin.ch/. Accessed July 2026
2026
-
[19]
FreshLLMs : Refreshing large language models with search engine augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., et al. FreshLLMs : Refreshing large language models with search engine augmentation. arXiv preprint arXiv:2310.03214, 2023
2023 arXiv
-
[20]
2026 , url =
Opsahl-Ong, Krista and Singhvi, Arnav and Collins, Jasmine and Zhou, Ivan and Wang, Cindy and Baheti, Ashutosh and Oertell, Owen and Portes, Jacob and Havens, Sam and Elsen, Erich and Bendersky, Michael and Zaharia, Matei and Chen, Xing , journal =. 2026 , url =
2026
-
[21]
arXiv preprint arXiv:2603.05218 , year =
-
[22]
NeurIPS Datasets and Benchmarks Track , year =
Guha, Neel and Nyarko, Julian and Ho, Daniel E.\ and R. NeurIPS Datasets and Benchmarks Track , year =
-
[23]
Findings of EMNLP , year =
Niklaus, Joel and Matoshi, Veton and Rani, Pooja and Galassi, Andrea and St. Findings of EMNLP , year =
-
[24]
2026 , url =
Fan, Yu and Ni, Jingwei and Huang, Yujie and Tian, Yang and Stammbach, Dominik and Ash, Elliott and Engel, Christoph and others , booktitle =. 2026 , url =
2026
-
[25]
Douka, Stella and Abdine, Hadi and Vazirgiannis, Michalis and El Hamdani, Rajaa and Restrepo Amariles, David , booktitle =
-
[26]
arXiv preprint arXiv:2505.03970 , year =
Reasoning-Focused Legal Retrieval , author =. arXiv preprint arXiv:2505.03970 , year =
-
[27]
arXiv preprint arXiv:2502.13660 , year =
Conditioning Large Language Models on Legal Systems , author =. arXiv preprint arXiv:2502.13660 , year =
-
[28]
arXiv preprint arXiv:2512.24572 , year =
A Korean Canonical Legal Benchmark , author =. arXiv preprint arXiv:2512.24572 , year =
-
[29]
Proceedings of ACL , year =
Niklaus, Joel and Matoshi, Veton and St. Proceedings of ACL , year =
-
[30]
Fine-grained Claim-level
Das, Souvick and Abualhaija, Sallam and Bianculli, Domenico , journal=. Fine-grained Claim-level
-
[31]
, journal =
Dhingra, Bhuwan and Cole, Jeremy R.\ and Eisenschlos, Julian Martin and Gillick, Daniel and Eisenstein, Jacob and Cohen, William W. , journal =
-
[32]
Chen, Wenhu and Wang, Xinyi and Wang, William Yang , booktitle =
-
[33]
Vu, Tu and Iyyer, Mohit and Wang, Xuezhi and Constant, Noah and others , journal =
-
[34]
2025 , url =
Barale, Claire , booktitle =. 2025 , url =
2025
-
[35]
arXiv preprint arXiv:2505.20243 , year=
It's High Time: A Survey of Temporal Question Answering , author=. arXiv preprint arXiv:2505.20243 , year=
-
[36]
Fan, Wei and Zhou, Yining and Zhang, Mufan and Weng, Yanbing and Hu, Yiran and Zheng, Tianshi and Xu, Baixuan and Li, Chunyang and Yang, Jianhui and Li, Haoran and Song, Yangqiu , journal=. Can
-
[37]
Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in
Prior, Max and Schultz, Andreas and Grabmair, Matthias , journal=. Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in
-
[38]
An Ontology-Driven Graph
de Martim, Hudson , booktitle=. An Ontology-Driven Graph. doi:10.3233/FAIA251598 , year=
-
[39]
Proceedings of ACL , year =
Martin, Louis and Muller, Benjamin and Su. Proceedings of ACL , year =
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.