Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

The paper claims that a multi-turn judge-and-search loop, L-MARS, nearly doubles strict citation F1 (0.13 to 0.25) and cuts the no-citation rate from 34% to 13% on a stratified 100-question Bar Exam audit, while barely moving multiple-choic

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

The abstract reports that a judge-driven multi-agent loop improves legal-citation faithfulness from 0.13 to 0.25 strict F1 and cuts the no-citation rate from 34% to 13%, but the provided manuscript text does not contain that audit.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The body is a serviceable multi-agent RAG legal QA paper, but the abstract's advertised citation-faithfulness audit and its headline numbers do not exist in the submitted text. the 4 major comments →

arxiv 2509.00761 v4 pith:HQYW7CTE submitted 2025-08-31 cs.AI cs.CL

L-MARS: Legal Multi-Agent System with Agentic Search and Citation-Faithfulness Audit

Boqin Yuan , Ziqi Wang This is my paper
classification cs.AI cs.CL
keywords multi-agent systemslegal question answeringagentic searchcitation faithfulnessstrict-ALCEretrieval-augmented generationhallucination mitigationLLM-as-judge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

L-MARS is a multi-agent legal question-answering system whose load-bearing claim is about citations, not answer accuracy: on a stratified 100-question Bar Exam audit, its judge-driven search loop doubles strict citation F1 from 0.13 (naive RAG) to 0.25 and cuts the no-citation rate from 34% to 13%, while multiple-choice accuracy barely moves. A final repair step, Faith-Search, re-verifies each cited source and rewrites unreachable references, pushing the unreachable-citation rate below 1%—evidence that citation problems split into 'claim not supported' and 'source cannot be reached' and that the second is easier to fix. The audit crosses model families, with the answerer and verifier drawn from different providers, to reduce shared hallucination. The full text supplied with this record reports a different evaluation (200-question LegalSearchQA, accuracy 0.98, U-Score 0.39) and does not define strict-ALCE, the six-class taxonomy, or Faith-Search; the citation-F1 numbers therefore currently stand on the abstract alone. If true, the result would give legal AI practitioners a measurable target—citation faithfulness—that is distinct from accuracy and closer to what lawyers actually need.

Core claim

On its own terms, the discovery is that a multi-turn judge loop changes where the system fails: it does not improve multiple-choice accuracy much, but it makes citations real. L-MARS runs a Query Agent to parse the question, a Search Agent over web, local BM25, and CourtListener, a Judge Agent that applies a deterministic sufficiency checklist (factual support, jurisdiction, date, contradiction) and issues refinement queries, and a Summary Agent that writes the final cited answer. That loop raises strict citation F1 from 0.13 to 0.25 on a 100-question Bar Exam audit and drops no-citation answers from 34% to 13%. A post-draft step, Faith-Search, re-verifies and rewrites unreachable sources, c

What carries the argument

The mechanism that carries the argument is a three-part loop. (1) Judge Agent: a deterministic (T=0) LLM that runs a chain-of-thought sufficiency checklist—factual support, jurisdiction, temporal specificity, contradiction scan—and either declares SUFFICIENT or returns specific refinement queries; this is what routes the system back to search. (2) Agentic Search: triggered by the judge's gaps, it combines Serper web titles/snippets, deep full-content extraction, a local BM25 index, and CourtListener case law, so citations carry distinct source metadata. (3) Faith-Search: a post-draft procedure that re-verifies each cited URL against the supporting text and rewrites or drops sources that cann

Load-bearing premise

The reported 0.13-to-0.25 citation-F1 gain depends on an audit procedure ('strict-ALCE' with cross-provider judging) whose definition, implementation, and validation do not appear in the full text, so the gain could be an artifact of how the audit was run.

What would settle it

Run the same L-MARS system against the same 100-question stratified Bar Exam audit, computing citation faithfulness with an independently implemented strict-ALCE scorer and with the answerer and judge drawn from the same model family; if the F1 gap between naive RAG and L-MARS shrinks to near zero or flips, the reported improvement is an artifact of the audit rather than the system.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the judge loop is what moves citation F1, then retrieval-heavy pipelines should budget compute for post-retrieval verification, not just more search rounds.
  • Citation faithfulness can be decoupled from accuracy: the audit shows a system can jump from 0.13 to 0.25 F1 while accuracy stays flat, so evaluations that report only accuracy will miss the failure mode lawyers fear most.
  • Because the answerer and verifier come from different model families, the design implies that cross-provider judging is a cheap independence check against shared hallucination; the same principle could transfer to medical or financial QA.
  • Faith-Search's success at cutting unreachable citations below 1% suggests a practical fix for legal citation rot: a final 'link check' pass can remove broken or inaccessible authority without hurting the answer.
  • The 0.13-to-0.25 range on the Bar Exam audit, versus 0.75 saturation on LegalSearchQA, suggests the harder, lower-F1 audits are where judge loops earn their latency.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The full text stored with this record is an earlier LegalSearchQA manuscript: it reports accuracy (0.96–0.98) and U-Score (0.39–0.42) but contains no strict-ALCE definition, no six-class taxonomy, and no Faith-Search procedure. Readers should treat the citation-F1 numbers as abstract-level claims that the appended body does not yet substantiate.
  • If unreachability, not unsupportedness, is the dominant citation failure after a judge loop, then the next testable improvement is an answerer that cites only from a pre-fetched snapshot of the retrieved corpus, eliminating link rot at the source.
  • The cross-provider judge design could be stress-tested by swapping judge families systematically; if F1 gains vanish when judge and answerer share a family, 'independence' is doing the work, not the loop topology.
  • A practical extension: deploy the strict-ALCE audit as a monitor on live legal-assistance traffic, where accuracy labels are unavailable but citation checks can be automated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The arXiv metadata for this submission advertises L-MARS as a legal multi-agent QA system with agentic search and a citation-faithfulness audit: on a stratified 100-question Bar Exam set, each atomic claim is labelled with a six-class taxonomy, scored with strict-ALCE under cross-provider judging, and the multi-turn judge loop raises strict citation F1 from 0.13 to 0.25 while cutting the no-citation rate from 34% to 13%; Faith-Search reduces unreachable citations below 1%. The full-text manuscript, however, presents a different system description and evaluation. It introduces L-MARS as a LangGraph workflow with Query, Search, Judge, and Summary agents and evaluates it on LegalSearchQA, a new 200-question multiple-choice benchmark, reporting accuracy, a rule-based U-Score, and GPT-o3 LLM-as-Judge preferences (Table 3). The full text contains no Bar Exam audit, no strict-ALCE implementation, no six-class taxonomy, no Faith-Search, and no cross-provider judging protocol. The body's central claims are that L-MARS improves multiple-choice accuracy (from 86–89% to 96–98%) and lowers U-Score (from 0.55–0.62 to 0.39–0.42), with higher latency.

Significance. If the advertised audit were present and methodologically sound, the paper would address an important gap: most legal QA evaluations ignore whether citations exist and support the attributed rules. The reported improvement in strict citation F1 and the cross-provider judging design would be a substantive contribution. The full text also has some praiseworthy features: a reproducible LangGraph implementation is released, agent prompts are included in Appendix A, and a blind human evaluation on 50 questions is reported. However, the submitted body does not contain the audit that is the headline of the abstract; the claims that remain are on a different benchmark and are undermined by the overlap between the system's Judge Agent and the LLM-as-Judge evaluator. As submitted, the significance of the work cannot be assessed because the primary advertised evidence is missing.

major comments (4)
  1. [Abstract (metadata) vs. §§4.1–4.4] The arXiv abstract promises a claim-by-claim citation audit on a 100-question Bar Exam set with a six-class taxonomy, strict-ALCE scoring, cross-provider judging, and a Faith-Search post-draft intervention, reporting strict citation F1 0.13→0.25 and no-citation rate 34%→13%. None of this appears in the body. Sections 4.1–4.4 and Table 3 evaluate a different 200-question LegalSearchQA benchmark with accuracy and U-Score; there is no strict-ALCE definition, no taxonomy, no Bar Exam experiment, no Faith-Search, and no verifier configuration. The headline result is therefore untestable as submitted. This is a load-bearing internal inconsistency, not a matter of presentation.
  2. [§4.2 and §3.4] The LLM-as-Judge evaluation is performed by GPT-o3 (§4.2), while the Judge Agent inside L-MARS is also instantiated with GPT-o3 by default (§3.4). The model family used for qualitative evaluation is therefore the same as the model family that decides sufficiency and guides retrieval. The abstract's 'cross-provider judging' defense is not implemented or described in the body. Any shared formatting bias or shared hallucination pattern between the answerer and the verifier can inflate the reported preference scores. A protocol with a verifier from a different provider and a statement of which components were frozen would be needed.
  3. [§4.2, Eq. (1)] U-Score is introduced as a new metric with fixed linear weights and no validation. The five sub-scores are described only in prose; no rubrics, no example scorings, no inter-annotator agreement on the sub-scores, and no ablation are reported. Since Table 3's main message about 'reduced uncertainty' rests entirely on this metric, the 0.55–0.62 → 0.39–0.42 decrease is not interpretable. The metric should be validated against human uncertainty judgments or at least shown to be stable under perturbations of the weights.
  4. [§4.4, Table 3] Accuracy and U-Score are point estimates over 200 questions with no confidence intervals, seed variation, or significance tests. Given the multi-turn pipeline has stochastic retrieval and LLM calls, the reported 98% vs 96% vs 89% differences may be within noise. Additionally, the LegalSearchQA construction is not fully specified in §4.1: no question-writing protocol, no contamination check, and no dataset release link are provided (the repository link is for code). This makes it difficult to assess the body's central accuracy claim.
minor comments (6)
  1. [§4.1] The benchmark is introduced as LegalSearchQA, but the text switches to 'UncertainLegalQA' in the same paragraph; this appears to be a leftover from a previous version.
  2. [§3.3] Typo: 'guaranty bounded latency' should be 'guarantee bounded latency'.
  3. [§4.5] The human expert evaluation reports a 0.92 inter-annotator agreement rate, but the number of annotators, the annotation instructions, and whether the agreement is between humans or between humans and GPT-o3 are not specified.
  4. [Appendix B] The case study reports a 'Citation Score: 1.0' and 'U-Score: 0.611', but neither score is defined in the main text. The U-Score of 0.611 is higher (worse) than the average multi-turn 0.39 in Table 3, which is unexplained.
  5. [References] References [16] and [18] duplicate the same chain-of-thought citation, and [24] repeats [18].
  6. [Title/version] The metadata title ('...Citation-Faithfulness Audit') differs from the full-text title ('...Orchestrated Reasoning and Agentic Search'); this version mismatch compounds the abstract/body inconsistency.

Circularity Check

0 steps flagged

No circularity found: the body's accuracy and human-evaluation results are externally grounded; the abstract's separate audit is absent and the LLM-judge overlap is a bias concern, not a by-construction reduction.

full rationale

The body's derivation chain is not circular. Accuracy on LegalSearchQA is measured against expert-annotated multiple-choice ground truth, an external benchmark, and the reported gains (0.86–0.89 to 0.96–0.98) do not reduce to the system's own outputs or definitions. U-Score is a rule-based composite with fixed weights; although L-MARS's Judge Agent is explicitly prompted to prefer authoritative sources and to penalize hedging, the metric is not fitted to L-MARS and the accuracy result stands independently. The LLM-as-Judge ratings use GPT-o3, which is also the model instantiating the system's internal Judge Agent; this creates a same-family evaluator-bias risk and weakens the qualitative-preference claim, but it is not a by-construction equivalence, and the paper separately reports a blind human evaluation with 0.92 agreement with the LLM judge, providing independent content. The metadata abstract's strict-ALCE Bar Exam audit and Faith-Search numbers are not implemented or defined in the body, so they cannot be audited for circularity; absence of support is a verifiability and correctness concern, not a demonstrated circularity. No self-citation chain bears on the results, and the paper does not invoke any uniqueness theorem or prior work by the authors to force its architecture. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The claimed citation-faithfulness gains rest on unvalidated components: a custom judge prompt that decides when evidence is sufficient, and a strict-ALCE scoring protocol referenced only in the abstract. The body's separate evaluation adds hand-weighted U-Score components and LLM-as-judge ratings. None of these are grounded in external legal ground truth beyond one human preference study on 50 questions.

free parameters (3)
  • Judge sufficiency threshold and max iterations M = unspecified
    Algorithm 1 stops after a fixed maximum M and the Judge Agent's SUFFICIENT/INSUFFICIENT threshold is set by prompt rather than by an objective criterion; these choices affect citation F1 and no-citation rate.
  • Search depth parameters = top-m=3, top-k=5, 2.5k context window
    Hand-chosen defaults in Section 3.3; the central result could depend on how much and which content the search agent retrieves.
  • U-Score weights = 0.25, 0.20, 0.25, 0.15, 0.15
    Section 4.2 defines U-Score with hand-assigned linear weights that are not calibrated to any external ground truth.
axioms (4)
  • domain assumption An LLM judge can reliably assess legal evidence sufficiency, jurisdiction, temporal validity, and contradictions
    The Judge Agent prompt in Appendix A instructs the model to check sources, dates, jurisdiction, and contradictions; the paper does not validate these judgments against expert legal labels beyond a 50-question preference study.
  • domain assumption strict-ALCE scoring correctly measures citation faithfulness
    The abstract asserts the audit was scored with strict-ALCE under cross-provider judging, but the body neither defines strict-ALCE nor cites its source; an incorrect or lenient implementation would make the F1 gains meaningless.
  • domain assumption Retrieved web text accurately represents current law
    The system relies on Serper web snippets, page scrapes, and CourtListener; no independent legal verification of retrieved content is described.
  • domain assumption The stratified 100-question Bar Exam audit in the abstract is representative of legal QA citation failure
    The abstract gives no construction details, stratification procedure, or question source; if the sample is skewed, the headline numbers would not generalize.
invented entities (2)
  • Faith-Search no independent evidence
    purpose: Post-draft step that re-verifies and repairs unreachable citations
    Introduced only in the metadata abstract; the body does not describe its implementation, stopping rule, or evaluation. No external benchmark demonstrates its effect beyond the abstract's own claim of an unreachable rate below 1%.
  • Six-class citation taxonomy no independent evidence
    purpose: Labels each atomic claim with one of six classes for citation-faithfulness auditing
    Named in the abstract but not defined in the body; no class definitions, examples, or inter-annotator agreement are provided, so there is no independent evidence of its validity.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of L-MARS: Legal Multi-Agent System with Agentic Search and Citation-Faithfulness Audit." pith.science (2026). https://pith.science/paper/HQYW7CTE

@misc{pith2026250900761,
  author       = {Pith},
  title        = {Pith review of: L-MARS: Legal Multi-Agent System with Agentic Search and Citation-Faithfulness Audit},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HQYW7CTE}},
  note         = {Machine review of arXiv:2509.00761}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models are increasingly deployed for legal question answering, where evaluations typically focus on multiple-choice accuracy. This measure overlooks a common failure: whether the citation source attached to an answer exists and supports the rule the system attributes to it. We present L-MARS, an open multi-agent legal QA system with agentic search and judge-driven evidence checks, and audit it claim by claim against its cited source. Each atomic claim is labelled with a six-class taxonomy and scored with strict-ALCE under cross-provider judging, where the answerer and verifier come from different model families. On a stratified 100-question Bar Exam audit, retrieval barely moves accuracy, yet the multi-turn judge loop lifts strict citation F1 from 0.13 (naive RAG) to 0.25 and cuts the no-citation rate from 34% to 13%. We further introduce Faith-Search, a post-draft step that re-verifies and repairs unreachable citations; it drops the unreachable rate below 1% but does not improve F1 over the multi-turn loop, so we report it as a targeted reachability intervention rather than a faithfulness breakthrough. A 50-question LegalSearchQA case study confirms the picture: retrieve-then-draft pipelines saturate near 0.75 citation F1, while a single-agent web-search baseline collapses to 0.22 under external audit.

Figures

Figures reproduced from arXiv: 2509.00761 by Boqin Yuan, Ziqi Wang.

Figure 1
Figure 1. Figure 1: Comparison of the two L-MARS operating modes. (a) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Composition of the LegalSearchQA dataset. retrieve and reason over external legal sources—an essential capability for real-world deployment. To avoid reliance on static pre-training knowledge, all questions explicitly reference the legal status as of 2025, requiring retrieval of up-to-date authorities. The dataset spans multiple domains, with a strong emphasis on federal executive orders (60%), along with … view at source ↗
Figure 3
Figure 3. Figure 3: Average performance of baseline LLMs and our proposed [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Interplay of Harness Design and Post-Training in LLM Agents

    cs.LG 2026-06 unverdicted novelty 6.0

    Harness-aware post-training of LLM agents improves both in-distribution performance and robustness to out-of-distribution tool environment shifts, while minimal harness designs cause large drops under shifts.

  2. ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation

    cs.AI 2026-04 unverdicted novelty 6.0

    ClawTrace enables cost-aware LLM agent skill distillation by tracing per-step costs and generating preserve, prune, and repair patches, with ablations showing reduced regressions and prune rules transferring to cut co...

  3. Maat: The Agentic Legal Research Assistant for Competition Protection

    cs.AI 2026-05 unverdicted novelty 4.0

    Maat is a ReAct agent that orchestrates tools and RAG for competition law research, outperforming baselines on case-specific tasks while providing official citations.

Reference graph

Works this paper leans on

61 extracted references · 40 canonical work pages · cited by 3 Pith papers

  1. [1]

    Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al

    Hurst, Aaron, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. (2024). Gpt-4o system card. arXiv preprint arXiv:2410.21276

    Pith/arXiv arXiv 2024

  2. [2]

    Salazar-Molina, William A

    Rincón-Riveros, Daniel A., Sergio M. Salazar-Molina, William A. Pinto-Cáceres, Sindy P. Amaya, and Juan M. Calderon. (2021). Automation System Based on NLP for Legal Clinic Assistance. IFAC-PapersOnLine

    work page 2021

  3. [3]

    Anthropic. (2024). Introducing Claude 3.5 Sonnet. https://www.anthropic.com/news/ claude-3-5-sonnet

    work page 2024

  4. [4]

    Gemini Team and DeepMind. (2024). Gemini 1.5: Unlocking Multimodal Understanding Across Millions of Tokens of Context. arXiv preprint arXiv:2403.05530. 8

    Pith/arXiv arXiv 2024

  5. [5]

    Ji, Ziwei, Nayeon Lee, Jason Fries, Tao Yu, and Pascale Fung. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12):1–38

    work page 2023

  6. [6]

    Chalkidis, Ilias, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion An- droutsopoulos. (2020). LEGAL-BERT: The Muppets straight out of Law School. arXiv preprint arXiv:2010.02559

    Pith/arXiv arXiv 2020

  7. [7]

    Lewis, Patrick, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In NeurIPS

    work page 2020

  8. [8]

    Serper. (2024). Serper: Google Search API. https://serper.dev/. Accessed: 2025-08-21

    work page 2024

  9. [9]

    Free Law Project. (2024). CourtListener API. https://www.courtlistener.com/api/ rest/v4/search/

    work page 2024

  10. [10]

    Khattab, Omar, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. (2023). Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP. arXiv preprint arXiv:2212.14024

    Pith/arXiv arXiv 2023

  11. [11]

    Wu, Qianfan, Gagan Bansal, Jingfeng Zhang, et al. (2023). AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv preprint arXiv:2308.08155

    Pith/arXiv arXiv 2023

  12. [12]

    LangChain. (2024). LangGraph: State-Driven Multi-Agent Workflows for LLM Applications. https://www.langchain.com/langgraph

    work page 2024

  13. [13]

    Li, Xiaoxi, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. (2025). Search-o1: Agentic Search-Enhanced Large Reasoning Models. arXiv preprint arXiv:2501.05366

    Pith/arXiv arXiv 2025

  14. [14]

    Li, Xiaoxi, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. (2025). WebThinker: Empowering Large Reasoning Models with Deep Research Capability. arXiv preprint arXiv:2504.21776

    Pith/arXiv arXiv 2025

  15. [15]

    OpenAI. (2025). Introducing Deep Research. https://openai.com/index/ introducing-deep-research/

    work page 2025

  16. [16]

    OpenAI. (2024). OpenAI o1 System Card. arXiv preprint arXiv:2412.16720

    Pith/arXiv arXiv 2024

  17. [17]

    OpenAI. (2025). OpenAI o3 and o4-mini System Card. https://openai.com/index/ o3-o4-mini-system-card/

    work page 2025

  18. [18]

    Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. (2023). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv preprint arXiv:2201.11903

    Pith/arXiv arXiv 2023

  19. [19]

    Wang, Xuezhi, Jason Wei, Dale Schuurmans,et al. (2022). Self-Consistency Improves Chain- of-Thought Reasoning in Language Models. arXiv preprint arXiv:2203.11171

    Pith/arXiv arXiv 2022

  20. [20]

    Qwen Team. (2024). QwQ: Reflect Deeply on the Boundaries of the Unknown. https: //qwenlm.github.io/blog/qwq-32b-preview/

    work page 2024

  21. [21]

    DeepSeek-AI. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Rein- forcement Learning. arXiv preprint arXiv:2501.12948

    Pith/arXiv arXiv 2025

  22. [22]

    Chalkidis, Ilias, Manos Fergadiotis, Prodromos Malakasiotis, Spyridon Spanakis, and Nikolaos Aletras. (2022). LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. In ACL, pp. 4310–4330

    work page 2022

  23. [23]

    Krass, Lucy Zheng, et al

    Henderson, Peter, Massimiliano S. Krass, Lucy Zheng, et al. (2022). Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset. arXiv preprint arXiv:2207.00220. 9

    Pith/arXiv arXiv 2022

  24. [24]

    Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS

    work page 2022

  25. [25]

    Ho, et al

    Guha, Neel, Julian Nyarko, Daniel E. Ho, et al. (2023). LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in LLMs. arXiv preprint arXiv:2308.11462

    Pith/arXiv arXiv 2023

  26. [26]

    Robertson, Stephen and Hugo Zaragoza. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4):333–389

    work page 2009

  27. [27]

    Locke, Daniel and Guido Zuccon. (2022). Case law retrieval: problems, methods, challenges and evaluations in the last 20 years. arXiv preprint arXiv:2202.07209

    Pith/arXiv arXiv 2022

  28. [28]

    Xu, Ziwei, Sanjay Jain, and Mohan Kankanhalli. (2025). Hallucination is Inevitable: An Innate Limitation of Large Language Models. arXiv preprint arXiv:2401.11817

    Pith/arXiv arXiv 2025

  29. [29]

    Huang, Hsiu-Yuan, Yutong Yang, Zhaoxi Zhang, Sanwoo Lee, and Yunfang Wu. (2024). A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice. arXiv preprint arXiv:2410.15326

    Pith/arXiv arXiv 2024

  30. [30]

    Man- ning, Peter Henderson, and Daniel E

    Zheng, Lucia, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Man- ning, Peter Henderson, and Daniel E. Ho. (2025). A Reasoning-Focused Legal Retrieval Benchmark. In CSLAW ’25, pp. 169–193. ACM

    work page 2025

  31. [31]

    Fan, Yu, Jingwei Ni, Jakob Merane, Etienne Salimbeni, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, and Joel Niklaus. (2025). LEXam: Benchmarking Legal Reasoning on 340 Law Exams. arXiv preprint arXiv:2505.12864

    arXiv 2025

  32. [32]

    Yao, Shunyu, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv preprint arXiv:2210.03629

    Pith/arXiv arXiv 2023

  33. [33]

    Singh, Aditi, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. (2025). Agentic Retrieval- Augmented Generation: A Survey on Agentic RAG. arXiv preprint arXiv:2501.09136

    Pith/arXiv arXiv 2025

  34. [34]

    Kadavath, Saurav, Tom Conerly, Amanda Askell, et al. (2022). Language Models (Mostly) Know What They Know. arXiv preprint arXiv:2207.05221

    Pith/arXiv arXiv 2022

  35. [35]

    Hendrycks, Dan, Collin Burns, Anya Chen, and Spencer Ball. (2021). CUAD: An Expert- Annotated NLP Dataset for Legal Contract Review. arXiv preprint arXiv:2103.06268. 10 Appendix A Agent Instructions Query Agent Prompts # Follow-up Questions Generation Prompt You are a legal assistant helping users with their legal questions. User’s original question: {use...

    Pith/arXiv arXiv 2021

  36. [36]

    Legal category/area (e.g., contract law, criminal law, family law, employment law)

    work page

  37. [37]

    People involved with their roles (plaintiff, defendant, client, witness, etc.)

    work page

  38. [38]

    Jurisdiction if mentioned or inferable

    work page

  39. [39]

    Urgency level based on the nature of the query

    work page

  40. [40]

    Specific legal areas involved

    work page

  41. [41]

    Timeline or deadlines if mentioned

    work page

  42. [42]

    Summary Agent Prompt Create a comprehensive answer to the user’s legal question based on the search results

    Additional context about the situation Be thorough but accurate in your categorization. Summary Agent Prompt Create a comprehensive answer to the user’s legal question based on the search results. User’s Question: {user_query} Search Results: {results_content} Provide:

    work page

  43. [43]

    A clear, comprehensive answer

    work page

  44. [44]

    Key legal points and considerations

    work page

  45. [45]

    11 Judge Agent Prompt You are a legal research judge evaluating search results

    Important disclaimers about legal advice Remember: This is informational only and not legal advice. 11 Judge Agent Prompt You are a legal research judge evaluating search results. This is iteration {iteration_count + 1}. Original Question: {user_query} Conversation History: {conversation_context} Current Search Results ({len(search_results)} results): {re...

    work page

  46. [46]

    Consider: What was asked? What information was provided? What is still missing?

    REASONING (Chain of Thought): Think step by step about whether the search results answer the user’s specific question. Consider: What was asked? What information was provided? What is still missing?

    work page

  47. [47]

    SOURCE QUALITY CHECK: Analyze source authority - are there sources from: - Government (.gov) sites? - Court decisions/legal databases? - Educational institutions (.edu)? - How many authoritative vs user-generated content sources?

    work page

  48. [48]

    DATE CHECK: - Are the sources current and relevant to today’s date? - If there are older sources, do we also have recent confirmations? - Flag if critical information might be outdated

    work page

  49. [49]

    No specific jurisdiction mentioned

    JURISDICTION CHECK: - Does the jurisdiction of sources match the user’s location/scope? - For US federal vs state law, is the distinction clear? - User mentioned: {conversation_context if conversation_context else "No specific jurisdiction mentioned"}

    work page

  50. [50]

    California SB 365 employment arbitration 2025 full text

    CONTRADICTION SCAN: - Do any sources contradict each other? - If yes, what specific elements conflict? - Do we need more specific queries to resolve conflicts? STOP RULE: Mark as SUFFICIENT when: - Primary claims have authoritative support (gov/court/edu when available) - No critical information gaps for practical guidance - No unresolved contradictions -...

    work page 2025

  51. [51]

    Restoring Gold Standard Science

    whitehouse.gov - "Restoring Gold Standard Science" Official White House publication of the Executive Order with full text

    work page

  52. [52]

    New Executive Order on ’Gold Standard Science’: FOIA

    justice.gov - "New Executive Order on ’Gold Standard Science’: FOIA..." Department of Justice guidance on FOIA implications and compliance timelines

    work page

  53. [53]

    Implementing Gold Standard Science

    hhs.gov - "Implementing Gold Standard Science" Department of Health and Human Services implementation guidelines

    work page

  54. [54]

    White House OSTP Issues Agency Guidance for Gold

    lawbc.com - "White House OSTP Issues Agency Guidance for Gold..." Legal analysis of OSTP guidance requirements and deadlines

    work page

  55. [55]

    Federal Agencies Respond to ’Gold Standard Science

    library.washu.edu - "Federal Agencies Respond to ’Gold Standard Science..." Academic repository with detailed timeline analysis stating explicitly: "Section 3 mandates 30-day timeline for OSTP guidance issuance" Key Evidence Retrieved: - Executive Order issued: May 23, 2025 - OSTP guidance deadline: Within 30 days (by June 22, 2025) - Agencies must implem...

    work page 2025

  56. [56]

    reasonable period

    Reliance on General Patterns: GPT-4o relied on typical patterns in executive orders, assuming 90 days as a “reasonable period” without access to the specific order text

    work page

  57. [57]

    Lack of Specific Knowledge: The model’s training data cutoff prevented it from having specific information about an executive order in May 2025

    work page 2025

  58. [58]

    B.1.2 Why L-MARS Succeeded

    Statistical Bias: The model defaulted to a common timeline (90 days) that appears fre- quently in government documents, demonstrating the limitation of pattern-based reasoning without factual verification. B.1.2 Why L-MARS Succeeded

    work page

  59. [59]

    Direct Evidence Retrieval: L-MARS retrieved the actual executive order text from authori- tative sources, finding the explicit 30-day requirement in Section 3

    work page

  60. [60]

    Source Authority: The system identified and prioritized information from library.washu.edu, an authoritative academic source with direct access to government documents

    work page

  61. [61]

    B.2 Implications for Legal AI Systems This case study highlights several important considerations for legal AI applications

    Evidence-Based Reasoning: Rather than relying on typical patterns, L-MARS based its answer on explicit textual evidence from the retrieved documents. B.2 Implications for Legal AI Systems This case study highlights several important considerations for legal AI applications. • Temporal Sensitivity: Legal questions often require access to current and specif...

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.