Pith. sign in

REVIEW 5 major objections 4 minor 32 references

A legal citation is trustworthy only when the source exists, is described faithfully, and supports the claim it is made to carry—this paper builds a benchmark that measures all three in Chinese long-form legal research reports.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:05 UTC pith:QGHHFQOD

load-bearing objection A solid benchmark contribution built on a sensible E/F/A decomposition for citation trustworthiness; the headline trade-offs are suggestive but need tighter controls around the Trust metric and the verification ablation. the 5 major comments →

arxiv 2607.20872 v1 pith:QGHHFQOD submitted 2026-07-23 cs.CL

LegalCiteTrust: Benchmarking Citation Trustworthiness in Chinese Long-Form Legal Research Reports

Yunhan Li , Mingjie Xie , Zeyang Shi , Gengshen Wu , Min Yang This is my paper
classification cs.CL
keywords Chinese legal NLPcitation trustworthinesslong-form legal researchbenchmarkhallucination evaluationExistence-Fidelity-Applicabilityretrieval-augmented generationevidence governance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that evaluating legal AI research reports by coverage or evidence richness misses the real danger: a citation can be a real statute or case and still undermine the report by being misdescribed or stretched past what the source supports. To make that measurable, the authors introduce LegalCiteTrust, a 72-task benchmark for Chinese long-form legal research reports that scores Coverage, Support, and a Trust score built from citation-level Existence, Fidelity, and Applicability judgments. Across general LLMs, deep-research agents, and legal-specific systems, the benchmark shows the three dimensions behave differently: retrieval tools raise evidence support but do not reliably raise Trust, and denser citation correlates with lower Trust. Feeding the three judgments back into a revision stage improves Trust and final score more than filtering out nonexistent citations alone. The paper concludes that trustworthy legal research generation needs post-retrieval governance of how authorities are selected, described, and applied, not just more retrieval.

Core claim

On the paper's own terms, the discovery is that citation trustworthiness is a distinct, measurable axis of legal AI quality, irreducible to whether a report covers the requested structure or contains abundant legal references. A real citation can fail in three ways: the source does not exist (Existence), the report misstates its content (Fidelity), or the source does not support the local claim it is made to carry (Applicability). The benchmark multiplies these three scores per citation and aggregates them into a report-level Trust score, so a report that cites real sources can still score low when it reverses a holding, omits a limiting condition, or uses a statute for a proposition it does

What carries the argument

The load-bearing object is the E/F/A citation-level judgment: Existence, Fidelity, and Applicability. Each citation in a generated report is turned into a verification item with the citation text, its context, the local claim it supports, and retrieved source content; an LLM judge assigns a 0–1 score on each of the three layers, the three are multiplied per citation, and the product is averaged into a report-level Trust score. Fidelity and Applicability are the novel layers—they catch contexts where the source is real but the report misdescribes it or overextends it. The final score multiplies (Coverage + Support) by Trust, so a sparse report cannot hide behind a high Trust score and an evid

Load-bearing premise

The load-bearing premise—stated in Section 3.3 and qualified in the paper's Limitations—is that the automated E/F/A judge's Fidelity and Applicability labels are roughly correct; the human audit's reference labeler helped design the protocol, and a valid citation absent from the commercial retrieval store can be scored unverifiable, so if the judge is biased or the source store incomplete, every Trust-based ranking shifts.

What would settle it

Have an independent legal expert who did not participate in protocol design re-judge a random sample of citation verification items against an independently assembled source store, then recompute the report-level Trust scores and system ordering; if score-level agreement falls materially below the audit reported in Appendix C.1, or if any system crosses another in the leaderboard, the core metric is not reproducible.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A system can cover the requested structure and cite many real authorities yet still be untrustworthy because Fidelity or Applicability fails.
  • Retrieval tooling gains—more evidence and denser citations—do not automatically improve citation reliability; in the paper's ablation, case search raised Support but lowered Trust.
  • Feeding E/F/A diagnostics back into a revision stage improves Trust and Final score more than filtering on Existence alone.
  • Evaluations that reward evidence richness or citation density without checking reliability overstate the quality of legal AI systems.
  • Trust alone is insufficient: the system with the highest Trust in the leaderboard had very low Coverage and Support, so benchmarks must report all three dimensions jointly.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Outside the paper's claims: the exist-describe-apply decomposition generalizes to any domain where a cited source can be real yet misapplied—clinical guidelines, financial disclosures, policy briefs—so the benchmark's method, not its Chinese legal surface, is the transferable core.
  • Outside the paper's claims: the negative Support–Trust trend should be re-tested holding citation extraction and judge prompts fixed across systems; if the trend persists, evidence governance is a structural property, but if it flips, part of the signal is an evaluation artifact.
  • Outside the paper's claims: the E/F/A feedback result predicts that revision with rationales will beat binary filtering in other long-form generation settings, a prediction that is directly testable outside law.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces LegalCiteTrust, a benchmark for evaluating citation trustworthiness in Chinese long-form legal research reports. The benchmark consists of 72 tasks with rubrics and evidence points, and it separates three evaluation dimensions: Coverage (adherence to the requested research structure), Support (evidence richness), and Citation Trustworthiness (a report-level Trust score aggregated from citation-level Existence, Fidelity, and Applicability judgments). Experiments compare seven end-to-end systems (general LLMs, deep-research systems, legal-specific systems), plus tool and verification ablations. The main findings are that retrieval tools can increase Support without reliably improving Trust, and that full E/F/A feedback improves revision more clearly than existence-only filtering. The paper argues that trustworthy legal research generation requires citation-aware evidence governance after retrieval, not merely more retrieval.

Significance. If the central claims hold, the benchmark fills a gap in legal NLP evaluation by distinguishing whether a report covers and supports a research plan from whether the cited authorities are actually trustworthy. The paper’s E/F/A decomposition is a principled operationalization of citation reliability, and the benchmark is dense (585 rubrics, 1,459 evidence points). The authors also provide a detailed protocol, human-audit and bootstrap analyses, and item-level diagnostics, which are strengths. The main empirical contribution—that Support and Trust are not aligned—has practical implications for designing legal research agents. However, the validity of the Trust score and the strength of the verification-ablation conclusions need additional support, as detailed below.

major comments (5)
  1. [Section 3.3, Appendix C.1, Limitations] The human audit validates the LLM E/F/A judge against labels from the primary expert who co-developed the protocol. The paper acknowledges this in Limitations, but because every headline result (including the Support–Trust trade-off and the verification-ablation gains) builds on the Trust score, the metric’s construct validity requires a more independent audit. The secondary-expert check in C.2.2 is a useful step, but the second auditor was from the same collaborating organization and audited only a subset. Please add an independent, external expert audit on a held-out sample, ideally with a pre-registered protocol, or substantially soften the main claims until such validation exists. Without this, the benchmark cannot be distinguished from a consistency check of the authors’ preferred protocol.
  2. [Section 5.3, Table 20] The abstract and conclusion state that E/F/A-based revision improves Trust and Final score 'more clearly than existence-only filtering.' The bootstrap confidence intervals in Table 20 support the Trust claim: E-Filter [0.7839, 0.8416] and EFA-Revise [0.8631, 0.9120] do not overlap. However, the Final-score CIs overlap substantially (E-Filter [1.0501, 1.1435]; EFA-Revise [1.1252, 1.2190]). Thus the Final-score improvement is within sampling error. Please either qualify this claim, report a paired bootstrap test over the 72 tasks, or add evidence that the Final-score difference is reliable. The current wording overstates the support for a central conclusion.
  3. [Section 5.3] The verification ablation lacks a revision-without-feedback control. Base is the unrevised system, while both E-Filter and EFA-Revise include an LLM revision stage. The improvement from Base to EFA-Revise could be partly due to the generic instruction to revise the report, rather than to the E/F/A feedback itself. Add a No-Feedback-Revise condition (e.g., 'revise the report to improve its accuracy and reliability') to isolate the effect of the feedback signal. This control is necessary to support the claim that citation-verification feedback, rather than revision in general, drives the Trust gain.
  4. [Section 6, Figures 2–3, Table 15] The headline Support–Trust trade-off (ρ = −0.9167) and the reference-count–Trust association are computed over only n = 9 systems. The manuscript correctly describes these as descriptive, but the claims in the Abstract and Conclusion depend on these correlations. The sensitivity analysis in Table 15 shows that excluding a single system (GPT5) changes the Coverage–Trust correlation from 0.683 to 0.548, and including Farui drops it to 0.224. Please add leave-one-out correlations or a system-level bootstrap confidence interval for the key ρ values, and discuss the instability. This is important because the evidence-governance argument is built on these system-level summaries.
  5. [Section 5.3, Eq. (2), Appendix C] The E/F/A judge is used both to generate revision feedback in EFA-Revise and to score the revised reports. This introduces a potential feedback loop: the revision may optimize the judge’s preferences rather than the underlying legal reliability of citations. The paper does not address this circularity. Please evaluate the revised reports with an independent LLM judge (e.g., a different model or a separate judge instance) or with a human-audit sample, and report whether the Trust gains persist. Without this, the external validity of the verification-ablation results remains questionable.
minor comments (4)
  1. [Section 3.4] The sentence 'If a report contains no included law/case citation items, strust is set to 1.0' is a strong design choice. It would be helpful to repeat this explicitly in the main text (beyond the appendix) and to discuss its effect on systems that produce sparse citations. As written, a report with no citations receives a perfect Trust score, which may surprise readers.
  2. [Appendix C] The reliability statistics report 'label κ' and 'score κ' without specifying the exact kappa variant (e.g., Cohen’s kappa, weighted kappa) or how ties are handled. Also, the bootstrap CIs in Tables 18–20 are described as '95% bootstrap confidence intervals' but the resampling procedure (percentile, BCa, etc.) is not stated. Please clarify these details.
  3. [Figures 2 and 3] The figure captions state Spearman’s ρ, but the main text in Section 6.1 writes 'ρ = 0.6833' and 'ρ = −0.9167' without consistently noting that these are Spearman correlations. Please make the notation uniform across text and figures.
  4. [Table 24] For Farui, case E/F/A columns are empty (—). This is because Farui generated zero case citations. In a table footnote, clarify that missing entries indicate zero cases, not unmeasured values. The current formatting could be misread as missing data.

Circularity Check

2 steps flagged

Trust-score validation is self-referential (the reference expert co-authored the E/F/A protocol), and the EFA-revision gain is partly scored by the same signal that generated the feedback; leaderboard and tool-ablation findings retain independent empirical content.

specific steps
  1. self definitional [Appendix C preamble and C.1; echoed in Limitations and F.2]
    "One primary expert led most of the protocol refinement and produced the final human-audit labels used in C.1; other members contributed mainly through discussion and consultation... The human-audit results should therefore be interpreted as agreement with the expert-guided annotation protocol, not as proof of objective legal truth."

    The LLM E/F/A judge is the instrument that produces every Trust score (Eq. 2). Its validation (C.1) measures agreement (κ=0.87) against labels produced by the primary expert who led the protocol refinement — the same protocol the judge implements. So the audit establishes that the judge follows its own defining protocol, not that the protocol captures an externally meaningful legal property; the paper concedes exactly this ('consistency with the protocol, not agreement with objective legal truth'). Section 6 nonetheless cites this audit as evidence that the main E/F/A and leaderboard trends 'are stable,' making the Trust metric's reliability claim self-referential. The secondary-expert check (C.2.2, κ=0.93) softens but does not break the self-reference: it uses the same protocol and projec

  2. fitted input called prediction [Section 5.3 and Eq. (2); Conclusion]
    "The verification ablation tests whether E/F/A signals can improve generated reports rather than merely score them after the fact. We use Qwen36_tool_law_case as the Base system, add an LLM revision stage, and compare two feedback variants: E-Filter receives only Existence results, while EFA-Revise receives the full E/F/A verification report. Trust increases monotonically from Base to E-Filter and EFA-Revise: 0.7433, 0.8134, and 0.8883."

    strust is defined (Eq. 2) as the mean E·F·A over included citation items. EFA-Revise's input is the full E/F/A verification report — i.e., the judge's own E, F, A labels for exactly those items — and its output is then re-scored by the same pipeline. Items the judge flagged as failing have below-mean E·F·A products; removing or revising precisely those items mechanically raises the recomputed mean, and the judge is confirming its own earlier feedback. Hence 'E/F/A signals improve reports' partly reduces to 'E/F/A feedback improves E/F/A scores,' with no independent verification of the revised text. The E-Filter vs. EFA-Revise comparison keeps empirical content (both arms receive the same judge's signals; richer feedback helps more, and case F/A fall under E-Filter), so the circularity is p

full rationale

This is an empirical benchmark paper, not a derivation from fitted parameters. Coverage, Support, and Trust are computed independently (Eqs. 1-3), the 72-task leaderboard, tool ablation, and item-level diagnostics are direct measurements of system outputs, and the paper is self-contained relative to external benchmarks; no self-citation chain carries the load, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The core empirical findings — retrieval raises Support without reliably raising Trust (Table 6), and systems split across Coverage/Support/Trust (Table 5) — are not forced by the formulas: nothing in Eqs. 1-3 requires denser citation behavior to correlate with lower mean E·F·A. Two self-referential elements warrant flags. (1) The human audit of the LLM E/F/A judge (C.1) uses labels from the primary expert who led the protocol refinement; the paper itself concedes the audit tests consistency with the protocol, not objective legal truth (C preamble; Limitations; F.2), so the reliability evidence for Trust is agreement with its own construction. The secondary-expert check (C.2.2) mitigates but does not break this, as it uses the same protocol and project-internal experts. (2) In the verification ablation (5.3), the revision feedback is the full E/F/A verification report and the measured outcome is strust (Eq. 2) from the same pipeline; removing or revising the exact items the judge flagged mechanically raises the mean, so 'E/F/A improves reports' is partly 'E/F/A improves E/F/A scores.' The E-Filter vs. EFA-Revise comparison remains empirically informative. Both circularities are partly disclosed by the authors, and neither reduces the central benchmark results to their inputs, so a moderate score of 4 is appropriate. Additionally, Trust is bounded by reference-store and commercial-retrieval coverage (Limitations), which could confound the Support–Trust correlation with retrieval coverage of less-common authorities; this is a validity risk rather than a circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 1 invented entities

The benchmark's conclusions rest on several hand-chosen design parameters (metric weights, NA handling, dataset thresholds) and on the reliability of the LLM E/F/A judgment pipeline. None are fitted to external data; they are normative choices that could affect the scores.

free parameters (6)
  • Support mixing weight = 0.5/0.5
    Equal weight between rubric evidence support and subtopic evidence support (Eq. 1); chosen by hand, not fitted to data.
  • Final composite weights = Coverage + Support
    Final = (Coverage + Support) * Trust (Eq. 3); equal weighting of coverage and support is an arbitrary design choice.
  • Trust NA handling = 1.0 for NA/skipped F/A
    NA or skipped Fidelity/Applicability scores are treated as 1.0, potentially inflating Trust when semantic judgments are unavailable (Section A.4.6).
  • Label-to-score mappings = 1.0 / 0.5 / 0.0
    E/F/A label-to-score mappings assign partial credit (e.g., 'Key omission' = 0.5) without an external calibration basis (Tables 12, 14).
  • Dataset inclusion thresholds = ≥1 case citation, ≥5 statutory references
    The 13,622 candidate documents are filtered to 72 authority-demanding tasks, which shapes the benchmark's difficulty and may bias the Support–Trust trade-off (Section 3.2).
  • SES raw score thresholds = 0/1/2/3 authorities
    Subtopic Evidence Support uses hand-chosen count thresholds and a requirement that three authorities include both law and case (Table 10).
axioms (4)
  • domain assumption The E/F/A protocol as defined by the authors can be reliably applied by LLM judges.
    Trust scores are computed from LLM judgments; the validation in Appendix C is based on a small human audit with shared protocol design, so the assumption is not fully independently verified.
  • domain assumption Chinese legal citations can be resolved against the reference stores and commercial retrieval APIs used.
    The pipeline's Existence checks depend on external legal databases and APIs whose coverage is not fully disclosed (Section A.4.1).
  • domain assumption The 72 authority-demanding tasks are representative enough to support general claims about long-form legal research systems.
    The dataset is deliberately filtered to documents with at least one case and five statutory refs; the authors acknowledge it is 'dense rather than broad' (Section 8).
  • domain assumption Spearman correlations over 9 system-level points provide a stable estimate of the relationship between metrics.
    System-level correlations in Section 6 are computed over 9 points (or 10 with Farui), giving wide uncertainty intervals; the authors themselves call the analyses descriptive.
invented entities (1)
  • Existence/Fidelity/Applicability (E/F/A) scoring construct independent evidence
    purpose: To operationalize citation trustworthiness in legal research reports
    The construct yields numeric scores that can be compared against independent expert annotations (Appendix C), providing a falsifiable handle outside the paper.

pith-pipeline@v1.3.0-alltime-deepseek · 21553 in / 13771 out tokens · 122381 ms · 2026-08-01T09:05:32.782874+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of LegalCiteTrust: Benchmarking Citation Trustworthiness in Chinese Long-Form Legal Research Reports." pith.science (2026). https://pith.science/paper/QGHHFQOD

@misc{pith2026260720872,
  author       = {Pith},
  title        = {Pith review of: LegalCiteTrust: Benchmarking Citation Trustworthiness in Chinese Long-Form Legal Research Reports},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QGHHFQOD}},
  note         = {Machine review of arXiv:2607.20872}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Long-form legal research reports increasingly rely on LLMs and agentic research systems, but their reliability depends not only on answering the task, but also on whether cited legal authorities are trustworthy. A citation can be risky even when it points to a real source: the report may omit limiting conditions, misdescribe the authority, or use it to support a stronger claim than the source allows. We introduce LegalCiteTrust, a benchmark for evaluating citation trustworthiness in Chinese long-form legal research reports. It contains 72 densely annotated report-level tasks and evaluates reports along three dimensions: Coverage, Support, and Citation Trustworthiness. Citation Trustworthiness is operationalized through citation-level Existence, Fidelity, and Applicability (E/F/A). Experiments on general-purpose LLMs, deep-research systems, and legal-specific systems show that task completion, evidence richness, citation density, and citation reliability expose different system behaviors. Retrieval tools can improve evidence support without reliably improving the Trust score, while E/F/A-based revision improves Trust and Final score more clearly than existence-only filtering. These results suggest that trustworthy legal research generation requires citation-aware evidence governance after retrieval: systems must not only retrieve legal authorities, but also select, describe, and apply them reliably.

Figures

Figures reproduced from arXiv: 2607.20872 by Gengshen Wu, Mingjie Xie, Min Yang, Yunhan Li, Zeyang Shi.

Figure 1
Figure 1. Figure 1: Overview of LegalCiteTrust. The benchmark connects report-level Coverage and Support with citation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Coverage-like report completion and evidence support capture different aspects of citation trustworthiness. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: System-level relationship between reference [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: High-level citation verification protocol in LegalCiteTrust. Model-generated reports are converted into [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 9 linked inside Pith

  1. [1]

    arXiv preprint arXiv:1807.02478 , year=

    Cail2018: A large-scale legal dataset for judgment prediction , author=. arXiv preprint arXiv:1807.02478 , year=

    Pith/arXiv arXiv

  2. [2]

    arXiv preprint arXiv:2103.06268 , year=

    Cuad: An expert-annotated nlp dataset for legal contract review , author=. arXiv preprint arXiv:2103.06268 , year=

    Pith/arXiv arXiv

  3. [3]

    Findings of the Association for Computational Linguistics: EMNLP 2021 , pages=

    ContractNLI: A dataset for document-level natural language inference for contracts , author=. Findings of the Association for Computational Linguistics: EMNLP 2021 , pages=

    2021

  4. [4]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    LexGLUE: A benchmark dataset for legal language understanding in English , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  5. [5]

    Proceedings of the nineteenth international conference on artificial intelligence and law , pages=

    Summary of the competition on legal information, extraction/entailment (COLIEE) 2023 , author=. Proceedings of the nineteenth international conference on artificial intelligence and law , pages=

    2023

  6. [6]

    Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    CLERC: A dataset for US legal case retrieval and retrieval-augmented analysis generation , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    2025

  7. [7]

    Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval , pages=

    LeCaRD: a legal case retrieval dataset for Chinese law system , author=. Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval , pages=

  8. [8]

    Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    Lecardv2: A large-scale chinese legal case retrieval dataset , author=. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

  9. [9]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    STARD: A Chinese Statute Retrieval Dataset Derived from Real-life Queries by Non-professionals , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    2024

  10. [10]

    arXiv preprint arXiv:2605.30497 , year=

    CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Law , author=. arXiv preprint arXiv:2605.30497 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2601.17692 , year=

    LegalMALR: Multi-Agent Query Understanding and LLM-Based Reranking for Chinese Statute Retrieval , author=. arXiv preprint arXiv:2601.17692 , year=

    arXiv

  12. [12]

    Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    Lawbench: Benchmarking legal knowledge of large language models , author=. Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    2024

  13. [13]

    Advances in neural information processing systems , volume=

    Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models , author=. Advances in neural information processing systems , volume=

  14. [14]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    PLAWBENCH: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  15. [15]

    International Conference on Learning Representations , volume=

    Agentbench: Evaluating llms as agents , author=. International Conference on Learning Representations , volume=

  16. [16]

    International Conference on Learning Representations , volume=

    Openhands: An open platform for ai software developers as generalist agents , author=. International Conference on Learning Representations , volume=

  17. [17]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Legalagentbench: Evaluating llm agents in legal domain , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  18. [18]

    arXiv preprint arXiv:2508.15804 , year=

    Reportbench: Evaluating deep research agents via academic survey tasks , author=. arXiv preprint arXiv:2508.15804 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2506.06287 , year=

    Deep research bench: Evaluating ai web research agents , author=. arXiv preprint arXiv:2506.06287 , year=

    Pith/arXiv arXiv

  20. [20]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Deep research arena: The first exam of LLMs' research abilities via seminar-grounded tasks , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  21. [21]

    Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

    Truthfulqa: Measuring how models mimic human falsehoods , author=. Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

  22. [22]

    Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    Halueval: A large-scale hallucination evaluation benchmark for large language models , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    2023

  23. [23]

    Proceedings of the 18th conference of the european chapter of the association for computational linguistics: system demonstrations , pages=

    Ragas: Automated evaluation of retrieval augmented generation , author=. Proceedings of the 18th conference of the european chapter of the association for computational linguistics: system demonstrations , pages=

  24. [24]

    Advances in Neural Information Processing Systems , volume=

    Crag-comprehensive rag benchmark , author=. Advances in Neural Information Processing Systems , volume=

  25. [25]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  26. [26]

    arXiv preprint arXiv:2605.10186 , year=

    LegalCiteBench: Evaluating Citation Reliability in Legal Language Models , author=. arXiv preprint arXiv:2605.10186 , year=

    Pith/arXiv arXiv

  27. [27]

    arXiv preprint arXiv:2601.03267 , year=

    Openai gpt-5 system card , author=. arXiv preprint arXiv:2601.03267 , year=

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2606.19348 , year=

    DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , author=. arXiv preprint arXiv:2606.19348 , year=

    arXiv

  29. [29]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  30. [30]

    arXiv preprint arXiv:2510.24701 , year=

    Tongyi DeepResearch Technical Report , author=. arXiv preprint arXiv:2510.24701 , year=

    Pith/arXiv arXiv

  31. [31]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Dream: Deep research evaluation with agentic metrics , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  32. [32]

    The University of New South Wales Law Journal , volume=

    The promise and the peril of the use of generative Artificial Intelligence in litigation , author=. The University of New South Wales Law Journal , volume=. 2025 , publisher=

    2025