REVIEW 4 major objections 6 minor 3 cited by
CitaLaw: Enhancing LLM with Citations in Legal Domain
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read CitaLaw is a benchmark that tests whether legal LLMs ground every sentence in a cited law article or precedent case, and its experiments show that adding citations improves answer quality while a syllogism-based scorer matches human…
desk verdict A genuinely new legal-citation benchmark with a thoughtful layperson/practitioner design, but the citation-quality metric is scored by a model whose training data sits inside the benchmark corpus and which was selected on the same benchmark, so the reported agreement and entailment scores are optimistic. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key mechanism is the syllogism decomposition of legal reasoning into three components—major premise (law article or precedent case), minor premise (circumstances and illegal acts), and conclusion (legal decision)—combined with an NLI-based entailment scorer. The paper uses Qwen2 to extract these components and DISC-LawLLM as the NLI model to measure whether the cited reference entails the response component, producing scores such as CitaLaw, Citac, Citaa, and Citad. This machinery replaces surface overlap metrics with a logical consistency check.
What would settle it
Check whether DISC-LawLLM's fine-tuning data overlaps with CitaLaw's corpus; if removing overlapping QA pairs from the benchmark noticeably changes the entailment scores, the conclusion that references improve citation quality is weakened. Alternatively, replace DISC-LawLLM with a scorer trained only on non-overlapping data and see if the human-agreement kappas still hold.
Extended reading notes
Core claim
The central claim is that CitaLaw provides a reusable testbed for citation-grounded legal QA, and that a syllogism-inspired evaluation—treating the cited law article or precedent as the major premise, the case facts as the minor premise, and the legal decision as the conclusion—can reliably assess whether a response is legally coherent and properly supported by citations. The authors show that this syllogism-level scoring agrees strongly with human legal annotators, outperforming surface-level metrics like ROUGE in capturing logical alignment. They also find that incorporating references into the LLM's input (citation-guided generation) beats answer-refinement approaches on correctness, while refinement better aligns citations with sentences, and that open-domain LLMs like Qwen2 and Llama3 sometimes outperform older legal-specific models.
Load-bearing premise
The citation-quality scores assume that the NLI model used to measure entailment has not already seen the benchmark's own questions and answers during training; if it has, the reported citation-quality improvements could be inflated.
Editorial extensions
If this is right
- If CitaLaw is accepted, citation-grounded legal QA can be evaluated on a standard 1,000-question benchmark with half a million reference documents, making model comparisons reproducible.
- The syllogism-level metrics (circumstances, illegal acts, legal decisions) could replace or supplement ROUGE and BERTScore for judging legal reasoning quality.
- The finding that citation-guided generation (references in the prompt) outperforms answer refinement on correctness suggests that legal RAG systems should pass retrieved texts into the LLM's context.
- The finding that answer refinement better aligns citations with individual sentences suggests a hybrid approach could maximize both correctness and citation trustworthiness.
Reading between the lines
- Because DISC-LawLLM's fine-tuning QA pairs are included in CitaLaw's corpus, using that same model as the NLI scorer may inflate entailment scores; a holdout evaluation with a non-overlapping scorer would test this directly.
- The benchmark is built on Chinese law, so extending CitaLaw to other jurisdictions requires rebuilding the reference corpus, though the syllogism decomposition itself should transfer across legal systems.
- A testable extension would vary the NLI scorer and measure human agreement for each, revealing whether the reported human-model agreement depends on the scorer's training data overlap.
- The syllogism evaluation could generalize beyond law to any domain with a premise-evidence-conclusion structure, such as medical advice or financial claims, where citations must support each assertion.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CitaLaw, a Chinese-language benchmark for evaluating LLM-generated legal responses with citations. It contains two subsets (layperson and practitioner) and a large reference corpus of law articles and precedent cases, including QA pairs taken from legal LLM fine-tuning datasets. The authors propose two generation paradigms (citation-guided generation, CGG, and answer-refinement generation, ARG) and a two-level evaluation framework: global metrics (MAUVE, ROUGE, BERTScore) and a syllogism-inspired level that extracts circumstances, illegal acts, and legal decisions and scores citation entailment with an NLI model. Experiments over seven legal-specific and two open-domain LLMs are used to claim that incorporating legal references improves response quality and that the proposed syllogism-based evaluation agrees substantially with human judgments (kappa 0.7876 for component extraction and 0.6923 for citation entailment).
Significance. If the evaluation signal were validated, CitaLaw would be a useful first benchmark for citation-grounded legal QA: it separates layperson and practitioner needs, includes both law articles and precedent cases, and proposes a reasoning-aware alternative to surface-level citation metrics. The design of the CGG/ARG comparison, the breadth of the reference corpus, the use of temperature-0 generation, and the attempt to validate with human annotators are all strengths. However, the paper's central validation currently rests on a circular evaluator choice, on a corpus that contains training data of the scored models, and on aggregate comparisons without significance testing; these issues must be addressed before the benchmark's evaluation claims can be accepted.
major comments (4)
- [§5.3 with §3 and §6.3] The choice of DISC-LawLLM as the NLI scorer is circular. The CitaLaw corpus explicitly includes QA pairs from DISC-LawLLM's fine-tuning data as precedent cases (Section 3), and Section 5.3 states that DISC-LawLLM was selected because of its agreement with human evaluations on this same benchmark (Section 6.3). Since the scorer's training data overlaps the benchmark and the scorer was selected after observing its agreement on a subset of this benchmark, the reported kappa of 0.6923 and the entailment scores in Tables 2 and 3 are optimistic upper bounds. I would like to see the citation-quality evaluation repeated with a general-purpose NLI model not trained on CitaLaw's sources, and with a version of the corpus that excludes the DISC-LawLLM fine-tuning pairs, with a report of whether the CGG/ARG/CloseBook ranking is preserved.
- [§3, Table 5, §6.1.1, §6.2.2] The corpus contains fine-tuning QA pairs from DISC-LawLLM, LawGPT_zh, and HanFei as precedent cases, and these exact three models are later evaluated (Section 6.1.1). Retrieval and generation for these models under CGG/ARG can therefore benefit from memorized training instances, confounding the claim that legal-specific LLMs perform well (for example, 'HanFei achieves the best results in the Practitioner dataset'). The authors should either remove each evaluated model's own fine-tuning data from the corpus during that model's evaluation, or demonstrate disjointness between the evaluated queries/cases and the models' training data.
- [Tables 2, 3 and §6.2] The 'Avg' column averages twelve heterogeneous metrics (MAUVE, ROUGE-1/2/L, BERT-F, Correctc/a/d, CitaLaw, Citac/a/d) with different scales and no weights, and no error bars or significance tests are reported. Conclusions such as 'CGG achieves optimal performance' and 'LexiLaw achieves optimal performance overall' rely on small aggregate differences (for example, 53.82 vs 53.11 for Qwen2 in Table 2). Please report per-metric differences with bootstrap confidence intervals or paired significance tests, and either define a principled aggregation or remove the single 'Avg' column.
- [§6.3 and Appendix D] The human-agreement validation is too thinly documented to support the claim of 'strong agreement.' Stage 2 uses only 50 randomly selected Practitioner questions, does not report inter-annotator agreement, and converts continuous model probabilities to a 1-5 scale by multiplying by 5 and rounding; it is also unclear whether the reported kappa is between the model and each annotator or between the model and pooled human labels. Please report the full annotation protocol, human-human kappa, model-human kappa with confidence intervals, and ideally include Layperson data in Stage 2.
minor comments (6)
- [Throughout] The benchmark and the global citation-entailment metric are both named 'CitaLaw', which is confusing; please rename one of them.
- [§2] The phrase 'A amount of work' is ungrammatical and should read 'A number of works' or similar.
- [§7] The section title 'Conclution' should be 'Conclusion'.
- [Appendix C] The text 'For RGUGE' appears to be a typo for 'For ROUGE'.
- [Table 2] The header 'Llam3-8B-Instruct' should be 'Llama3-8B-Instruct'.
- [§5.2] The extraction prompt used with Qwen2 for the three syllogism components is not provided in the appendix; including it would improve reproducibility.
Circularity Check
Citation-quality evaluation is scored by a model whose fine-tuning data sits inside the benchmark corpus and that is selected after it agrees with humans on the same benchmark, making the reported human agreement and CitaLaw scores optimistic.
-
fitted input called prediction
[Section 3 (Dataset Construction) and Section 5.3 (Citation Quality)]
"As a supplement to precedent cases, we also incorporate question-and-answer pairs from fine-tuning datasets of legal LLMs as part of the precedent cases. These QA pairs are collected from DISC-LawLLM (Yue et al., 2023), LawGPT_zh (Liu et al., 2023), and HanFei (He et al., 2023). ... We employ DISC-LawLLM (Yue et al., 2023) as the NLI model due to its strong agreement with human evaluations (as discussed in Sec. 6.3) and its superior performance compared to other NLI models (as detailed in Sec. 6.5)."
The citation-quality metric in CitaLaw is computed by DISC-LawLLM, and the same model's fine-tuning QA pairs were folded into CitaLaw's precedent corpus. When the NLI scorer evaluates whether a retrieved precedent entails a generated sentence, it may be recognizing examples from its own training distribution rather than performing independent legal reasoning. Because the human-agreement validation (Sec. 6.3) is also run on the same benchmark, the claimed kappa of 0.6923 does not certify the metric against external ground truth. The benchmark's headline conclusion that adding legal references improves response quality therefore depends on a citation-quality signal that is not independently measured.
-
fitted input called prediction
[Section 5.3 and Section 6.3 (Human Evaluation)]
"We randomly selected 50 questions from the Practitioner dataset and used Qwen2 to extract key components of pairs of responses and citations. Annotators assessed the degree to which the citations entailed the corresponding response components using a 5-point scale ... The entailment probabilities given by DISC-LawLLM, which range from 0 to 1, were scaled to the same 1–5 range by multiplying by 5 and rounding. We then compared the scaled model outputs with the human evaluations and calculated Cohen’s kappa coefficient."
DISC-LawLLM is adopted as the official NLI evaluator only after the authors observe its agreement with human annotators on a 50-question sample of the same Practitioner dataset that the evaluator will later score. Selecting a model based on its performance on the evaluation set and then reporting that performance as evidence of validity is selection on the test set. The reported Cohen's kappa is therefore an optimistic upper bound, and the sentence in Sec. 5.3 that justifies the model choice by 'its strong agreement with human evaluations' cites a measurement made on the benchmark itself, closing the validation loop.
full rationale
CitaLaw's dataset construction and response-generation comparisons are largely self-contained: the questions come from LawBench and LexEval, the reference corpus is assembled from public legal datasets, and the CGG/ARG comparisons are ordinary LLM evaluations. However, the central validation of the syllogism-based citation-quality metric is circular. The paper includes DISC-LawLLM's fine-tuning QA pairs in CitaLaw's precedent corpus (Sec. 3), then chooses DISC-LawLLM as the NLI scorer because it shows strong human agreement (Sec. 5.3), where that agreement was measured on a 50-question subset of the same benchmark (Sec. 6.3). This is selection on the test set and training-set overlap for the evaluator, so the reported kappa of 0.6923 and the CitaLaw citation-quality scores are optimistic upper bounds. The conclusion that references improve response quality rests partly on this signal. No self-citation chain or uniqueness theorem is involved, and the response-generation results themselves are not circular, so the paper is partially but not wholly circular.
Assumptions & free parameters
free parameters (3)
- Number of law articles for layperson (|CLay|=1) =
1
- Number of precedent cases for practitioner (|Dc|=3) =
3
- Sentence eligibility criteria (THUOCL terminology pool plus SpaCy entities)
assumptions (4)
- domain assumption Syllogism (major premise, minor premise, conclusion) adequately represents legal reasoning for evaluation
- domain assumption NLI entailment scores measure the degree to which a cited law article or case supports a generated sentence
- domain assumption The aggregated Avg metric, a simple mean of MAUVE, ROUGE, BERTScore, Correct*, and Cita* scores, is a meaningful overall performance measure
- ad hoc to paper The corpus assembled from public legal datasets and legal LLM fine-tuning data provides comprehensive coverage without leakage into evaluated models
Cite this review
Pith. "Pith review of CitaLaw: Enhancing LLM with Citations in Legal Domain." pith.science (2026). https://pith.science/paper/V6AH6KCR
@misc{pith2026241214556,
author = {Pith},
title = {Pith review of: CitaLaw: Enhancing LLM with Citations in Legal Domain},
year = {2026},
howpublished = {\url{https://pith.science/paper/V6AH6KCR}},
note = {Machine review of arXiv:2412.14556}
}
read the original abstract
In this paper, we propose CitaLaw, the first benchmark designed to evaluate LLMs' ability to produce legally sound responses with appropriate citations. CitaLaw features a diverse set of legal questions for both laypersons and practitioners, paired with a comprehensive corpus of law articles and precedent cases as a reference pool. This framework enables LLM-based systems to retrieve supporting citations from the reference corpus and align these citations with the corresponding sentences in their responses. Moreover, we introduce syllogism-inspired evaluation methods to assess the legal alignment between retrieved references and LLM-generated responses, as well as their consistency with user questions. Extensive experiments on 2 open-domain and 7 legal-specific LLMs demonstrate that integrating legal references substantially enhances response quality. Furthermore, our proposed syllogism-based evaluation method exhibits strong agreement with human judgments.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 3 Pith papers
-
The Missing Link: Joint Legal Citation Prediction using Heterogeneous Graph Enrichment
A graph neural network that enriches legal citation graphs with categorical metadata nodes predicts case and law citations more accurately than prior GNN baselines, and joint training boosts case citation prediction.
-
LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documents
On CUAD legal contracts, a prompt-engineered QWEN-2 pipeline with chunking and two answer-selection heuristics reportedly outperforms the fine-tuned DeBERTa-large baseline by about 9%, reaching claimed state-of-the-ar...
-
Bridging Search and Recommendation through Latent Cross Reasoning
A latent cross reasoning model with contrastive learning and GRPO reinforcement learning improves search-enhanced recommendation on Qilin and KuaiSAR.
Reference graph
Works this paper leans on
-
[3]
(2) Lex- iLaw7 (6B): It specifically utilizes legal articles and legal reference books for training
B More Details of Evaluated Models and Datasets For the Legal LLMs, we choose (1) fuzi.mingcha (6B) (Wu et al., 2023a): It leverages unsupervised judicial corpora for training and uses syllogistic reasoning judgment data for fine-tuning. (2) Lex- iLaw7 (6B): It specifically utilizes legal articles and legal reference books for training. (3) Tailing8 (7B):...
work page 2023
-
[4]
arXiv preprint arXiv:2309.16289
Lawbench: Benchmark- ing legal knowledge of large language models. arXiv preprint arXiv:2309.16289. Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vin- cent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. 2023a. Rarr: Researching and revising what language models say, using language models. In Proceedings of the...
arXiv 2023
-
[5]
arXiv preprint arXiv:2311.03731
A survey of large language models attribution. arXiv preprint arXiv:2311.03731. Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yiqun Liu
-
[6]
Lexeval: A compre- hensive chinese legal benchmark for evaluating large language models. Preprint, arXiv:2409.20288. Chin-Yew Lin
-
[7]
Expertqa: Expert-curated questions and attributed answers. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3025–3045. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui
work page 2024
-
[9]
arXiv preprint arXiv:2306.09525
Explain- ing legal concepts with augmented large language models (gpt-4). arXiv preprint arXiv:2306.09525. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al
-
[11]
arXiv preprint arXiv:2309.07597
C-pack: Packaged resources to advance general chinese em- bedding. arXiv preprint arXiv:2309.07597. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Hao- ran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zho...
-
[12]
arXiv preprint arXiv:2407.10671
Qwen2 technical report. arXiv preprint arXiv:2407.10671. Fangyi Yu, Lee Quartey, and Frank Schilder. 2022a. Legal prompting: Teaching a language model to think like a lawyer. arXiv preprint arXiv:2212.01326. Weijie Yu, Zhongxiang Sun, Jun Xu, Zhenhua Dong, Xu Chen, Hongteng Xu, and Ji-Rong Wen. 2022b. Explainable legal case matching via inverse optimal tr...
Show all 16 references
-
[13]
Preprint, arXiv:2309.11325
Disc-lawllm: Fine-tuning large lan- guage models for intelligent legal services. Preprint, arXiv:2309.11325. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi
-
[14]
arXiv preprint arXiv:1904.09675
Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.09675. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun
1904 arXiv
-
[15]
Preprint, arXiv:2406.04614
Lawgpt: A chinese legal knowledge-enhanced large language model. Preprint, arXiv:2406.04614. A The Used Prompts Figure 5 illustrates the prompts used in this paper, including p1, p2, p3 in Eq. 1, Eq. 2 and Eq
-
[2018]
arXiv preprint arXiv:1810.04805
Bert: Pre-training of deep bidi- rectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge
-
[2019]
arXiv preprint arXiv:1908.10084
Sentence-bert: Sentence embed- dings using siamese bert-networks. arXiv preprint arXiv:1908.10084. Jaromir Savelka, Kevin D Ashley, Morgan A Gray, Hannes Westermann, and Huihui Xu
1908 arXiv
-
[2020]
In Proceedings of the 2020 con- ference on empirical methods in natural language processing: system demonstrations, pages 38–45
Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 con- ference on empirical methods in natural language processing: system demonstrations, pages 38–45. Shiguang Wu, Zhongkun Liu, Zhen Zhang, Zheng Chen, Wentao Deng, Wenhao Zhang, Jiyuan Yan...
2020
-
[2023]
arXiv preprint arXiv:2310.05620
Laiw: A chinese le- gal large language models benchmark (a technical report). arXiv preprint arXiv:2310.05620. Haolin Deng, Chang Wang, Xin Li, Dezhang Yuan, Jun- lang Zhan, Tianhua Zhou, Jin Ma, Jun Gao, and Ruifeng Xu
-
[2024]
arXiv preprint arXiv:2403.01774
Webcites: Attributed query- focused summarization on chinese web search results with citations. arXiv preprint arXiv:2403.01774. Jacob Devlin
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.