{"id":"284ecd9b-e6e2-47b0-bd30-00294717427b","arxiv_id":"2412.14556","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CitaLaw is a Chinese legal benchmark that tests citation-grounded answers for laypeople and legal practitioners, with a syllogism-based evaluation that shows substantial agreement with human judges.","lead":"This paper introduces CitaLaw, a benchmark for testing whether legal AI assistants can answer questions with proper citations to Chinese laws and court cases. It also proposes a syllogism-based scoring method, and reports that giving models legal references improves answer quality across nine language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Citation-quality metric is scored by a model whose fine-tuning data is partly inside the benchmark corpus and is selected after observing its agreement on the same benchmark, risking inflated entailment scores and overstated human agreement.","rationale":"I agree with the reader's weakest-assumption analysis: the load-bearing weakness is the use of DISC-LawLLM as the NLI scorer given that its fine-tuning data is part of the CitaLaw corpus and that it was selected after seeing its agreement on the benchmark. This creates a contamination and selection bias that directly affects the credibility of the citation-quality metric and the human-agreement claim. The paper does provide other evidence, such as ROUGE/BERTScore improvements and separate human kappa for component extraction (0.7876), but the 0.6923 kappa for citation entailment is the key empirical support for the proposed evaluation method. If this kappa is inflated, the central claim about the evaluation method's validity is not established. The concern is addressable—by reporting overlap statistics, by evaluating on a held-out legal set, or by fixing the NLI model a priori—so the verdict should remain CONDITIONAL rather than REJECT. The reader already recommended CONDITIONAL; I see no reason to change it, hence UNCHANGED. The concrete test I propose directly measures whether the reported agreement persists on non-contaminated data, which would settle the dispute.","tokens_in":16224,"tokens_out":5611,"duration_ms":40225,"concrete_test":"Using the publicly released fine-tuning data of DISC-LawLLM, check the overlap between that training set and the CitaLaw corpus (law articles and precedent cases). For the 50 human-annotated entailment pairs in §6.3, split them into those whose cited document (law article or precedent) appears in DISC-LawLLM's training data and those that do not. Compute Cohen's kappa between DISC-LawLLM and humans separately on each subset. If the kappa on the non-overlap subset drops substantially (e.g., below 0.6) while the overall kappa is 0.6923, the reported human agreement is inflated by training-set leakage. Alternatively, re-run the human study with a pre-registered NLI model that has no legal fine-tuning (e.g., a general-domain MNLI model) and report the kappa without post-hoc model selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central validation of CitaLaw's syllogism-based evaluation rests on the reported Cohen's kappa of 0.6923 between DISC-LawLLM's entailment scores and human judgments (§6.3). This validation is circular for two reasons. First, the CitaLaw corpus explicitly includes question-answer pairs from DISC-LawLLM's fine-tuning data as precedent cases (§3: 'These QA pairs are collected from DISC-LawLLM, LawGPT_zh, and HanFei'), so the model has likely seen the law articles, precedents, and even answer patterns used in the evaluation. Second, DISC-LawLLM was chosen as the NLI scorer specifically because it 'exhibits strong agreement with human evaluations (as discussed in Sec. 6.3)' (§5.3), meaning the model was selected post hoc after measuring its performance on the same benchmark. This selection-on-the-test-set procedure makes the reported kappa an optimistic upper bound. If the entailment metric is inflated, the paper's conclusion that 'integrating legal references substantially enhances response quality'—partly based on the CitaLaw citation-quality column in Tables 2 and 3—is weakened, and the benchmark's claim to provide a reusable, reasoning-aware evaluation is not substantiated. The concern is not about the novelty of the benchmark, but about the validity of its key evaluation signal.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CitaLaw, a Chinese-language benchmark for evaluating LLM-generated legal responses with citations. It contains two subsets (layperson and practitioner) and a large reference corpus of law articles and precedent cases, including QA pairs taken from legal LLM fine-tuning datasets. The authors propose two generation paradigms (citation-guided generation, CGG, and answer-refinement generation, ARG) and a two-level evaluation framework: global metrics (MAUVE, ROUGE, BERTScore) and a syllogism-inspired level that extracts circumstances, illegal acts, and legal decisions and scores citation entailment with an NLI model. Experiments over seven legal-specific and two open-domain LLMs are used to claim that incorporating legal references improves response quality and that the proposed syllogism-based evaluation agrees substantially with human judgments (kappa 0.7876 for component extraction and 0.6923 for citation entailment).","tokens_in":16488,"tokens_out":5826,"duration_ms":48957,"significance":"If the evaluation signal were validated, CitaLaw would be a useful first benchmark for citation-grounded legal QA: it separates layperson and practitioner needs, includes both law articles and precedent cases, and proposes a reasoning-aware alternative to surface-level citation metrics. The design of the CGG/ARG comparison, the breadth of the reference corpus, the use of temperature-0 generation, and the attempt to validate with human annotators are all strengths. However, the paper's central validation currently rests on a circular evaluator choice, on a corpus that contains training data of the scored models, and on aggregate comparisons without significance testing; these issues must be addressed before the benchmark's evaluation claims can be accepted.","major_comments":[{"comment":"The choice of DISC-LawLLM as the NLI scorer is circular. The CitaLaw corpus explicitly includes QA pairs from DISC-LawLLM's fine-tuning data as precedent cases (Section 3), and Section 5.3 states that DISC-LawLLM was selected because of its agreement with human evaluations on this same benchmark (Section 6.3). Since the scorer's training data overlaps the benchmark and the scorer was selected after observing its agreement on a subset of this benchmark, the reported kappa of 0.6923 and the entailment scores in Tables 2 and 3 are optimistic upper bounds. I would like to see the citation-quality evaluation repeated with a general-purpose NLI model not trained on CitaLaw's sources, and with a version of the corpus that excludes the DISC-LawLLM fine-tuning pairs, with a report of whether the CGG/ARG/CloseBook ranking is preserved.","section":"§5.3 with §3 and §6.3"},{"comment":"The corpus contains fine-tuning QA pairs from DISC-LawLLM, LawGPT_zh, and HanFei as precedent cases, and these exact three models are later evaluated (Section 6.1.1). Retrieval and generation for these models under CGG/ARG can therefore benefit from memorized training instances, confounding the claim that legal-specific LLMs perform well (for example, 'HanFei achieves the best results in the Practitioner dataset'). The authors should either remove each evaluated model's own fine-tuning data from the corpus during that model's evaluation, or demonstrate disjointness between the evaluated queries/cases and the models' training data.","section":"§3, Table 5, §6.1.1, §6.2.2"},{"comment":"The 'Avg' column averages twelve heterogeneous metrics (MAUVE, ROUGE-1/2/L, BERT-F, Correctc/a/d, CitaLaw, Citac/a/d) with different scales and no weights, and no error bars or significance tests are reported. Conclusions such as 'CGG achieves optimal performance' and 'LexiLaw achieves optimal performance overall' rely on small aggregate differences (for example, 53.82 vs 53.11 for Qwen2 in Table 2). Please report per-metric differences with bootstrap confidence intervals or paired significance tests, and either define a principled aggregation or remove the single 'Avg' column.","section":"Tables 2, 3 and §6.2"},{"comment":"The human-agreement validation is too thinly documented to support the claim of 'strong agreement.' Stage 2 uses only 50 randomly selected Practitioner questions, does not report inter-annotator agreement, and converts continuous model probabilities to a 1-5 scale by multiplying by 5 and rounding; it is also unclear whether the reported kappa is between the model and each annotator or between the model and pooled human labels. Please report the full annotation protocol, human-human kappa, model-human kappa with confidence intervals, and ideally include Layperson data in Stage 2.","section":"§6.3 and Appendix D"}],"minor_comments":[{"comment":"The benchmark and the global citation-entailment metric are both named 'CitaLaw', which is confusing; please rename one of them.","section":"Throughout"},{"comment":"The phrase 'A amount of work' is ungrammatical and should read 'A number of works' or similar.","section":"§2"},{"comment":"The section title 'Conclution' should be 'Conclusion'.","section":"§7"},{"comment":"The text 'For RGUGE' appears to be a typo for 'For ROUGE'.","section":"Appendix C"},{"comment":"The header 'Llam3-8B-Instruct' should be 'Llama3-8B-Instruct'.","section":"Table 2"},{"comment":"The extraction prompt used with Qwen2 for the three syllogism components is not provided in the appendix; including it would improve reproducibility.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful benchmark contribution, but the evaluation loop needs de-contamination before the central claims are credible. I would be willing to accept after the authors re-run the analysis with a held-out corpus, an alternative NLI scorer, and significance testing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the benchmark is real and mostly new, and the syllogism-based evaluation is a reasonable idea. But the paper's central citation-quality metric is scored by a model whose training data is partly inside the benchmark corpus, and the model was chosen after measuring its agreement on the same benchmark. That makes the reported kappa and the entailment scores optimistic, and the paper does not currently ship the data or code to check it.\n\nWhat's actually new: CitaLaw splits users into laypersons and practitioners, gives them different reference pools (law articles only vs. law articles plus precedent cases), and evaluates at both surface and syllogism levels (circumstances, illegal acts, legal decisions). That combination isn't in ALCE, WebCiteS, LawBench, or LexEval. The experiments on nine LLMs are reasonably broad, and the findings that CGG beats ARG on correctness and that open-domain models can beat legal-specific ones are coherent. The human evaluation with four legal annotators is a credible start: kappa 0.7876 for component extraction and 0.6923 for entailment, both in the substantial range. It's just a small sample (50 per stage).\n\nThe soft spots, in order of weight. First, the citation-quality scorer. DISC-LawLLM's QA pairs are explicitly added to the CitaLaw corpus as precedent cases (Section 3), and the same model is then used to score citation entailment and is selected because it 'exhibits strong agreement with human evaluations' on this benchmark (Sections 5.3, 6.3). That is selection on the test set, and it risks inflating both the entailment scores and the reported kappa. The stress-test's strongest formulation goes a bit far: the test questions themselves come from LawBench and LexEval, not from DISC-LawLLM's fine-tuning data, so this isn't direct answer leakage. But the corpus does contain that model's training examples, and no independent scorer or held-out calibration is provided. That is a real weakness, not a fatal one.\n\nSecond, the paper doesn't release data or code. Third, the main tables have no error bars or significance tests, and the Avg column mixes heterogeneous metrics. These are standard issues for a benchmark paper but they matter because the contribution is a measurement tool.\n\nThe limitations section is honest about the Chinese-only coverage and the simplification of legal reasoning into three syllogism components. The 'first benchmark' claim is plausible given the cited prior work.\n\nVerdict: the benchmark deserves a serious referee. The right outcome is probably a cautious 'revise and resubmit' rather than an accept. The fix is straightforward: release the data/code, use a held-out NLI scorer or at least re-validate with a model whose training data is disjoint from the corpus, and report variance. I'd bring it to a reading group once those are in.","headline":"A genuinely new legal-citation benchmark with a thoughtful layperson/practitioner design, but the citation-quality metric is scored by a model whose training data sits inside the benchmark corpus and which was selected on the same benchmark, so the reported agreement and entailment scores are optimistic.","tokens_in":17046,"tokens_out":3934,"would_cite":false,"duration_ms":30758,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CitaLaw is a benchmark that tests whether legal LLMs ground every sentence in a cited law article or precedent case, and its experiments show that adding citations improves answer quality while a syllogism-based scorer matches human…","keywords":["legal benchmark","citations","legal question answering","LLM evaluation","syllogism","retrieval-augmented generation","entailment","Chinese law"],"falsifier":"Check whether DISC-LawLLM's fine-tuning data overlaps with CitaLaw's corpus; if removing overlapping QA pairs from the benchmark noticeably changes the entailment scores, the conclusion that references improve citation quality is weakened. Alternatively, replace DISC-LawLLM with a scorer trained only on non-overlapping data and see if the human-agreement kappas still hold.","tokens_in":15969,"feed_emoji":"⚖️","tokens_out":3938,"duration_ms":31051,"temperature":0.7,"pith_summary":"CitaLaw is a new benchmark for evaluating whether large language models can produce legally sound answers with citations that support each sentence. It contains 1,000 questions split between laypersons and legal practitioners, paired with a reference corpus of roughly 500,000 law articles and precedent cases. The paper introduces a syllogism-based evaluation method that checks whether cited legal materials actually entail the response's circumstances, illegal acts, and legal decisions. Experiments across nine open-domain and legal-specific LLMs show that adding legal references improves response quality, and that the syllogism scoring agrees substantially with human judgments (Cohen's kappa 0.7876 for component extraction and 0.6923 for entailment). The authors argue this is the first benchmark designed specifically for citation-grounded legal question answering.","feed_headline":"Legal LLMs answer better when forced to cite law and precedent","feed_subtitle":"New benchmark scores citations with syllogisms and matches human judges (kappa 0.79 and 0.69).","key_machinery":"The key mechanism is the syllogism decomposition of legal reasoning into three components—major premise (law article or precedent case), minor premise (circumstances and illegal acts), and conclusion (legal decision)—combined with an NLI-based entailment scorer. The paper uses Qwen2 to extract these components and DISC-LawLLM as the NLI model to measure whether the cited reference entails the response component, producing scores such as CitaLaw, Citac, Citaa, and Citad. This machinery replaces surface overlap metrics with a logical consistency check.","core_discovery":"The central claim is that CitaLaw provides a reusable testbed for citation-grounded legal QA, and that a syllogism-inspired evaluation—treating the cited law article or precedent as the major premise, the case facts as the minor premise, and the legal decision as the conclusion—can reliably assess whether a response is legally coherent and properly supported by citations. The authors show that this syllogism-level scoring agrees strongly with human legal annotators, outperforming surface-level metrics like ROUGE in capturing logical alignment. They also find that incorporating references into the LLM's input (citation-guided generation) beats answer-refinement approaches on correctness, while refinement better aligns citations with sentences, and that open-domain LLMs like Qwen2 and Llama3 sometimes outperform older legal-specific models.","pith_inferences":["Because DISC-LawLLM's fine-tuning QA pairs are included in CitaLaw's corpus, using that same model as the NLI scorer may inflate entailment scores; a holdout evaluation with a non-overlapping scorer would test this directly.","The benchmark is built on Chinese law, so extending CitaLaw to other jurisdictions requires rebuilding the reference corpus, though the syllogism decomposition itself should transfer across legal systems.","A testable extension would vary the NLI scorer and measure human agreement for each, revealing whether the reported human-model agreement depends on the scorer's training data overlap.","The syllogism evaluation could generalize beyond law to any domain with a premise-evidence-conclusion structure, such as medical advice or financial claims, where citations must support each assertion."],"forward_implications":["If CitaLaw is accepted, citation-grounded legal QA can be evaluated on a standard 1,000-question benchmark with half a million reference documents, making model comparisons reproducible.","The syllogism-level metrics (circumstances, illegal acts, legal decisions) could replace or supplement ROUGE and BERTScore for judging legal reasoning quality.","The finding that citation-guided generation (references in the prompt) outperforms answer refinement on correctness suggests that legal RAG systems should pass retrieved texts into the LLM's context.","The finding that answer refinement better aligns citations with individual sentences suggests a hybrid approach could maximize both correctness and citation trustworthiness."],"supporting_citations":[{"why":"LawBench supplies the layperson consultation questions used in the Layperson subset.","marker":"Fei et al., 2023"},{"why":"LexEval supplies the practitioner open-ended questions from the National Uniform Legal Profession Qualification Examination.","marker":"Li et al., 2024"},{"why":"DISC-LawLLM contributes QA pairs to the precedent-case corpus and is used as the NLI scorer for citation quality.","marker":"Yue et al., 2023"},{"why":"BGE is the dense retriever used to select law articles and precedents for citations.","marker":"Xiao et al., 2023"},{"why":"Qwen2 is used to extract syllogism components (circumstances, illegal acts, legal decisions) from responses.","marker":"Yang et al., 2024"},{"why":"LeCaRD provides criminal precedent cases that populate the reference corpus for practitioner questions.","marker":"Ma et al., 2021b"}],"fun_headline_variants":["New benchmark tests legal AI citations with syllogisms","Legal AI scored on citation logic, matches human judges","CitaLaw: legal QA benchmark rewards cited answers","Syllogism-style scoring aligns legal AI with human experts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The citation-quality scores assume that the NLI model used to measure entailment has not already seen the benchmark's own questions and answers during training; if it has, the reported citation-quality improvements could be inflated.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark tests legal AI citations with syllogisms","Legal AI scored on citation logic, matches human judges","CitaLaw: legal QA benchmark rewards cited answers","Syllogism-style scoring aligns legal AI with human experts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1194,"prompt_tokens":835,"completion_tokens":359,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":294}},"tokens_in":451,"tokens_out":359,"duration_ms":3993,"temperature":1.0,"reasoning_tokens":294,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:06:45.937296+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check whether DISC-LawLLM's fine-tuning data overlaps with CitaLaw's corpus; if removing overlapping QA pairs from the benchmark noticeably changes the entailment scores, the conclusion that references improve citation quality is weakened. Alternatively, replace DISC-LawLLM with a scorer trained only on non-overlapping data and see if the human-agreement kappas still hold.","supporting_citations":[],"review_version":1}