{"id":"51c19a5d-549b-4ade-939c-9cdd6a789d0e","arxiv_id":"2412.14588","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper introduces LJPIV, the first legal judgment prediction benchmark with innocent verdicts, and shows that trichotomous reasoning prompts and fine-tuning improve prediction of not-guilty outcomes.","lead":"This paper creates a legal judgment prediction benchmark, LJPIV, that includes innocent verdicts by modifying real case facts with LLM-generated justifications. It shows existing legal LLMs almost never predict 'not guilty', and that adding trichotomous reasoning (elements, unlawfulness, culpability) improves innocent-verdict prediction accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LJPIV's innocent cases cover only three affirmative-defense scenarios; if real acquittals are dominated by insufficient evidence, the reported gains on innocent verdicts may not transfer to actual legal settings.","rationale":"The reader's weakest assumption identified the general risk that LLM-generated counterfactual cases may be legally invalid or unrepresentative. My stress-test agrees with that direction but sharpens it to a specific, testable construct-validity gap: LJPIV's innocent class is built exclusively from three affirmative-defense-like scenarios and entirely omits insufficient-evidence acquittals, which are likely the dominant real-world route to innocence in Chinese criminal law. This is more load-bearing than generic label-correctness concerns because even perfect manual verification cannot fix the systematic exclusion of an entire innocence category. The concrete test—evaluating the same models on real acquittal judgments—would settle whether the benchmark measures a transferable capability. I keep the verdict UNCHANGED because the paper is an honest first step and its claims, while conditional, are not disproven by the existing evidence; an external validation set is needed before the benchmark can support strong conclusions about legal LLMs' innocence-prediction abilities.","tokens_in":14359,"tokens_out":7994,"duration_ms":56739,"concrete_test":"Construct an external validation set of real acquittal judgments from Chinese courts (e.g., sampled from public judgment databases such as裁判文书网), matched by charge distribution to LJPIV-CAIL. Have legal experts label each case's reason for acquittal: insufficient evidence, failure of elements, self-defense/necessity, or culpability. Run the Fine-Tuning-Tri and the strongest legal-LLM baselines on this real-acquittal set. If the model's F1 on real innocent cases is substantially lower than its LJPIV-CAIL score, or if a large fraction of real acquittals are insufficient-evidence cases that LJPIV never includes, then the benchmark's innocent class is not representative and the central claims are artifacts of the synthetic augmentation distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that legal LLMs lack trichotomous reasoning and that fine-tuning on LJPIV substantially improves innocent-verdict prediction—rests on the benchmark's innocent labels being a valid sample of real-world innocence. Section 3.2 constructs counterfactual innocent cases by injecting exactly three grounds for exoneration: failure of the elements of the offense (e.g., felling volume below the statutory threshold), self-defense or necessity at the unlawfulness level, and culpability defects (age, deaf/mute/blind, or mental illness). This construction omits the most common basis for acquittal in Chinese criminal practice: insufficient evidence or failure to prove guilt beyond a reasonable doubt. The 1:1 guilty-to-innocent ratio and the 3:1:1 split among innocence reasons are also arbitrary and do not reflect real base rates. The manual verification protocol described in Section 3.3 checks legal and logical consistency of the generated samples, but it cannot correct for representativeness. Consequently, a model that learns to recognize the specific injected scenarios—such as self-defense phrases or age statements—can achieve high scores on LJPIV without acquiring a general capability to identify innocence. The reported cross-domain improvements (e.g., LeCaRD F1 from 8.15 to 20.22 on only 80 test cases) may be driven by the benchmark's synthetic distribution rather than by trichotomous reasoning. This is not an internal inconsistency, but the external validity of the benchmark and the headline conclusions remain unestablished.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LJPIV, a Chinese legal judgment prediction (LJP) benchmark that extends CAIL, ELAM, and LeCaRD with LLM-generated counterfactual 'innocent' verdicts. The construction pipeline has three stages: sentence extraction for trichotomous reasoning, injection of exonerating grounds at the offense-element, unlawfulness, and culpability levels, and quality verification via LLM self-check plus manual inspection. The paper then proposes zero-shot and fine-tuning methods that force an LLM to answer three sequential trichotomous questions, and reports experiments on Qwen2-7B-Instruct and Baichuan2-7B-Chat. The main findings are that existing legal LLMs achieve F1 below 0.3 on LJPIV, while the proposed Fine-Tuning-Tri raises in-domain F1 to about 67 on LJPIV-CAIL and improves cross-domain F1 from about 18 to 23 on LJPIV-ELAM and from about 8 to 20 on LJPIV-LeCaRD.","tokens_in":14628,"tokens_out":6404,"duration_ms":48025,"significance":"If the LJPIV labels are trustworthy, the paper makes a useful and fairly original contribution: it is among the first LJP benchmarks to include non-guilt verdicts, and it provides sentence-level trichotomous annotations rather than only charge labels. The three-stage augmentation pipeline is nontrivial, the ablations in Section 5.3 show that both the unlawfulness and culpability levels contribute, and the reproduced gap between legal LLMs and the proposed fine-tuned models is large. The methodological strengths include explicit attention to over-modification, a RAG-based grounding of the counterfactual edits in legal articles, and a multi-round manual verification process. However, the significance is conditional on the external validity of the synthetic innocence distribution, the availability of the dataset, and the reproducibility of the label-quality claims, none of which is fully established in the current manuscript.","major_comments":[{"comment":"The innocent cases in LJPIV are constructed by injecting only three exonerating grounds: failure to satisfy offense elements, self-defense/necessity at the unlawfulness level, and three culpability scenarios. Real acquittals in Chinese criminal practice often rest on insufficient evidence or failure to prove guilt beyond a reasonable doubt, a category that cannot appear in LJPIV by construction. Consequently, the headline claim that existing legal LLMs 'automatically assign a charge' and cannot predict innocence, and the reported cross-domain gains on LJPIV-LeCaRD (F1 from 8.15 to 20.22 on 80 test cases), are established only with respect to the synthetic counterfactual distribution. The manuscript should either add a real-acquittal or human-written validation subset, or explicitly reframe the contribution as a stress-test benchmark for trichotomous reasoning rather than a representative sample of actual innocence.","section":"Section 3.2 and Section 5.1.1"},{"comment":"The manual verification protocol reports five rounds of random inspection (20% per round per annotator) with collaborative revision, but it does not report the number of samples rejected or revised in each round, any inter-annotator agreement statistic, or the legal qualifications of the annotators. The example of over-modification (a fracture incorrectly treated as a minor injury and changed to a scratch) shows that LLM-generated facts were materially edited during verification, yet the final dataset is not released: Table 5 lists URLs only for the source datasets and base models, not for LJPIV itself. Without releasing LJPIV and reporting label-quality statistics, the correctness of the benchmark cannot be independently audited.","section":"Section 3.3"},{"comment":"The fixed 1:1 guilty-to-innocent ratio and the 3:1:1 innocence-reason split are arbitrary design choices. Because the central negative result (legal LLMs with F1 below 0.3) is driven by the fact that these models never output the innocent class, the magnitude of the reported gap is sensitive to the innocence base rate; under a real-world distribution in which acquittals are rare, the comparison would look different. The paper should justify these ratios with reference to legal practice or report sensitivity analyses across different guilty-to-innocent ratios and innocence-reason distributions.","section":"Section 5.1.1"},{"comment":"The paper does not state which LLM was used for the counterfactual augmentation, nor whether that model is from the same family as the evaluated models. If the same model family (e.g., Qwen or Baichuan) generated both the training and test counterfactuals, a model fine-tuned on LJPIV-CAIL could be learning the generator's textual patterns rather than general trichotomous legal reasoning. Please report the augmentation model and include a robustness check in which test counterfactuals are generated by a different LLM or are replaced by human-written acquittal cases.","section":"Section 3.2 and Section 5.2"}],"minor_comments":[{"comment":"The naming is inconsistent: the text in Section 5.1.2 uses 'Fine-Tuning-Direct' and 'Fine-Tuning-Tri', but Table 3 and some later paragraphs use 'Fine-Tuing-Direct' and 'Fine-Tuing-Tri'; all occurrences should be unified.","section":"Table 3 and Section 5.1.2"},{"comment":"There are formatting errors in the table, e.g., '18.4619.80', '32.30 35.69', and '9,27'; these should be corrected and the table typeset consistently.","section":"Table 3"},{"comment":"The caption for example (b) says 'the victim x2 self-defensed and caused one wound in the chin,' but the fact description states that x1 grabbed the knife and stabbed x2; the actor and the grammatical subject should be clarified.","section":"Figure 2"},{"comment":"The evaluation section does not specify whether F1 is macro-averaged or weighted; this matters because the label space mixes a binary innocence decision with many charge classes, and the definition should be stated explicitly.","section":"Section 5.1.1"},{"comment":"The last column header is 'Avg_Sent_Num', but the surrounding text refers to 'Avg_Num_Sent'; the terminology should be made consistent.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The skeptical note's central concern is legitimate: the external validity of LJPIV is the main open question. In my reading, the paper's internal experiments are consistent with its stated claims, so this is not a case where the central derivation collapses. The key fixes are within scope: report the augmentation model, provide label-quality and inter-annotator statistics, release the dataset, and add a validation set that includes real acquittals or at least a distinct generator. If those additions are made, the benchmark could be a useful resource for the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this paper builds the first Chinese legal judgment prediction benchmark that includes not-guilty verdicts, and it shows that existing legal LLMs essentially cannot produce not-guilty predictions. That is a real blind spot, and the trichotomous reasoning framing (elements, unlawfulness, culpability) is a good match to civil law doctrine. The paper is worth reading and sending to review. But the benchmark's validity is not yet established, and the authors need to do more work on what the synthetic innocent cases actually represent.\n\nWhat is genuinely new: LJPIV is the first LJP benchmark with innocent labels and sentence-level trichotomous annotations. The motivating observation is solid — legal LLMs fine-tuned only on guilty cases have no route to an acquittal, and the paper documents that failure. The construction pipeline is described in detail: sentence extraction, RAG-based counterfactual generation, self-check, and manual verification. The experiments show a clear jump from fine-tuning directly (F1 about 61 on CAIL) to fine-tuning with trichotomous reasoning (F1 about 67), and the ablations confirm that each of the three levels contributes. That is a useful, honest experimental contribution.\n\nThe soft spots are mostly about external validity. The counterfactual generation injects exactly three kinds of innocence: failure of the elements (e.g., quantity below the statutory threshold), self-defense or necessity, and culpability defects (age, sensory disability, mental illness). The most common route to acquittal in real Chinese criminal practice — insufficient evidence or failure to prove guilt beyond a reasonable doubt — is absent. So a model can learn to recognize the injected signals (e.g., \"self-defense\" language) without acquiring a general ability to identify innocence. The 1:1 guilty/innocent ratio and the 3:1:1 split among innocence reasons are also arbitrary; they shape the reported metrics. The manual verification is described but no inter-annotator agreement or error rates are reported, and the dataset is not released. The cross-domain results rest on small test sets (80 cases for LeCaRD), so the cross-domain improvements are suggestive, not definitive. There are also no error bars or significance tests anywhere.\n\nNone of this is fatal to the conceptual contribution. The paper is transparent about its augmentation process, and it does not claim to cover all acquittal types. The limitations section is thin, though — it mentions jurisdiction scope and model scale, but not the representativeness gap in the counterfactual construction. That is the first thing I would ask the authors to address.\n\nWho benefits: legal NLP researchers working on judgment prediction, benchmark design, or counterfactual data augmentation. The paper deserves peer review, with the expectation that the authors release the dataset, report annotation quality metrics, and either broaden the innocence scenarios or clearly scope the benchmark's coverage. I would bring it to a reading group; the tension between synthetic convenience and legal validity is a good discussion topic. If my own work touched legal LJP, I would cite it for the benchmark idea, though not as a finished evaluation resource.\n\nNet: send it out. The idea matters and the execution is a reasonable first step, but it needs revision and data release before the claims can be fully accepted.","headline":"First benchmark with innocent verdicts for legal judgment prediction; useful direction, but the synthetic counterfactuals cover only a narrow slice of real acquittals and the dataset is not yet released.","tokens_in":15175,"tokens_out":3120,"would_cite":true,"duration_ms":24232,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current legal AI models cannot return a 'not guilty' verdict; this paper builds a benchmark and a three-step reasoning method that lets them.","keywords":["legal judgment prediction","trichotomous reasoning","innocent verdict","counterfactual augmentation","large language models","benchmark dataset","Chinese criminal law","retrieval-augmented generation"],"falsifier":"Have a fresh panel of criminal-law experts independently review every \"innocent\" case in LJPIV and compare its verdict with the benchmark label; if a non-trivial share of innocence labels is reversed, the benchmark's labels—and the measured gains built on them—do not hold.","tokens_in":14143,"feed_emoji":"⚖️","tokens_out":8525,"duration_ms":67022,"temperature":0.7,"pith_summary":"This paper claims that legal judgment prediction models fail in practice because they collapse the criminal-law decision into a single step: pick a charge. Standard criminal-law doctrine instead applies three sequential tests—whether the conduct satisfies the elements of the offense, whether there is a ground for justification such as self-defense, and whether the person is culpable—and a case that fails any one of them is innocent. To make this structure learnable, the paper constructs LJPIV, a benchmark that injects LLM-generated and manually verified \"innocent\" cases into three existing Chinese legal datasets. On that benchmark the best existing legal LLM scores below 0.3 F1, while a three-step prompt method and a fine-tuning variant raise in-domain F1 to about 67 and improve cross-domain verdict accuracy. The practical consequence is that an AI legal assistant that cannot return \"not guilty\" has little utility for real legal advice.","feed_headline":"Legal LLMs fail innocence tests; new method lifts F1 to 67","feed_subtitle":"A counterfactual benchmark plus three-step trichotomous reasoning lifts in-domain F1 from below 30 to about 67.","key_machinery":"The load-bearing mechanism is the trichotomous dogmatics of criminal law, a three-step decision procedure: first, does the conduct satisfy the elements of the offense; second, is there a ground for justification such as self-defense or necessity; third, was the person culpable, meaning of responsible age and sound mind. Failure at any step yields an \"innocent\" verdict. The paper operationalizes this as a prediction rule in which an LLM first predicts the charge or innocence, then answers yes or no at the unlawfulness and culpability levels, with any \"yes\" at those levels overriding the predicted charge. Around this rule the paper builds LJPIV, a dataset whose counterfactual innocent cases are generated by injecting justification grounds into extracted fact sentences via retrieval-augmented generation and then filtered by an LLM self-check plus five rounds of manual verification.","core_discovery":"The central discovery is that the inability of legal large language models to predict innocence is not a model-size problem but a missing-reasoning-structure problem. Trained only on guilty-verdict corpora, a model never sees an acquittal, so it maps every fact description to a charge and overlooks self-defense, necessity, age limits, or mental incapacity. The paper shows that if training data is augmented with counterfactual cases that fail one of the three trichotomous levels—and if inference is organized as sequential checks of charge, justification, and responsibility—open-domain LLMs improve sharply, with accuracy on innocent verdicts rising well above the overall accuracy. The 1:1 guilty-to-innocent split in the benchmark makes this behavior measurable, and ablations show that both the unlawfulness and the culpability check each contribute independently to the gains.","pith_inferences":["If LJPIV's labels generalize, the augmentation recipe could be ported to other legal families by replacing the three civil-law stages with the corresponding doctrine, for example precedent-based justifications in common law; the paper does not test this.","The benchmark's 1:1 guilty-to-innocent ratio is a design choice that inflates the apparent value of predicting innocence, since real dockets are guilty-heavy, so deployed systems would need recalibration to true base rates.","The paper's own limitations state that only 7B-parameter models and Chinese civil-law datasets were used, so claims about larger models or common-law jurisdictions remain open questions."],"forward_implications":["Legal LLMs fine-tuned only on guilty-verdict data cannot be trusted for innocence-sensitive tasks, and adding trichotomous reasoning stages is a direct fix rather than a prompt trick.","A three-step decision rule beats single-step prediction: Fine-Tuning-Tri raises in-domain F1 to about 67, and the ablation shows that removing the culpability check or the unlawfulness check each costs several F1 points.","Acquittal reasoning transfers across datasets: the same CAIL-fine-tuned model lifts cross-domain F1 from about 18 to 23 on ELAM and from about 8 to 20 on LeCaRD.","Zero-shot trichotomous prompting alone improves open-domain LLMs, raising the CAIL test accuracy of Qwen2 from 29.82 to 50.71, so even without fine-tuning the reasoning structure helps."],"supporting_citations":[{"why":"Supplies CAIL-2018, the in-domain guilty-verdict corpus that LJPIV-CAIL is built from and the source of the training split.","marker":"Xiao et al., 2018"},{"why":"Supplies ELAM, the cross-domain test set whose query cases are extended into LJPIV-ELAM.","marker":"Yu et al., 2022b"},{"why":"Supplies LeCaRD, the second cross-domain test set extended into LJPIV-LeCaRD.","marker":"Ma et al., 2021"},{"why":"DISC-LawLLM, the state-of-the-art legal LLM baseline that the paper shows fails on innocent cases because it was fine-tuned on guilty-only data.","marker":"Yue et al., 2023"},{"why":"LeCaRD-Elem provides the legal-element annotations used to train the sentence extractor that identifies trichotomous-relevant facts.","marker":"Deng et al., 2024"},{"why":"Retrieval-augmented generation retrieves the relevant criminal law used to prompt LLM-generated grounds for justification.","marker":"Lewis et al., 2020"},{"why":"LoRA is the efficient adaptation method used for both the sentence extractor and Fine-Tuning-Tri.","marker":"Hu et al., 2021"},{"why":"Qwen technical report; Qwen2-7B-Instruct is one of the two open-domain LLMs evaluated with zero-shot and fine-tuning methods.","marker":"Bai et al., 2023"}],"fun_headline_variants":["Trichotomous reasoning lifts legal AI's innocence F1 from 30 to 67","Legal LLMs flunk innocence cases; new method boosts F1","Beyond guilt: benchmark exposes legal AI's blindness to acquittals","Three-step legal reasoning helps models predict not-guilty verdicts","Counterfactual training teaches legal LLMs to say 'not guilty'"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole benchmark rests on the premise that the LLM-generated counterfactual cases, after five rounds of spot-check manual review, are legally valid and representative examples of genuine acquittals—if those innocence labels are wrong, every measured improvement is built on artificial data.","fun_headline_variants_meta":{"raw":{"variants":["Trichotomous reasoning lifts legal AI's innocence F1 from 30 to 67","Legal LLMs flunk innocence cases; new method boosts F1","Beyond guilt: benchmark exposes legal AI's blindness to acquittals","Three-step legal reasoning helps models predict not-guilty verdicts","Counterfactual training teaches legal LLMs to say 'not guilty'"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1569,"prompt_tokens":937,"completion_tokens":632,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":536}},"tokens_in":553,"tokens_out":632,"duration_ms":5645,"temperature":1.0,"reasoning_tokens":536,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:05:22.892103+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a fresh panel of criminal-law experts independently review every \"innocent\" case in LJPIV and compare its verdict with the benchmark label; if a non-trivial share of innocence labels is reversed, the benchmark's labels—and the measured gains built on them—do not hold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LeCaRD-Elem provides the legal-element annotations used to train the sentence extractor that identifies trichotomous-relevant facts."}],"review_version":1}