{"id":"2921aa08-0c5c-4e3b-9af0-af01f2c57781","arxiv_id":"2505.14104","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The authors present LRI, a task and benchmark for inducing legal rules from analogous Chinese judicial precedents, and show that fine-tuning improves LLM performance significantly.","lead":"This paper formalizes legal rule induction, the task of deriving abstract legal rules from sets of analogous court cases, and introduces a large Chinese benchmark plus an expert-annotated test set. Fine-tuning small LLMs on this dataset nearly doubles their F1 scores, suggesting a path toward automated discovery of doctrinal principles from precedents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All F1 results rest on the DeepSeek-V3 judge, validated on only 114 decisions; since LRI-AUTO rules were generated by DeepSeek-R1, same-family stylistic bias could inflate the fine-tuned models' apparent advantage.","rationale":"In good faith, the paper makes a real contribution: a clearly defined task, a large-scale automatically constructed benchmark, a gold test set, and an iterative induction-verification method (SILVER). The fine-tuning results are large and directionally plausible. The central claim is not that LRI is solved, but that it is benchmarkable and that training on LRI-AUTO markedly helps. My stress-test focuses on the evidential basis for the headline numbers. All reported F1 scores are computed from DeepSeek-V3's binary 'logically equivalent' judgments. Any bias in those judgments is a direct threat to the central claim. The threat is concrete: the LRI-AUTO training labels were generated by DeepSeek-R1, and the judge is a sibling DeepSeek model. Fine-tuned models are likely to imitate the training data's style, while direct-prompted models are not. If the judge treats DeepSeek-like phrasing as equivalent more readily than other phrasing, the fine-tuned models' advantage is partly an artifact. The manual check of 114 decisions is not sufficient because it samples only 3 per model/setting and reports an aggregate accuracy. It cannot detect a bias that applies disproportionately to the LoRA models, which are exactly the models whose superiority is claimed. A per-model, larger-sample human audit is the natural settlement. Other potential concerns (gold-set inter-annotator agreement, benchmark release, pretraining contamination) are real but secondary: the gold set is expert-annotated with a systematic protocol, and contamination would likely favor the large closed-source models rather than the small fine-tuned ones. The judge-bias concern directly targets the comparative claim and is testable with a moderate annotation effort. Verdict: keep CONDITIONAL. The paper should not be accepted as-is with the reported numbers unless the judge bias is ruled out; but rejection is not warranted because the task and dataset are valuable and the concern is empirically checkable.","tokens_in":25184,"tokens_out":10765,"duration_ms":99978,"concrete_test":"Have three legal experts independently judge a random sample of predicted rules: e.g., 50 rules from each of the four LoRA fine-tuned models and 50 from each of the three strongest closed-source direct models (Claude-3.7-Sonnet, Gemini-2.5-Flash, DeepSeek-V3-0324) on the LRI-GOLD case sets, with model identities masked. Experts score each rule for logical equivalence to the gold rules. Compute Mic-F1 per model from human judgments and compare to the DeepSeek-V3 Mic-F1 for exactly the same rule samples. If the human-judged gap between the LoRA models and the direct closed-source models shrinks by more than ~3 F1 points, or if human-judged Mic-F1 for LoRA models is >5 points below the corresponding DeepSeek-V3 value, the judge-bias concern is confirmed. As a secondary check, recompute with a different judge (e.g., GPT-4o) and report the rank correlation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section C.3 states that all rule-level evaluations use DeepSeek-V3 as an automated judge to determine logical equivalence between predicted and gold rules, and that only 114 judge decisions (3 per each of 38 model/settings) were manually checked (Table 5). This validation is too small to detect per-model bias: for the four LoRA fine-tuned models (Table 3) only 12 judge decisions were audited. Because LRI-AUTO is generated by DeepSeek-R1 (Section 4.3) and the judge is DeepSeek-V3, the judge may systematically favor the rule style and phrasing produced by DeepSeek-family models. The fine-tuned 3-8B models are trained on DeepSeek-R1-generated rules, so they would inherit exactly that style; larger closed-source models prompted directly would not. Thus the headline advantage of fine-tuned small models (Mic-F1 70.9-73.2% vs. 66.7% for the best direct model) could be inflated by judge agreement bias rather than by genuine rule-induction ability. The paper's manual audit does not rule this out because it aggregates 114 decisions across all models and settings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes a new task, Legal Rule Induction (LRI), in which a model must derive concise normative rules from a set of analogous judicial precedents, structured as hypothetical condition, behavior pattern, and legal consequence. The authors construct LRI-AUTO, a large automatically built dataset of 5,121 case sets (38,088 Chinese cases) with rules extracted by DeepSeek-R1, and LRI-GOLD, a smaller expert-annotated test set of 216 case sets. They evaluate direct prompting, chain-of-thought, long-chain-of-thought, and an iterative induction-verification pipeline (SILVER) across several LLMs and reasoning models, and fine-tune four 3B-8B models with LoRA on LRI-AUTO. The central empirical claim is that LoRA fine-tuning raises small models to Micro-F1 scores of 70.96-73.18%, surpassing larger closed models under direct prompting. Rule-level evaluation uses DeepSeek-V3 as an automated judge for logical equivalence, with manual validation on 114 judge decisions.","tokens_in":25484,"tokens_out":5117,"duration_ms":48536,"significance":"The task definition and benchmark are a useful contribution: LRI-GOLD is expert-annotated, the three-element rule schema is principled, and the authors include human quality audits of both the automatic dataset and the judge, plus detailed prompts. The paper also makes several careful design choices, such as removing statutory citations from inputs and filtering rules by applicability. If the headline results survive a stronger judge-validation protocol, the finding that small fine-tuned models outperform much larger models on this inductive task would be practically meaningful. The main risk is not the task definition but the reliability of the automated judge for the central comparison.","major_comments":[{"comment":"The claim that fine-tuned 3-8B models outperform larger models (Section 5.2) rests entirely on the DeepSeek-V3 judge, but the manual validation of the judge is too thin to rule out a family-specific bias. Only 114 judge decisions were manually checked in total (3 per model/setting), and for the four LoRA models this amounts to 12 audited decisions. Because LRI-AUTO rules were generated by DeepSeek-R1 and the judge is DeepSeek-V3, models trained on LRI-AUTO may produce rules in a DeepSeek stylistic register that the judge is systematically more likely to declare logically equivalent to gold rules, inflating the reported Mic-F1 gains. The aggregated audit in Table 5 cannot detect such per-model bias. I request a per-model audit of judge decisions (especially for the four LoRA models), a comparison with a second judge from a different model family (e.g., GPT-4o or Claude), and a report of the distribution of judge verdicts with an error analysis.","section":"Section C.3, Tables 2 and 3"},{"comment":"The headline comparisons are presented without any measure of variability. Each F1 number appears to come from a single evaluation pass, and the fine-tuned models differ from each other by only 2-3 F1 points (e.g., 70.96, 71.70, 73.18). Without repeated runs, bootstrap confidence intervals, or a significance test, the paper cannot support the 'markedly enhances' claim beyond the large pre/post gap, and even that gap could be affected by judge noise. Please report variance over at least a few independent runs or a bootstrap over case sets.","section":"Section 5.2, Tables 2 and 3"},{"comment":"The rule-level micro-F1 treats each predicted rule as an independent unit, but the judge's 'logical equivalence' criterion is binary and the 'partial match' category is collapsed into incorrect. This makes the metric very strict and potentially sensitive to the rule set size. The paper should report the frequency of the judge's verdict categories (equivalent / partial / non-match) for at least a sample of models, and should state explicitly how partial matches are handled in the reported F1, since the judge prompt in Table 15 lists 'Partially Matches' but the scoring rule treats only 'Logically Equivalent' as correct.","section":"Equations (1)-(2), Table 15"},{"comment":"The LRI-AUTO filtering inherits errors from DeepSeek-R1's applicability assessments, since both explicit-rule 100% applicability and implicit-rule >50% applicability are judged by the same model that generated the rules. The human audit of 100 sets reports 93-95% field correctness, which is reassuring, but the audit is not tied to the filtering step and does not measure the impact of the residual noise on fine-tuning. Please provide a breakdown of the audit by rule type and report whether the cases where 'all fields are correct' fails are concentrated in implicit rules.","section":"Section 4.3, Section 4.4"}],"minor_comments":[{"comment":"The phrase 'over 76% gains' is ambiguous; it should specify whether this is the absolute percentage-point improvement or the relative improvement, because the numbers in Table 3 (e.g., 19.21 to 70.96) correspond to much larger relative gains.","section":"Abstract"},{"comment":"There is a typo in the caption: 'Webold the best' should read 'We bold the best'.","section":"Table 2 caption"},{"comment":"The labels 'TrainInfer' and 'Infer' in the bottom-right of Figure 2 are awkward; consider 'Training' and 'Inference' for clarity.","section":"Figure 2"},{"comment":"The formal definition says each rule must apply to 'strictly more than half' of cases; since case sets contain 5 to 10 cases, this is equivalent to a majority, but the wording could be simplified for readability.","section":"Section 3.1"},{"comment":"The paper does not state whether LRI-AUTO, LRI-GOLD, and the evaluation code will be publicly released; a data/code availability statement would improve reproducibility and is important for a benchmark paper.","section":"Dataset release"},{"comment":"Several bibliographic entries are incomplete or inconsistent, such as 'Mistral AI. Ministral 8b instruct.' missing a year, and some entries for 'Deepmind' and 'Anthropic' lack full publication details.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computational linguistics venue and the benchmark itself is a useful resource. The main obstacle is the reliability of the automated judge for the central comparison; if the authors can supply a stronger judge validation, the contribution would be solid. I would not reject on the current evidence, but the revision must address the judge reliability before the headline claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read this. First, it's the first serious benchmark for legal rule induction (LRI): case sets labeled with expert-annotated explicit and implicit rules, plus a large auto-curated training set. Second, all F1 numbers are mediated by a single LLM judge (DeepSeek-V3) whose equivalence decisions were spot-checked on only 114 rules, and the training rules were generated by DeepSeek-R1, so a same-family stylistic bias is a live concern.\n\nWhat's genuinely new: the task formalization (three-element rule structure: condition, behavior, consequence), the LRI-AUTO and LRI-GOLD datasets (38k Chinese cases in 5,121 auto-annotated sets, 216 expert-gold test sets), and the SILVER iterative induction-verification pipeline. The curation is careful: statutory citations are removed from input, PII is anonymized, explicit rules must apply to all cases, implicit rules to a majority, and a human audit of 100 auto sets reports 93-99% quality on several dimensions. That is real work and real evidence.\n\nThe experiments are informative, too: direct LLMs over-generalize and hallucinate; CoT mostly trades recall for precision; SILVER boosts recall; and LoRA fine-tuning on LRI-AUTO lifts 3B-8B models from around 19% to 71-73% Micro-F1, beating several larger closed models in the direct setting. That is the headline result.\n\nThe soft spots are mostly in evaluation. The judge is one model, DeepSeek-V3, and the manual validation is only 114 decisions across 38 model/settings — about 3 per condition, and for the four LoRA models it's 12 total. That is not enough to rule out per-model bias, and since LRI-AUTO rules were generated by DeepSeek-R1, fine-tuned models inherit DeepSeek-style phrasing that the judge may treat as logically equivalent more often. I don't think this kills the paper — the qualitative finding that fine-tuning helps is consistent across four models and multiple metrics — but the precise F1 deltas should be treated as approximate until there is a second judge or a larger human re-evaluation.\n\nAlso: the dataset and code are not released, which is a major reproducibility gap for a benchmark paper. And the abstract's 'over 76% gains' doesn't match the tables (best LoRA Mac-F1 is 70.73). Minor, but sloppy. The Chinese-only scope is acknowledged in the limitations.\n\nWho should read it: anyone working on computational law, legal NLP, or inductive reasoning in LLMs. It deserves a serious referee — I'd send it to review with a request for a second judge, a larger judge validation, and dataset release. Recommendation: engage with it, but push for those revisions before acceptance.","headline":"First real benchmark for legal rule induction, with a plausible but judge-dependent fine-tuning win; dataset not released.","tokens_in":25980,"tokens_out":3450,"would_cite":true,"duration_ms":35004,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-07T15:39:12.546920+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}