{"id":"e180977f-8d9a-4805-ab07-5ff5cba00f21","arxiv_id":"2608.08830","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PROSLEX provides 1,623 expert-annotated Indian legal documents with 7,450 statute-explanation spans and benchmarks LLM prompting strategies for statute prediction and explanation.","lead":"This paper introduces PROSLEX, a new dataset of 1,623 Indian Supreme Court case facts annotated with applicable criminal statutes and expert explanations. It then benchmarks several large language models on statute prediction and explanation, finding GPT-4 with chain-of-thought prompting performs best when explanations are required.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PROSLEX's headline count of 1,623 documents may double-count judgments that carry multiple IPC statutes; the paper neither states nor demonstrates that deduplication or split design prevents train-test leakage.","rationale":"The reader's weakest assumption was that the expert annotations are correct and that the selection criteria yield a representative distribution. My concern is different and more concrete: the dataset's instance accounting is ambiguous in a way that can directly change the headline numbers. The per-statute counts in Table 2 sum to exactly 1,623, which is consistent either with a single-label corpus or with double-counted multi-label judgments. The paper itself presents a multi-label task and a worked example with two statutes in one judgment, so the single-label reading is unlikely; the multi-label reading implies the true unique-document count is below 1,623. This is not merely a counting quibble: if the split was performed on statute buckets rather than on unique judgments, the same judgment can appear in both training and test sets, producing inflated F1 scores and invalid model comparisons. A simple repository check can settle this. I would not move the reader's conditional verdict to accept or reject without that check; the dataset may be salvageable by clarifying deduplication and re-splitting, but the current text leaves a serious unresolved risk. This does not challenge the authors' integrity; it identifies a missing specification in a central quantitative claim.","tokens_in":15289,"tokens_out":6079,"duration_ms":66038,"concrete_test":"Download the PROSLEX repository; for every instance record a canonical case identifier (e.g., Supreme Court citation, party names, or a normalized hash of the fact text). Compute (a) the number of unique identifiers per split and overall, (b) the overlap between per-statute buckets, and (c) the overlap between train, validation, and test sets. Check whether the sums in Table 2 match unique-document counts. If any identifier appears in more than one split, or the unique count is below 1,623, rerun the headline statute-prediction experiments with one instance per unique judgment and compare macro-F1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 reports per-statute document counts whose train/validation/test sums are 1,132/159/332, totaling exactly the claimed 1,623 documents. But the task is multi-label, and Figure 1's worked example (Bheemanna) annotates both IPC 147 and IPC 302 within one judgment. If a single judgment can give rise to multiple statute annotations, then a document is counted in multiple per-statute buckets and the number of unique underlying judgments is less than 1,623; alternatively, if the 1,623 entries are really document-statute pairs, the abstract's '1,623 expert-annotated legal documents' overstates the corpus. The paper never states that duplicates were removed or that the 70/10/20 split was applied to unique judgments rather than independently within each statute bucket. If a duplicated judgment lands in train under one section and in test under another, the reported F1 numbers include train/test leakage. This is load-bearing because every benchmark and model comparison rests on the integrity of the instance set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"PROSLEX is presented as a new expert-annotated dataset for legal statute prediction in the Indian context, containing 1,623 legal documents paired with statute labels for seven Indian Penal Code sections and 7,450 explanation spans. The dataset was created by legal experts who selected cases, extracted case facts, and annotated text spans as rationales, with a senior expert adjudicating low-agreement cases. The paper reports experiments comparing fine-tuned language models (LegalBERT, InLegalBERT, etc.) and large language models under zero-shot, few-shot, chain-of-thought, and tree-of-thought prompting for both statute prediction and explanation generation. The main findings are that fine-tuned InLegalBERT achieves the best statute-only macro F1 (0.82), while GPT-4 with chain-of-thought achieves the best prediction-with-explanation performance (0.75 macro F1) and the highest expert explanation rating (3.91/4).","tokens_in":15480,"tokens_out":8126,"duration_ms":73812,"significance":"If the dataset construction and instance counts are as claimed, PROSLEX addresses a genuine gap by providing expert-annotated explanations for statute prediction in Indian law, and the detailed annotation protocol with inter-annotator agreement and adjudication is a strength. The paper also provides a useful comparative benchmark of prompting strategies. However, the significance is conditional on resolving the ambiguity about whether the 1,623 documents are unique judgments or document-statute pairs; the lack of a clear statement on deduplication and split methodology currently undermines the integrity of all reported benchmark numbers.","major_comments":[{"comment":"The paper needs to clarify whether the 1,623 'documents' are unique judgments or document-statute pairs. The per-statute counts in Table 2 sum to exactly 1,623 (306+307+277+150+247+170+166), and Figure 1 shows one judgment annotated with both IPC 147 and IPC 302. If a judgment can be counted under multiple statutes, the total number of unique judgments is less than 1,623, and the abstract's claim of 1,623 expert-annotated legal documents is an overcount; if the 1,623 entries are actually document-statute pairs, the dataset is not organized as multi-label documents. The paper never states that duplicates were removed or that the 70/10/20 split was applied to unique judgments. If a duplicated judgment appears in training under one statute and in test under another, the reported F1 scores are contaminated by leakage. The authors must specify the instance definition and, if needed, rerun the evaluation on a deduplicated document-level split.","section":"Section 3.3, Table 2, Section 4"},{"comment":"The explanation evaluation compares free-text LLM rationales against expert-extracted text spans. This mismatch means lexical metrics like ROUGE and BLEU cannot distinguish a legally valid paraphrase from a poor one: a model that copies the span verbatim would score higher than a model that provides equivalent reasoning in different words. The paper does not discuss this limitation when interpreting the low lexical scores (e.g., GPT-4 ROUGE-L 0.31) or when ranking models. Since the paper's stated goal is to evaluate 'coherence and legal validity,' the authors should either adapt the evaluation to the extraction setting (e.g., ask models to select spans) or report how the lexical/semantic metrics correlate with the expert ratings and discuss the mismatch explicitly.","section":"Section 5.2, Table 5"}],"minor_comments":[{"comment":"The rating-score distributions sum to 49 for each model; specify the total number of rated explanations per model or correct the counts.","section":"Table 7"},{"comment":"The text mentions METEOR as a semantic metric, but Table 5 and the reported results use BLEURT; align the text and table.","section":"Section 5.2"},{"comment":"The description of the stratified split does not state the stratification variable; specify whether stratification was by statute label, document, or something else, and confirm that no document appears in more than one split.","section":"Section 4"},{"comment":"The calibration study for the ROUGE-L threshold used 50 pairs; report the 95% confidence interval or the number of adjudication decisions to support the 90% claim.","section":"Section 3.4"},{"comment":"The paper lacks a limitations section; add one discussing the restriction to seven IPC statutes, the use of only Supreme Court judgments, and the exclusion of cases where the statute was initially charged but not upheld.","section":"Section 6"},{"comment":"The related work references [33], which shares authors with this submission; consider clarifying the relationship and differences with that earlier 45-document study.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the instance definition. Please ask the authors to state explicitly whether the dataset contains unique judgments or document-statute pairs, and to provide the number of unique judgments if duplicates exist. If duplicates exist, the abstract's headline count and all evaluation results need to be redone. The paper's current ambiguity is serious enough that it should not be accepted without this clarification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a resource paper, and the resource looks real. The expert-annotated corpus of Indian Supreme Court facts with statute labels and highlighted rationale spans is new, larger than Vats et al.'s 45 documents, and public. Seven IPC sections, 1,623 documents, 7,450 spans, with a documented two-annotator plus senior-adjudication process, IAA via ROUGE-L, and a calibrated adjudication threshold. That is a solid contribution to legal NLP.\n\nThe empirical study is secondary and mostly fine as a first pass: InLegalBERT beats LLMs on statute-only F1, CoT helps GPT-4 produce better-rated explanations than ToT. The expert evaluation is small (49 per model) but not useless; the error analysis figure is genuinely informative.\n\nThe soft spots are real but not disqualifying. First, and load-bearing: the paper never states whether the 1,623 entries are unique judgments or document-statute pairs. Figure 1 shows one judgment annotated with two statutes. If a single judgment contributes to multiple per-statute buckets, then Table 2's sums double-count and the 70/10/20 split may put the same judgment in train and test under different statute labels, leaking. The paper needs an explicit deduplication statement or a demonstration that the split is at the judgment level. This is the first thing a referee should ask for.\n\nSecond, the automatic explanation metrics (ROUGE, BLEU) compare free-text LLM rationales to extracted expert spans, which is a format mismatch; the ROUGE-L numbers are predictably low and not very meaningful. The semantic metrics and expert ratings carry more weight, and those are okay.\n\nThird, the dataset only includes cases where the target statute was upheld, which biases the label distribution and excludes legally relevant contexts. That is a stated design choice, but it should be flagged as a limitation in the paper.\n\nThe citation pattern looks fine; self-citation to Vats et al. is appropriate given the direct lineage. I don't see any fabricated entities.\n\nWho this is for: people building or evaluating explainable legal AI for India, and dataset-construction researchers who want a worked example of span-level adjudication. It deserves a serious referee, conditional on the dedup/leakage question being resolved. I would send it out.","headline":"PROSLEX is a genuinely useful expert-annotated dataset for Indian statute prediction with explanations, but the paper must clarify whether the 1,623 documents are unique judgments or document-statute pairs before the benchmark numbers can be trusted.","tokens_in":16083,"tokens_out":2160,"would_cite":true,"duration_ms":21639,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PROSLEX provides 1,623 expert-annotated Indian judgments with statute labels and 7,450 reasoning spans, enabling evaluation of models that both predict laws and justify them.","keywords":["Statute Prediction","Large Language Model","Explanation","PROSLEX","Indian Penal Code","Chain-of-thought","Legal dataset","Multi-label classification"],"falsifier":"Re-annotate a random sample of 100 documents with independent legal experts from a different institution, compute label agreement (Cohen's kappa) and span overlap (ROUGE-L) against the gold standard; a large drop from the reported 0.79 mean ROUGE-L or low kappa would indicate the annotations are not reproducible.","tokens_in":15081,"feed_emoji":"⚖️","tokens_out":9193,"duration_ms":87836,"temperature":0.7,"pith_summary":"The paper argues that legal statute prediction should not stop at accuracy because a court decision must be explainable and justifiable. To make that measurable, it introduces PROSLEX, a dataset of 1,623 expert-annotated documents from Indian Supreme Court judgments in which legal experts linked each applicable statute to a highlighted span of text, producing 7,450 explanations in total. The paper then benchmarks fine-tuned legal language models and large language models under zero-shot, few-shot, chain-of-thought, and tree-of-thoughts prompting. The empirical findings are that fine-tuned InLegalBERT is the best statute-only predictor with a macro F1 of 0.82, while GPT-4 in the chain-of-thought setting is the best predictor-with-explanation on both F1 (0.75) and expert-rated explanation quality (3.91/4). The dataset and code are released publicly for reproducibility.","feed_headline":"1,623 Indian cases get expert-annotated law and reasoning labels","feed_subtitle":"Benchmark shows GPT-4 chain-of-thought explains statutes best; InLegalBERT wins prediction alone.","key_machinery":"The load-bearing object is the PROSLEX dataset itself: 1,623 Indian case documents, each stripped to a plain statement of the facts, paired with gold labels from seven criminal statutes (rioting, disappearance of evidence, murder, rape, cheating, cruelty by a husband, and criminal intimidation) and with the 7,450 text spans that legal experts marked as the reason a statute applies. Annotation was done in parallel by two legal experts per section, and a senior expert adjudicated whenever their span agreement, measured by ROUGE-L, a longest-common-subsequence overlap metric, fell below 0.75. This dataset is what makes the empirical comparison possible: fine-tuned transformers and in-context-learning LLMs are all scored against the same gold spans and labels. The prompting side of the machinery is conventional chain-of-thought and tree-of-thoughts in-context learning, with few-shot exemplars retrieved by semantic similarity between statute descriptions and annotated spans.","core_discovery":"The central claim is that an expert-annotated, explanation-bearing dataset at this scale did not previously exist for Indian statute prediction, and that its existence changes the evaluation question from 'which model gets the label right' to 'which model can also say why.' PROSLEX annotates facts drawn from Supreme Court judgments with the applicable Indian Penal Code sections and with the exact span of text that justifies each section, producing 7,450 explanations. Benchmarked on this gold standard, fine-tuned InLegalBERT is the most accurate statute-only predictor with a macro F1 of 0.82, while GPT-4 prompted with chain-of-thought leads when predictions must come with explanations, reaching a macro F1 of 0.75 and an average expert rating of 3.91 out of 4 for explanation quality. The paper also shows that tree-of-thoughts prompting reduces prediction accuracy, and that a model can pick the right statute while pointing to the wrong supporting text, so label accuracy and reasoning quality must be measured separately.","pith_inferences":["Extending PROSLEX beyond the seven IPC sections and beyond cases where the statute was upheld in court would test whether models handle dropped or amended charges, a setting closer to real litigation where charges are routinely reframed.","The near-perfect expert ratings for GPT-4 explanations despite low lexical overlap with the gold spans (ROUGE-L 0.31) suggest the experts are rewarding legal plausibility rather than textual fidelity; a follow-up could score explanations element by element (e.g., presence of actus reus and mens rea) to see which features drive the ratings.","The error-analysis finding that a model can predict the right statute while citing the wrong text could be turned into an element-level benchmark: for each statute, check whether the model's selected span actually satisfies the statutory ingredients, and report precision and recall over ingredients.","PROSLEX could also support retrieval-augmented explanation: using similar expert-annotated cases as grounding exemplars may improve both prediction and explanation for rare statutes, a testable extension the paper does not run."],"forward_implications":["Fine-tuned domain models (InLegalBERT at 0.82 macro F1) remain the strongest choice for pure statute classification when no explanation is required.","Chain-of-thought prompting of GPT-4 is the most promising configuration for generating statute predictions together with legally acceptable explanations, at 0.75 macro F1 and a 3.91/4 expert rating.","Tree-of-thoughts prompting degrades statute prediction in this domain, so branching-exploration strategies need careful adaptation before use in legal tasks.","Evaluation of legal AI should separate prediction correctness from reasoning correctness, since the paper demonstrates cases where the statute is right but the supporting span is wrong.","The public release of PROSLEX gives future work a fixed benchmark for comparing explainable statute-prediction systems on Indian law."],"supporting_citations":[{"why":"The 45-document prior study of LLMs for Indian statute prediction that PROSLEX scales to 1,623 documents with expert explanations.","marker":"[33]"},{"why":"Existing Indian legal statute identification dataset without expert explanations, establishing the gap PROSLEX fills.","marker":"[26]"},{"why":"Large Indian statute-identification corpus that also lacks explanation annotations, contrasting with PROSLEX.","marker":"[27]"},{"why":"InLegalBERT, the legal language model fine-tuned for the statute-only baseline and used to embed statute descriptions for exemplar selection.","marker":"[28]"},{"why":"LegalBERT, the other legal-domain baseline model evaluated for statute-only prediction.","marker":"[8]"},{"why":"Chain-of-thought prompting, the in-context-learning method that yields the best statute-plus-explanation results in the study.","marker":"[36]"},{"why":"Tree-of-thoughts prompting, the alternative reasoning strategy shown to underperform chain-of-thought on this task.","marker":"[40]"},{"why":"ROUGE-L, the overlap metric used both for inter-annotator agreement in adjudication and for lexical evaluation of generated explanations.","marker":"[20]"}],"fun_headline_variants":["New dataset pairs Indian case law with expert statute labels and reasoning","InLegalBERT tops statute prediction; GPT-4 leads with explanations on PROSLEX","Explaining the law: PROSLEX dataset benchmarks legal reasoning in AI","Indian legal AI benchmark: 7,450 expert explanations for statute prediction","PROSLEX: New benchmark for explainable statute prediction in Indian law"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark depends on the assumption that the expert-annotated statute labels and supporting text spans are correct and complete for the 1,623 selected cases, and that the selection of seven IPC sections and only upheld statutes is representative of the legal contexts the models will face.","fun_headline_variants_meta":{"raw":{"variants":["New dataset pairs Indian case law with expert statute labels and reasoning","InLegalBERT tops statute prediction; GPT-4 leads with explanations on PROSLEX","Explaining the law: PROSLEX dataset benchmarks legal reasoning in AI","Indian legal AI benchmark: 7,450 expert explanations for statute prediction","PROSLEX: New benchmark for explainable statute prediction in Indian law"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000615,"raw_usage":{"total_tokens":2888,"prompt_tokens":1007,"completion_tokens":1881,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":1794}},"tokens_in":623,"tokens_out":1881,"duration_ms":14675,"temperature":1.0,"reasoning_tokens":1794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:22:34.707464+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of 100 documents with independent legal experts from a different institution, compute label agreement (Cohen's kappa) and span overlap (ROUGE-L) against the gold standard; a large drop from the reported 0.79 mean ROUGE-L or low kappa would indicate the annotations are not reproducible.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The 45-document prior study of LLMs for Indian statute prediction that PROSLEX scales to 1,623 documents with expert explanations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Existing Indian legal statute identification dataset without expert explanations, establishing the gap PROSLEX fills."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Large Indian statute-identification corpus that also lacks explanation annotations, contrasting with PROSLEX."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"InLegalBERT, the legal language model fine-tuned for the statute-only baseline and used to embed statute descriptions for exemplar selection."}],"review_version":1}