{"id":"50f851e7-31d6-461c-a0c0-77d82a6bd06d","arxiv_id":"2502.05836","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"LegalSeg provides 7,120 Indian judgments annotated with seven rhetorical roles and benchmarks several models, reporting that context-aware sequence models work best.","lead":"LegalSeg is a new dataset of 7,120 Indian court judgments with over a million sentences labeled by rhetorical role, such as Facts, Arguments, Reasoning, and Decision. It is meant to give legal AI systems a much larger training and evaluation resource for reading and summarizing court rulings.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's headline metrics for the best model are contradicted by the paper's own Figure 3 confusion matrix; recomputing macro F1 from that matrix yields ~0.27, not 0.77, so the central benchmark claim is unsupported.","rationale":"The paper's two headline contributions are the LegalSeg dataset and the benchmark conclusion that hierarchical/sequential models outperform sentence-level classifiers. The benchmark conclusion rests on Table 3. One row in that table—the best-performing model—is irreconcilable with the paper's own confusion matrix for that model, and the matrix row sums are inconsistent with the dataset statistics. Since the ToInLegalBERT matrix matches Table 3, the discrepancy is not a harmless formatting issue. A reader cannot trust any of the comparative claims (e.g., 'models incorporating broader context outperform sentence-level features') when the top row cannot be reproduced. The missing IAA is a data-quality concern, but a model could be the best even on noisy labels; the internal contradiction invalidates the specific reported numbers regardless of label quality. Therefore the rejection is warranted. I agree with the reader's verdict, but their stated 'weakest_assumption' (IAA) is not the same as the decisive flaw; the decisive flaw is the numeric contradiction between the headline results and the paper's own confusion matrices.","tokens_in":17288,"tokens_out":6933,"duration_ms":53088,"concrete_test":"Recompute macro precision, recall, F1, accuracy, and MCC from every confusion matrix in Figures 3 and 4–13 (using per-class formulas P_i = diag_i / column_sum_i, R_i = diag_i / row_sum_i, then averaging), and compare against Table 3. Also verify each matrix row sum matches the per-label test-set counts in Table 2 (None 58,500, Facts 24,909, Issue 1,843, AoP 14,520, AoR 9,579, Reasoning 36,689, Decision 3,841). If the Hier_BiLSTM CRF metrics do not reproduce, Table 3 and the 'best model' claim must be corrected or withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is internal inconsistency in the empirical centerpiece. Section 7.1 and the abstract claim that the hierarchical BiLSTM-CRF 'achieves the highest overall performance' and that transition modeling beats independent classification. Table 3 reports for Hier_BiLSTM CRF macro precision 0.78, recall 0.77, F1 0.77, accuracy 0.62, MCC 0.68. But Figure 3, the paper's own confusion matrix for this model, gives different numbers: summing rows and columns and computing macro-averaged metrics yields macro precision ≈ 0.27, macro recall ≈ 0.28, macro F1 ≈ 0.27, and accuracy ≈ 0.39. One class alone, Issue, has only 84 correct predictions out of 1,843 test instances (recall 0.046), and the Decision row is also near-chance. By contrast, the ToInLegalBERT confusion matrix in Figure 6 does reproduce Table 3's reported macro F1 ≈ 0.62, making the BiLSTM-CRF mismatch look like a genuine error rather than a difference in averaging convention. Additionally, the BiLSTM-CRF matrix row sums disagree with the per-label test counts in Table 2 (e.g., 'None' row sums to 58,470 vs. 58,500; 'AoP' to 14,507 vs. 14,520; 'Reasoning' to 36,678 vs. 36,689), so the matrix is not even consistent with the stated test split. If the headline result cannot be reproduced from the paper's own artifact, the central empirical claim—and any conclusion that sequential/CRF models beat sentence-level classifiers—is unsupported. This is more decisive than the missing inter-annotator agreement: even under perfect annotation, the printed results do not cohere.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LegalSeg, a corpus of 7,120 Indian Supreme Court and High Court judgments annotated at the sentence level with seven rhetorical roles, and evaluates a range of classifiers, including a hierarchical BiLSTM-CRF, TransformerOverInLegalBERT, GNNs, role-aware transformers, and an instruction-tuned LLaMA-2 variant. The authors claim that LegalSeg is the largest dataset for rhetorical role classification in legal documents and that the hierarchical BiLSTM-CRF is the best model, leading to the conclusion that sequential and structural context improves rhetorical role classification.","tokens_in":17648,"tokens_out":11799,"duration_ms":100943,"significance":"If the reported results were reliable, the dataset would be a useful contribution roughly an order of magnitude larger than prior Indian rhetorical-role resources, and the systematic comparison of sequential, hierarchical, and sentence-level models would be informative. Strengths of the work include the stated intention to release data and code, the choice of open-source models rather than proprietary APIs, and the breadth of model families considered. However, the paper is not currently reliable: the advertised dataset size is internally inconsistent, the headline metrics in Table 3 cannot be reproduced from the paper's own confusion matrix in Figure 3, and further figure/table mismatches indicate that the problem is not a single typo. The absence of inter-annotator agreement statistics further weakens the dataset claim. I find no circularity in the evaluation itself; the labels are held out from model training and the benchmark comparisons are empirical.","major_comments":[{"comment":"The abstract, Section 1, and Table 1 state a total of 14,87,149 sentences for 7,120 documents, but the train, validation, and test sentence counts in Table 2 (11,22,507 + 2,93,370 + 1,49,881 = 15,65,758) sum to roughly 78,609 more sentences, a difference larger than the entire test set. This discrepancy is not cosmetic: it affects the per-document average and the claimed scale of the resource, and it must be resolved before the dataset contribution can be assessed.","section":"Abstract, Section 1, Table 1, Table 2"},{"comment":"Table 3 reports macro precision 0.78, recall 0.77, F1 0.77, accuracy 0.62, and MCC 0.68 for the Hierarchical BiLSTM-CRF model. Recomputing these metrics from the confusion matrix in Figure 3 gives macro precision approximately 0.27, macro recall approximately 0.28, macro F1 approximately 0.27, and accuracy approximately 0.39; the Issue class alone has only 84 correct predictions out of 1,843 test sentences (recall approximately 0.046). The row sums of Figure 3 also disagree with the test-set label counts in Table 2 (for example, None: 58,470 vs 58,500; AoP: 14,507 vs 14,520; Reasoning: 36,678 vs 36,689). This is not a harmless plotting issue: if Figure 3 is accurate, ToInLegalBERT, whose matrix in Figure 6 reproduces Table 3's macro F1 of approximately 0.62, clearly outperforms the BiLSTM-CRF, reversing the paper's central conclusion. The paper's main empirical claim is therefore unsupported by its own artifact.","section":"Section 7.1, Table 3, Figure 3"},{"comment":"The mismatch is not confined to the BiLSTM-CRF row. For the MTL model, Table 3 reports accuracy 0.41, while the confusion matrix in Figure 4 gives an overall accuracy of 73,471/149,881, which is approximately 0.49. For InLegalBERT(i-1, label_p, i), Figure 13 shows a degenerate model that assigns only two of the seven labels and has an empty Decision row, yet Table 3 reports F1 0.48 and Section 7.3 discusses the predicted-label variant as a meaningful comparison. These additional inconsistencies show that the numerical results in Table 3 cannot be treated as trustworthy unless every row is reconciled with its corresponding confusion matrix.","section":"Table 3 and Figures 4 and 13"},{"comment":"No inter-annotator agreement statistic is reported. The annotation was performed by ten law students with disagreements escalated to senior experts (Sections 4.3 and 4.4), but no kappa, alpha, or agreement rate is given, and Section 7.5 acknowledges persistent confusion between Facts and Reasoning and between Arguments of Petitioner and Arguments of Respondent. Since LegalSeg's value as a benchmark depends on label reliability, the absence of quantified agreement is a significant gap; the Limitations section's assertion that such issues are 'opportunities for future work rather than major limitations' does not address this gap.","section":"Sections 4.3, 4.4, and 7.5"}],"minor_comments":[{"comment":"The annotation schema descriptions are brief, and the overlap between Facts and Reasoning that the paper itself acknowledges in Section 7.5 is not addressed by the role definitions; please make the full annotation guidelines and any boundary-case instructions available with the dataset.","section":"Section 4.5"},{"comment":"The MTL loss L = lambda*L_shift + (1-lambda)*L_RR is not accompanied by the value or tuning procedure for lambda, which is needed because the auxiliary label-shift task is a central component of the reported MTL result.","section":"Section 5.3"},{"comment":"The pie chart in Figure 2 is difficult to read for the minority classes (Issue at 1.2% and Decision at 2.1%); a bar chart or a table with counts would convey the distribution more clearly.","section":"Figure 2"},{"comment":"The Indian lakh/crore comma formatting (14,87,149) is inconsistent with the conventional thousands separators used elsewhere and can confuse readers; please use a single formatting convention consistently.","section":"Tables 1 and 2"},{"comment":"The claim that predicted-label variants 'sometimes' outperform true-label variants is based on a single comparison in Table 3; please specify which comparisons support the claim and add significance testing or variance estimates.","section":"Section 7.3"}],"recommendation":"reject","confidential_remarks":"The Figure 3 versus Table 3 contradiction is decisive: the paper's headline benchmarking claim cannot be reproduced from its own artifact, and the dataset-size inconsistency affects the primary resource contribution. I would not send this back for minor edits; the required work is a full re-check of the experiments, metrics, and dataset statistics, followed by a revision of the central conclusions if the confusion matrices are correct."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LegalSeg is potentially a real resource: 7,120 annotated Indian judgments with 7 rhetorical roles is roughly 10x the largest Indian corpus in Table 1, and the authors release data and code. That part deserves credit. The model work is mostly re-application of known architectures, but running them all on a new large corpus is useful benchmarking.\n\nThe problem is the empirical centerpiece. Table 3 reports the hierarchical BiLSTM-CRF at macro precision 0.78, recall 0.77, F1 0.77, accuracy 0.62, MCC 0.68. The paper's own confusion matrix for that model (Figure 3) gives different numbers: recomputing macro metrics from the matrix yields F1 around 0.27 and accuracy around 0.39. The Issue class alone has only 84/1843 correct predictions. Row sums in the matrix also don't match the test-set counts in Table 2 (e.g., None sums to 58,470 vs 58,500). ToInLegalBERT's matrix does reproduce Table 3, so this is not a macro-averaging convention issue. As printed, Section 7.1's conclusion that sequential/CRF modeling beats independent sentence classifiers is unsupported by the paper's own artifact.\n\nOther soft spots are smaller but real. The abstract and Table 1 say 14,87,149 sentences, while Table 2's train/val/test totals sum to 15,65,758. No inter-annotator agreement is reported despite 10 law students doing the annotation; for a resource paper that is a meaningful omission. There is also no majority-class baseline; with None at roughly 48%, every result needs that context. The error analysis is honest about Facts/Reasoning confusion, which is good.\n\nI would not desk-reject this. The resource, if genuine and cleaned up, is valuable enough to consume referee time. But the paper in its current form should not be accepted: the central benchmark claim is internally contradicted, and the dataset documentation needs another pass. My recommendation is to send to peer review with the expectation of major revision, or to reject with an explicit path to resubmission after the tables/figures are reconciled, IAA is reported, and a majority baseline is added.","headline":"The dataset is a genuine asset, but the paper's headline benchmark claim does not survive contact with its own confusion matrix.","tokens_in":18236,"tokens_out":2411,"would_cite":false,"duration_ms":23379,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LegalSeg, the largest annotated dataset of Indian legal judgments for rhetorical role classification, contains 7,120 documents and 1,487,149 sentences labeled with seven roles, and a benchmark on it shows that a hierarchical BiLSTM-CRF…","keywords":["rhetorical role classification","legal NLP","Indian legal judgments","discourse segmentation","hierarchical BiLSTM-CRF","legal dataset","transformer baselines","LLM fine-tuning"],"falsifier":"Take a random sample of 200 LegalSeg documents, have two independent senior legal experts re-annotate every sentence with the same seven-role schema, and compute Cohen's kappa between the new labels and the released labels; a kappa below roughly 0.6 would indicate that the corpus is too noisy to support the paper's model rankings.","tokens_in":17089,"feed_emoji":"⚖️","tokens_out":12579,"duration_ms":111317,"temperature":0.7,"pith_summary":"The paper introduces LegalSeg, which it presents as the largest annotated dataset for rhetorical role classification in legal documents: 7,120 Indian Supreme Court and High Court judgments, split into 1,487,149 sentences, each labeled with one of seven roles (Facts, Issue, Arguments of Petitioner, Arguments of Respondent, Reasoning, Decision, None). Using this dataset as a benchmark, the authors compare sentence-level transformers, a hierarchical transformer over InLegalBERT, a graph neural network, and an instruction-tuned LLaMA, among others. Their central finding is that models that see document context and model label transitions—above all a hierarchical BiLSTM-CRF—outperform models that classify sentences independently. If the claims hold, the field gains a roughly twenty-fold larger Indian legal-role corpus and evidence that legal judgments have a recoverable discourse structure that sequence models can exploit.","feed_headline":"Largest Indian legal-role corpus: 7,120 judgments, 1.4M sentences","feed_subtitle":"Benchmark shows sequence-aware models beat sentence-by-sentence classifiers on court rulings.","key_machinery":"The load-bearing mechanism is the hierarchical BiLSTM-CRF: a BiLSTM that reads sentence embeddings (from sent2vec trained on Indian Supreme Court judgments) in document order, topped by a Conditional Random Field layer that scores the sequence of rhetorical-role labels as a whole. The CRF learns transition constraints such as 'an Issue is typically followed by Arguments, then Reasoning, then a Decision,' and the paper credits this explicit sequential modeling for the best benchmark results. The dataset itself, with its seven-role schema and its split into train/validation/test, is the other central object: it is what makes the model comparison possible and is the paper's main claimed contribution.","core_discovery":"LegalSeg is the largest annotated corpus for rhetorical role classification in legal documents, comprising 7,120 judgments and 1,487,149 sentences labeled with seven rhetorical roles. On this corpus, the hierarchical BiLSTM-CRF achieves the highest overall performance, with a macro-F1 of 0.77 and an MCC of 0.68, outperforming all transformer-based models, including ToInLegalBERT (F1 0.62) and the InLegalBERT variants, as well as a graph neural network (F1 0.54) and the instruction-tuned RhetoricLLaMA (F1 0.09). This establishes that modeling sentence order and label transitions is more effective than classifying each sentence in isolation. The benchmark also reveals that the None role dominates (48.5% of sentences) and that confusion between Facts and Reasoning and between petitioner and respondent arguments persists across models.","pith_inferences":["Because the dataset exposes the discourse skeleton of judgments, it could be reused as a pretraining or distant-supervision signal for other legal tasks the paper does not evaluate, such as summarization, legal question answering, and explainable judgment prediction.","The result that predicted labels rival true labels suggests that explicitly modeling label uncertainty during training—for example with a label-noise-aware loss—might push BiLSTM-CRF-type models further; the paper does not test this directly.","The BiLSTM-CRF's edge may depend on the corpus's long documents (averaging about 210 sentences); a reader could test whether a long-document transformer with a CRF head narrows the gap on subsets of shorter judgments.","A natural companion study would measure inter-annotator agreement on a sample of LegalSeg documents; such a kappa study would put the benchmark's reliability on a quantified footing."],"forward_implications":["LegalSeg gives researchers a corpus roughly twenty times larger than the previous largest Indian rhetorical-role dataset, enabling training of data-hungry models for legal NLP.","The benchmark establishes that sequence-aware models (BiLSTM-CRF) outperform independent sentence classifiers, meaning legal judgments have discourse structure that can be learned and exploited.","Adding neighboring-sentence context improves transformer baselines, and training with predicted previous labels matches or beats true labels, suggesting robust sequential training is viable for role labeling.","The persistent confusion between Facts and Reasoning, and between petitioner and respondent arguments, together with the 48.5% None class, defines concrete next targets: class-imbalance handling and finer role definitions.","RhetoricLLaMA's low F1 (0.09) indicates that open-source instruction-tuned LLMs are not yet competitive for this structured legal labeling task without further domain adaptation."],"supporting_citations":[{"why":"Defines the hierarchical BiLSTM-CRF architecture and the sent2vec sentence embeddings on Indian judgments that the best-performing model uses.","marker":"Bhattacharya et al. (2019)"},{"why":"Supplies the multi-task learning framework with label-shift prediction and the 13-role corpus that LegalSeg compares against.","marker":"Malik et al. (2022)"},{"why":"Provides the previous largest Indian rhetorical-role corpus (354 documents) and a transformer baseline, the main size comparison for the 'largest' claim.","marker":"Kalamkar et al. (2022)"},{"why":"Introduces the TransformerOverBERT architecture that ToInLegalBERT adapts, and provides Italian and Indian corpora used for comparison.","marker":"Marino et al. (2023)"},{"why":"Introduces InLegalBERT, the Indian legal-domain encoder that underlies ToInLegalBERT, the InLegalBERT variants, Role-Aware Transformers, and the GNN node features.","marker":"Paul et al. (2023)"},{"why":"Releases LLaMA-2-7B, the base model for RhetoricLLaMA; the paper's instruction-tuned variant builds directly on this checkpoint.","marker":"Touvron et al. (2023)"},{"why":"Defines the Matthews Correlation Coefficient used for model ranking; MCC is what makes BiLSTM-CRF the top model rather than ToInLegalBERT, which has higher accuracy.","marker":"Chicco and Jurman (2020)"}],"fun_headline_variants":["BiLSTM-CRF beats transformers on Indian legal role labeling","1.4M legal sentences labeled for rhetorical roles","Hierarchical BiLSTM tops legal text role classification","Indian court rulings annotated: 7,120 judgments, 1.4M sentences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark rests on the assumption that ten law students applied the seven rhetorical-role labels consistently across 7,120 documents, yet the paper reports no inter-annotator agreement score.","fun_headline_variants_meta":{"raw":{"variants":["BiLSTM-CRF beats transformers on Indian legal role labeling","1.4M legal sentences labeled for rhetorical roles","Hierarchical BiLSTM tops legal text role classification","Indian court rulings annotated: 7,120 judgments, 1.4M sentences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00097,"raw_usage":{"total_tokens":4113,"prompt_tokens":919,"completion_tokens":3194,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":3122}},"tokens_in":535,"tokens_out":3194,"duration_ms":24044,"temperature":1.0,"reasoning_tokens":3122,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:47:55.656302+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 200 LegalSeg documents, have two independent senior legal experts re-annotate every sentence with the same seven-role schema, and compute Cohen's kappa between the new labels and the released labels; a kappa below roughly 0.6 would indicate that the corpus is too noisy to support the paper's model rankings.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the hierarchical BiLSTM-CRF architecture and the sent2vec sentence embeddings on Indian judgments that the best-performing model uses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the previous largest Indian rhetorical-role corpus (354 documents) and a transformer baseline, the main size comparison for the 'largest' claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the TransformerOverBERT architecture that ToInLegalBERT adapts, and provides Italian and Indian corpora used for comparison."},{"cited_title":"Pre-trained Language Models for the Legal Domain: A Case Study on Indian Law","cited_arxiv_id":"2209.06049","evidence_quote":"Introduces InLegalBERT, the Indian legal-domain encoder that underlies ToInLegalBERT, the InLegalBERT variants, Role-Aware Transformers, and the GNN node features."}],"review_version":1}