{"id":"17212808-00d3-4ce8-aa0a-2d9a7804dc97","arxiv_id":"2505.21689","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A petition-ranking model that reports near-perfect accuracy, but its target ranking is derived from the same gap-days features it feeds the model, making the result circular.","lead":"This paper proposes a machine-learning system to prioritize legal petitions by urgency, using the ILDC dataset of Indian Supreme Court cases. The main result, that simple numeric features alone give near-perfect rankings, is undermined because those features are also used to define the ranking being predicted.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is invalidated by target-feature leakage: the target rank_score_inverse_square defined in Eq. 7 is a deterministic function of the input feature gap_days (Eq. 5), so the reported R²=0.988 and ρ=0.998 are tautological.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the ground-truth rank is a deterministic function of the gap_days feature, making the reported metrics tautological. My stress-test confirms this is not merely a potential issue but is directly visible in the manuscript: §3.3.3 defines the target via Eqs. 6–7, §3.3.3 and §3.3.4 include these same quantities as features, and §3.4 states that the numeric features concatenated with embeddings include 'gap days, rank score log'. Consequently, the central claim — that the proposed framework achieves near-optimal petition ranking and that LLM embeddings are only marginally useful — does not follow from the experiments. The additional issues (e.g., inconsistent preprocessing, 128-token truncation, unverified GPT4o extraction) are secondary; the leakage alone is sufficient to reject. The proposed concrete test directly removes the leaked features and would quantitatively confirm the collapse of the reported R² and ρ, settling the concern. The verdict should remain REJECT, as the paper's stated central finding is not a valid demonstration of petition ranking capability.","tokens_in":17023,"tokens_out":1568,"duration_ms":16741,"concrete_test":"Retrain the Random Forest and Decision Tree models on the exact same train/test splits, but remove all temporal features (gap_days, rank_score_log, rank_score_inverse_square) from the feature matrix, keeping only text embeddings and word count/sentence count. Then evaluate R², Spearman ρ, and Tol-10% accuracy against the same GPT4o-derived rank labels. If R² collapses from 0.988 to near zero, the original results are confirmed to be leakage-driven. Additionally, train a single-feature model using only gap_days and verify that it reproduces the Table 3 'Numeric only' row (R²≈0.988, ρ≈0.998); this demonstrates that the reported performance is a deterministic consequence of the feature-target identity, not a learned ranking behavior.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's headline result — that numeric features alone achieve R²=0.988 and Spearman ρ=0.998 — is not evidence of learning to rank petitions; it is an artifact of the target being defined as a transform of an input feature. Section 3.3.3 defines gap days (Eq. 5), rank_score_log (Eq. 6), and rank_score_inverse_square (Eq. 7). Section 3.3.3 lists all three as model features, and §3.3.4 confirms they are concatenated into the feature matrix. Section 3.2 step 4 then states that the ground-truth rank is prepared 'from the extracted numerical score from the text' — the same score. Thus, the model is asked to predict y = 1/gap_days² while receiving gap_days (and rank_score_log) as inputs. Any reasonably flexible regressor, especially Random Forest or Decision Tree, can fit this deterministic inverse-square mapping almost exactly, producing the near-perfect R² and Spearman values. The accuracy metric (Eq. 16, Tol-10%) also becomes trivial: with gap_days as a feature, the model can predict y to within 10% almost always, explaining the 99%+ 'accuracy'. The claim that 'LLM-based embeddings offer only marginal gains' is also expected under this construction: the numeric feature already contains the target, so text embeddings are irrelevant. The paper's central contribution — that automated petition ranking can reduce backlog — is therefore unsupported. Even if the rank score were omitted from the features, using gap_days alone would still allow near-perfect prediction because the target is a deterministic monotone function of it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLMPR, a framework for ranking accepted legal petitions by urgency. It uses the ILDC dataset of 7,593 Indian Supreme Court petitions, extracts text embeddings from several transformer models (DistilBERT, LegalBERT, MiniLM, FLAN-T5, E5, etc.), and augments them with numeric features—gap days, rank-score log, rank-score inverse square, word count, and sentence count—to train regression models (RF, DT, XGBoost, LightGBM, CatBoost, ElasticNet, Linear Regression). The ground-truth rank is derived from gap days via Eq. (7). The paper reports that RF and DT achieve above 99% accuracy and Spearman correlation 0.99, and that numeric-only features achieve R²=0.988 and ρ=0.998, with LLM embeddings adding little. It concludes that automated ranking can reduce judicial backlog.","tokens_in":17417,"tokens_out":5440,"duration_ms":45986,"significance":"The paper addresses an important practical problem—automated prioritization of accepted legal petitions to reduce court backlog—and it takes some laudable steps: it respects the predefined ILDC train/test split, verifies split non-overlap via TF-IDF cosine similarity, and makes evaluation artefacts publicly available in a Zenodo repository. However, the central experimental design contains a direct target-feature leakage: the ground-truth rank is defined as a transform of gap days, and gap days together with the derived rank scores are used as model features. The reported near-perfect performance is therefore predetermined and does not support the paper's claims about ranking quality or the marginal utility of LLM embeddings. Because this flaw invalidates the main conclusion, the paper cannot be accepted in its current form. If the authors were to redo the study with a target that is not a deterministic function of the features (e.g., predicting court delay or a human-annotated urgency label), the results might be informative, but that would constitute a new study.","major_comments":[{"comment":"The target variable is a deterministic function of an input feature. The ground-truth rank is prepared in §3.2 step 4 'from the extracted numerical score from the text'; §3.3.3 defines gap days (Eq. 5), rank_score_log (Eq. 6), and rank_score_inverse_square = 1/gap_days² (Eq. 7); and §3.3.4 concatenates all three into the feature matrix F (Eq. 8). Consequently, the model receives gap_days as an input and is asked to predict 1/gap_days², so the reported R²=0.988 and ρ=0.998 in Table 3 and the abstract are tautological. The Tol-10% accuracy (Eq. 16) is also trivially satisfied by any model that fits the inverse-square mapping. The claim that 'numeric features alone achieve nearly optimal ranking results' is therefore an artifact of the experimental design, not evidence that the model has learned to rank petitions by urgency.","section":"§3.3.3–3.3.4 and §3.2 step 4, Eqs. (5)–(7)"},{"comment":"Tables 3 and 4 report irreconcilable numbers for the same models. Table 3 (LightGBM, numeric-only) reports R²=0.988, MSE=4.012e-5, and ρ=0.998, whereas Table 4 reports for LightGBM MSE=0.002 and R² in the range 0.001–0.007 across all embedding configurations (e.g., DistilBERT R²=0.001, LegalBERT R²=0.006). This discrepancy is not explained by the different feature sets, since the numeric-only configuration should be a subset of the combined configuration and cannot yield a 10⁴-fold larger MSE. The inconsistency undermines the reliability of the reported performance comparisons.","section":"Tables 3 and 4"},{"comment":"Table 4 shows negative R² values for Random Forest (–0.072 for DistilBERT; –0.002 for LegalBERT, MiniLM, Flan-T5, E5) and Decision Tree (–0.225 for DistilBERT; –0.002 for the others), while the text (§4.2, Fig. 5a) describes these models as achieving the best Spearman correlations (≈0.99) and 'superior performance.' Negative R² means the predictions are worse than predicting the constant mean on the squared-error scale; the paper does not reconcile this with the high Spearman values or explain how the ranking is 'near-perfect' when the regression is miscalibrated. This casts doubt on the evaluation protocol, including how the target is scaled and how the Spearman correlation is computed.","section":"Table 4, §4.2"}],"minor_comments":[{"comment":"The preprocessing description is contradictory: §3.3.1 states that stop words, non-alphanumeric characters, and punctuation are removed and stemming/lemmatization is applied, while §3.4 states that 'Petition texts were used as-is, without lowercasing, lemmatization, or stopword removal.' Please clarify which pipeline was actually used.","section":"§3.3.1 vs §3.4"},{"comment":"The section 'Embedding Pre-processing and Representation' is immediately followed by a second '3.4 Embedding Generation' subsection; renumber the sections and consolidate the embedding description.","section":"§3.4 (duplicated section)"},{"comment":"Table 5 is captioned 'Classification Metrics' but the task is regression; specify how 'Test Accuracy' and 'KFCV's Accuracy' are defined, presumably via Eq. (16), and relabel the caption accordingly.","section":"Table 5 caption"},{"comment":"Equation (4) is missing the fraction bar: L_contrastive = -log( exp(sim(ei,ej)) / sum_k exp(sim(ei,ek)) ); please typeset the formula correctly.","section":"Eq. (4)"},{"comment":"The contribution list in Section 1 contains the typo 'rank soore,' and the affiliation should read 'Indian Institute of Information Technology Guwahati' rather than 'India Institute of Information Technology Ghuwahati.'","section":"Section 1 and affiliations"},{"comment":"Table 3 introduces RoBERTa, Instructor XL, and FLAN-T5, but §3.3.2 only describes DistilBERT, MiniLM, Flan-T5, LegalBERT, and E5; list all models used and their sources consistently.","section":"Tables 3 and 4 model lists"}],"recommendation":"reject","confidential_remarks":"This manuscript has a fundamental target-feature leakage problem that cannot be fixed by local revision; the experiments would need to be redesigned with a different target variable. The inconsistency between Tables 3 and 4 also suggests that the computational results should be re-examined for data-handling errors before any future submission. I would not invite resubmission unless the authors substantially change the task definition and rerun all experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the central claim is invalidated by target-feature leakage. The target rank_score_inverse_square is defined in Eq. 7 as 1/gap_days², and gap_days and rank scores are used as model features (Sec. 3.3.3 and 3.3.4). Section 3.2 step 4 then prepares the ground-truth rank from that same extracted numeric score. So the model is predicting a deterministic function of one of its own inputs. The near-perfect R² and Spearman values in the abstract and Table 3 are artifacts, and the finding that embeddings add little follows directly from the leakage: the numeric feature already contains the target.\n\nWhat is genuinely decent here: the problem is real and important, the paper uses a public corpus (ILDC) plus several off-the-shelf embeddings and tree-based regressors, and the related work on e-petition decision support is reasonable. A clean version of this study would have been worth reading. The negative result that procedural metadata beats semantics could have been useful.\n\nBut the manuscript is not clean. Beyond the leakage, it contradicts itself. Section 3.3.1 describes heavy preprocessing (stopword removal, stemming, lemmatization), while Section 3.4 says texts were used as-is. Truncating 20,000+ word petitions to 128 tokens means the \"contextual urgency\" the title claims to capture is mostly discarded. And Table 4 reports negative R² values for most model/embedding combinations alongside MSE 0.002, which is hard to square with the R²=0.988 in Table 3. The GPT-4o date extraction is unverified, and the 10% relative-error accuracy metric becomes trivial once gap_days is a feature. The paper points to a Zenodo repository for evaluation artefacts, but no code is included.\n\nSo the central claim does not hold up. If the authors removed the leak—for example, by withholding gap_days from the features or using an externally assigned priority label—the experiment might work. As it stands, readers should not take the reported rankings as evidence that the system can prioritize petitions.\n\nMy take: it is a useful cautionary example for a research group, but it should not be published as is. I would not send this to peer review in its current form; the leakage is load-bearing and the internal inconsistencies would waste referee time. A corrected version might be worth a serious look.","headline":"The headline result is a tautology: the target rank is a deterministic function of a feature, so the reported R² and Spearman values do not measure genuine ranking ability.","tokens_in":17939,"tokens_out":3433,"would_cite":false,"duration_ms":31870,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that accepted legal petitions can be ranked by urgency almost perfectly using procedural timing features alone, with LLM text embeddings adding only marginal gains.","keywords":["petition ranking","legal text classification","ILDC dataset","LLM embeddings","transfer learning","judicial backlog","Spearman rank correlation","Random Forest"],"falsifier":"Retrain the same models with $\\text{rank\\_score\\_inverse\\_square}$ as the target but with the gap-days and rank-score features removed, leaving only text embeddings and word or sentence counts as inputs: if $R^2$ and Spearman $\\rho$ collapse toward zero, the headline results depended on handing the model the very quantity it was asked to predict. Separately, test the predicted ranks against an urgency signal not derived from gap days — for instance, time from first hearing to final disposal, or priority ratings assigned by judges — and check whether the Spearman correlation survives; if it does not, the ranking does not track real urgency.","tokens_in":16721,"feed_emoji":"⚖️","tokens_out":15852,"duration_ms":132817,"temperature":0.7,"pith_summary":"The paper proposes LLMPR, an automated pipeline that assigns priority rankings to accepted legal petitions so that courts can attend to urgent cases first and reduce the backlog that plagues the Indian judiciary, where roughly 30 million cases are pending. Using the ILDC corpus of 7,593 annotated petitions, the authors combine text embeddings from models such as DistilBERT, LegalBERT, and MiniLM with numerical features — gap days, rank scores, word counts, and sentence counts — and train regression models to predict an urgency score. The paper's central finding is that the numerical features do almost all of the work: models trained on numbers alone reach $R^2 = 0.988$ and a Spearman rank correlation of $0.998$, while LLM embeddings add at most marginal gains, and Random Forest and Decision Tree models exceed 99% test accuracy. If this finding holds, a court could build a fast, cheap triage system without expensive language-model inference, because urgency would be readable from procedural timing rather than semantic content. The authors position the contribution as a way to remove manual bias from prioritization and to streamline judicial workflows, while acknowledging the system has so far been tested only on English-language Indian petitions and offers no explanations for its decisions.","feed_headline":"Case timing, not LLM text, powers near-perfect petition ranking","feed_subtitle":"A 7,593-petition study suggests courts could triage urgency from procedural timing alone.","key_machinery":"The load-bearing object is the gap-days construction: both the ground-truth rank and the dominant model features are derived from the time between a petition's acceptance and its first hearing. The pipeline computes $\\text{gap days} = |\\text{date}_{\\text{proceeding}} - \\text{date}_{\\text{acceptance}}|$, then forms the rank targets $\\text{rank\\_score\\_log} = \\log(1 + \\text{gap days})$ and $\\text{rank\\_score\\_inverse\\_square} = 1/\\text{gap days}^2$, and then feeds the same temporal quantities — along with word count, sentence count, and concatenated transformer embeddings — into tree-based regressors such as Random Forest and Decision Tree. A three-stage zero-shot GPT-4o prompting pipeline extracts the acceptance and first-hearing dates from the unstructured petition text and computes these derived features. The experimental comparison rests on this construction: the ablation contrasts numeric-only models against models that add mean-pooled embeddings from LegalBERT, DistilBERT, and MiniLM, and the paper interprets the small gap between them ($R^2$ gains no larger than about 0.002) as evidence that urgency is carried by structural and temporal attributes instead of semantics.","core_discovery":"The paper claims that petition urgency can be captured almost entirely by procedural metadata, so that ranking accepted petitions reduces to a regression problem on a small set of numerical features. Its ground truth is a rank score built from the time gap between a petition's acceptance and its first scheduled hearing: the authors compute $\\text{gap days} = |\\text{date}_{\\text{proceeding}} - \\text{date}_{\\text{acceptance}}|$, then derive $\\text{rank\\_score\\_log} = \\log(1 + \\text{gap days})$ and $\\text{rank\\_score\\_inverse\\_square} = 1/\\text{gap days}^2$. A three-stage zero-shot prompting pipeline based on GPT-4o extracts the two dates from unstructured petition text, computes these scores, and adds word and sentence counts; the resulting numerical features are concatenated with mean-pooled transformer embeddings and fed to Random Forest, Decision Tree, XGBoost, LightGBM, CatBoost, ElasticNet, and linear regression. On the predefined test split, the authors report that numeric-only models achieve $R^2 = 0.988$ and Spearman $\\rho = 0.998$, that Random Forest and Decision Tree reach Spearman correlations near 0.99 with test accuracy above 99%, and that transformer embeddings shift $R^2$ by no more than about 0.002. The paper concludes that urgency in petitions is encoded in structural and temporal attributes rather than semantic content, and presents LLMPR as an automated decision-support tool that could streamline judicial workflows and reduce case backlog.","pith_inferences":["A step beyond the paper: because the target in Equations 5-7 is a deterministic function of the gap-days feature that is also fed to the model, the reported $R^2 \\approx 0.988$ and $\\rho \\approx 0.998$ largely measure how exactly the regressor reconstructs the label formula from its own input — they do not, by themselves, establish that the ranking predicts real-world urgency.","The paper's practical value therefore depends on an untested empirical question: whether time-to-first-hearing is actually correlated with urgency; a validation study comparing LLMPR ranks against judge-assigned priority or time to final disposal would settle it.","Because the dominant features are extracted by an LLM (GPT-4o), the 'LLM-driven' description applies mainly to the metadata pipeline rather than the ranking itself; a rule-based date parser might recover most of the same signal, a comparison the paper does not run."],"forward_implications":["A court could deploy the framework as decision support to surface urgent accepted petitions first, directly attacking the roughly 30-million-case backlog the paper cites.","Expensive LLM inference is unnecessary for the ranking step, since numeric-only models already reach $R^2 = 0.988$ and Spearman $\\rho = 0.998$; resource-constrained judiciaries could use simple tabular models instead.","Future ranking systems should invest feature engineering in procedural timelines and document statistics rather than semantic embeddings, following the paper's conclusion that urgency lives in structural attributes.","Tree-based models are the right model class for this task, while linear models (ElasticNet, Linear Regression) fail — the paper reports Spearman correlations as low as $-0.338$ for ElasticNet."],"supporting_citations":[{"why":"Supplies the ILDC corpus of 7,593 annotated Indian Supreme Court petitions that every trained model is evaluated on.","marker":"[13]"},{"why":"The GPT-4o system card behind the three-stage zero-shot prompting pipeline that extracts acceptance and first-hearing dates and computes the gap-days and rank-score features.","marker":"[41]"},{"why":"Defines DistilBERT, one of the text-embedding models whose marginal gains over numeric-only features are measured in the ablation.","marker":"[42]"},{"why":"Defines MiniLM, the sentence-level embedding model whose vectors are concatenated with the numerical features in the evaluated framework.","marker":"[44]"},{"why":"Defines LegalBERT, the domain-specific legal embedding that achieves the near-0.99 Spearman correlations reported in the results.","marker":"[46]"}],"fun_headline_variants":["Timing alone near-perfect for petition ranking, LLM adds little","Case timing, not LLM text, drives petition urgency ranking","LLMPR finds procedural timing outperforms semantic embeddings","Skip LLM? Court timing predicts petition priority with 0.998 rho","Petition triage: numeric timing beats transformer embeddings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a rank score computed from the time gap between a petition's acceptance and its first hearing is a valid measure of genuine urgency, and that it can simultaneously be used as a model feature without making the prediction task circular; if the second half of that premise fails, the near-perfect scores are predetermined and do not demonstrate an ability to rank petitions by real urgency.","fun_headline_variants_meta":{"raw":{"variants":["Timing alone near-perfect for petition ranking, LLM adds little","Case timing, not LLM text, drives petition urgency ranking","LLMPR finds procedural timing outperforms semantic embeddings","Skip LLM? Court timing predicts petition priority with 0.998 rho","Petition triage: numeric timing beats transformer embeddings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000321,"raw_usage":{"total_tokens":1889,"prompt_tokens":1109,"completion_tokens":780,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":725,"completion_tokens_details":{"reasoning_tokens":692}},"tokens_in":725,"tokens_out":780,"duration_ms":8990,"temperature":1.0,"reasoning_tokens":692,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:24:28.203419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same models with $\\text{rank\\_score\\_inverse\\_square}$ as the target but with the gap-days and rank-score features removed, leaving only text embeddings and word or sentence counts as inputs: if $R^2$ and Spearman $\\rho$ collapse toward zero, the headline results depended on handing the model the very quantity it was asked to predict. Separately, test the predicted ranks against an urgency signal not derived from gap days — for instance, time from first hearing to final disposal, or priority ratings assigned by judges — and check whether the Spearman correlation survives; if it does not, the ranking does not track real urgency.","supporting_citations":[],"review_version":1}