{"id":"6d0e9d03-5bb8-4ce0-be0e-9d029b3cbed8","arxiv_id":"2511.12290","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A ROUGE+MMR pipeline converts abstractive legal gold summaries into extractive sentence summaries across seven datasets, but the main quality check compares the output to the same summaries used to build it.","lead":"This paper builds a pipeline that turns human-written abstractive summaries of court judgments into extractive summaries made of sentences pulled directly from the judgment. The authors add these to seven legal summarization datasets and argue the resulting summaries are accurate enough to train and test new summarizers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TEG quality is validated only against the OAG summaries used to build it, so the claim that TEG encodes expert salience as extractive gold is untested; human evaluation is tiny and cherry-picked.","rationale":"I read the paper in good faith: the practical resource (TEG summaries for seven legal datasets) would be useful if the central claim held, and the pipeline is transparent. But the claim is that ROUGE-selected sentences inherit the legal expert's notion of salience. That is precisely the assumption the paper leaves untested. The automatic evaluation compares TEG to the same OAG text used for candidate selection, so all high similarity scores (Tables 3, 5, 8) are consistent with a well-tuned string-matching system rather than with expert-aligned extractive gold. The human study is not a corrective: it is small, non-random, restricted to the datasets where TEG already looks better, and frames the question as information overlap with OAG, not as salience annotation of the judgment. The k=2 selection from the same evaluation curves further weakens the evidentiary value of those curves. I also note the abstract promises comparisons to five methods, but only the LSA baseline appears; this does not by itself destroy the dataset-creation contribution, but it undermines the 'outperforms all competing methods' sentence. None of this is an internal inconsistency in the pipeline, and a future version that releases the data and provides an independent human/oracle extractive-gold evaluation could support the claim. As submitted, however, the reader's rejection is warranted; my concern names the same weakest assumption and adds the circular-validation and missing-comparison specifics.","tokens_in":18348,"tokens_out":7285,"duration_ms":70036,"concrete_test":"Sample ~100 judgments stratified across all seven datasets. Have two legal experts, blind to OAG and TEG, independently create extractive gold summaries by marking/ranking the judgment sentences that must appear in a summary to preserve salient legal content. Compute sentence-level F1/ROUGE between each expert's extractive gold and (a) the TEG summary, (b) an LSA baseline, (c) a random-sentence baseline of matched length; also compute expert-expert agreement. If TEG does not significantly outperform LSA/random or achieve acceptable agreement with at least one expert, then ROUGE-based selection has not been shown to track expert salience and the TEG-as-gold claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that TEG summaries are high-quality extractive gold standards that carry over the legal experts' sense of salience from OAG summaries. The pipeline in Section 2 selects candidate judgment sentences by ROUGE-1/2/L overlap with OAG sentences, and Section 5 then evaluates the resulting TEG summaries against those same OAG summaries using ROUGE, JSD, LSA and LegalBERT similarity, and legal-entity/provision recall. High scores on these comparisons are partly by construction: the selected sentences are the ones lexically closest to OAG, so measuring lexical/semantic proximity back to OAG cannot establish that an expert would regard them as the salient sentences of the judgment. The only non-automatic validation (Sec. 5.6) is not an independent gold standard: it covers 16 summaries, deliberately chosen as the top- and least-scoring from only the four datasets where TEG already scores better, and the experts are asked how well TEG carries OAG information, not whether the TEG sentences are the salient sentences an extractive gold annotator would pick. Parameter k=2 is also selected from the same evaluation curves (Sec. 5), adding selection bias. The abstract's claim that AugAbEx is compared with two baselines and three SOTA methods is not supported by the body, which reports only the LSA baseline (Sec. 5.5). Thus the assumption that ROUGE overlap tracks expert salience is load-bearing and never independently tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AugAbEx, a pipeline that transforms human-written abstractive gold (OAG) summaries in seven legal case-summarization datasets into extractive summaries (TEG). The transformation selects candidate judgment sentences by ROUGE-1/2/L overlap with each OAG sentence and then applies MMR with λ=0.5 to produce a summary of length similar to the OAG. The authors evaluate TEG against OAG along structural, lexical, semantic, and domain-specific dimensions, compare TEG with LSA-based extractive summaries, and report a small human evaluation. The central claim is that TEG summaries are of sufficient quality to serve as extractive gold-standard summaries for training and evaluating legal summarizers.","tokens_in":18775,"tokens_out":4790,"duration_ms":47607,"significance":"If the central claim could be established, the contribution would be valuable: a low-cost, transparent way to create extractive gold-standard data from existing abstractive datasets, with code and data committed for public release. The evaluation framework is multidimensional and uses the Bradley-Terry model for paired comparison, which is a sound methodological choice. However, the validation as presented is largely circular: TEG sentences are selected for lexical overlap with OAG, and the main quality evidence measures similarity back to that same OAG. The only external human evaluation is very small and not independent of the construction objective. The paper therefore does not currently provide convincing evidence for the gold-standard claim.","major_comments":[{"comment":"The validation is circular. Section 2 builds TEG by selecting, for each OAG sentence, the top-k judgment sentences with highest ROUGE-1/2/L overlap. Section 5 then reports high ROUGE scores, high LSA/LegalBERT semantic similarity, and low JSD between TEG and the same OAG used to construct TEG. These results are partly by construction: the selected sentences are the ones lexically closest to OAG. High semantic similarity is plausible but does not demonstrate that TEG sentences are the salient extractive sentences of the judgment. The paper needs an independent gold standard—for example, expert annotations of salient sentences in a held-out sample of judgments—against which to measure TEG quality.","section":"Sec. 2 vs. Sec. 5.1–5.4"},{"comment":"The human evaluation is not an independent validation. It covers only 16 summaries, deliberately selected as top- and least-scoring on ROUGE-L from the four datasets where TEG already scores better. The grading scale (Table 9) asks how well TEG carries the information in OAG, not whether TEG sentences are the salient sentences an extractive-gold annotator would choose. The high correlation between expert scores and ROUGE-L/embedding similarity is unsurprising because those are the same signals used to build TEG. A valid study would ask experts to rate the extractive summary against the judgment itself, or to independently identify salient sentences, on a random sample.","section":"Sec. 5.6"},{"comment":"The parameter k=2 is selected post hoc from the provision-recall curves (Fig. 3) computed on the same datasets used in the main evaluation. This introduces selection bias: the reported results are for a parameter chosen on the evaluation metric. The authors should use a held-out validation split for choosing k, or report results across a range of k values to show robustness.","section":"Sec. 5 and Fig. 3"},{"comment":"The abstract states that AugAbEx is compared with 'two baselines and three current state-of-the-art methods' and outperforms all competing methods. The body, however, reports only the LSA baseline (Sec. 5.5). No other baselines or SOTA methods appear anywhere in the full text. This discrepancy must be corrected: either add the missing comparisons or revise the abstract to match the actual scope.","section":"Abstract vs. Sec. 5.5"}],"minor_comments":[{"comment":"The paper acknowledges that LegalNER is trained on Indian judgments and underperforms on UK, US, and Australian data (Table 2). This limitation is appropriate, but the text says 'in-depth analysis of ILC summaries failed to reveal a convincing explanation'—consider reporting the analysis or removing the sentence.","section":"Sec. 5.1"},{"comment":"λ=0.5 is set without any sensitivity analysis. A brief robustness check over λ values would strengthen the claim that the pipeline is not sensitive to this choice.","section":"Sec. 2"},{"comment":"The conclusion states TEG summaries 'lag behind the original summaries in capturing the legal provisions in the judgment.' This is an honest limitation but sits uneasily with the paper's earlier claim that TEG is 'at par' with OAG; the abstract and conclusion should be aligned in tone.","section":"Conclusion"},{"comment":"Minor typographical and capitalization issues: 'macro-Averaged' in Fig. 3 caption should be 'macro-averaged'; 'legal entites' in Sec. 3.1; 'IndianKanoon' in Sec. 5.6 should be 'Indian Kanoon'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core issue is the circular validation; this is the main reason I could not accept in the current form. The paper's resource may be useful as a 'silver standard' if framed as such, but the claim of 'gold standard' quality requires external validation. The abstract/body mismatch regarding comparisons with SOTA methods should be fixed. I would be willing to consider a revised version with an independent human evaluation and corrected claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe paper has a useful idea but doesn't earn its central claim. The pipeline that converts abstractive gold summaries (OAG) into extractive ones (TEG) is simple and transparent, and the resulting resource for seven legal datasets could be handy if properly validated. But the validation is mostly circular: TEG sentences are chosen for high ROUGE overlap with OAG, then measured for ROUGE and semantic similarity back to OAG. High scores are partly baked in. The only external check is a human evaluation of 16 summaries, picked as the top- and least-scoring from the four datasets where TEG already looked better, with experts asked how well TEG carries OAG information — not whether the extracted sentences are the salient ones an extractive gold annotator would choose. That doesn't test the load-bearing assumption that ROUGE overlap tracks expert salience.\n\nWhat the paper does well: the evaluation is broad — structural, lexical, semantic, and domain-specific — and the comparison with an LSA baseline is honest enough to show TEG is closer to OAG than a generic extractive system. The commitment to release the data is a plus.\n\nThe soft spots are real. Besides the circularity, k=2 is picked by looking at provision-recall curves, which is selection on the evaluation metric. And the abstract says the pipeline was compared with two baselines and three state-of-the-art methods; the body only reports LSA. That claim shouldn't be in the abstract as written.\n\nVerdict: don't reject the idea, but don't accept the paper as is. The resource may still be useful, but the gold-standard claim needs independent support. A serious referee can push for that, so I'd send it out rather than desk reject. If they release the data and either soften the claim or add a non-circular evaluation (e.g., human salience annotation or downstream extractive-summarization experiments), it could be a legitimate contribution. For me, I wouldn't cite it as a gold-standard resource yet, but I might bring it to a reading group to talk about why this kind of validation is circular.","headline":"Useful dataset resource whose central claim is undermined by circular validation and an abstract that overstates the comparisons.","tokens_in":19184,"tokens_out":3294,"would_cite":false,"duration_ms":31701,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AugAbEx turns human-written abstractive legal summaries into extractive ones good enough to serve as gold standards for training and evaluating extractive case summarizers.","keywords":["legal summarization","extractive summarization","abstractive summarization","data augmentation","gold standard","legal entities","ROUGE","Maximum Marginal Relevance"],"falsifier":"A concrete falsifier: take a set of judgments and their abstractive gold summaries, run AugAbEx, and have law experts independently pick extractive sentences for the same summaries. If the experts' choices agree with the TEG sentences at no more than chance level, the claim that TEG preserves expert salience is false. Alternatively, if replacing ROUGE scores with random sentence selection produces TEG summaries that score equally well under the paper's own evaluation framework, then the selection step is not doing the work.","tokens_in":18284,"feed_emoji":"⚖️","tokens_out":4735,"duration_ms":36355,"temperature":0.7,"pith_summary":"Legal summarization is moving toward extractive methods because abstractive summaries from language models can misrepresent legal jargon. Hand-labeling extractive summaries is costly, so the paper repurposes existing datasets that contain expert-written abstractive summaries. It introduces a pipeline that selects judgment sentences with high lexical overlap to each abstractive sentence and then applies Maximum Marginal Relevance to craft a diverse extractive summary. The resulting 'transformed extractive gold' (TEG) summaries are shown to be structurally, lexically, semantically, and legally similar to the original abstractive ones across seven datasets, and to outperform baselines and state-of-the-art extractive methods. If correct, this gives the community a low-cost way to create extractive training and evaluation data at scale.","feed_headline":"Abstractive legal summaries become extractive gold standards","feed_subtitle":"A two-stage pipeline turns human-written summaries into extractive references for training and evaluating legal summarizers.","key_machinery":"The central mechanism is a two-stage augmentation pipeline. Stage one selects, for each sentence in the original abstractive gold (OAG) summary, the top-k judgment sentences whose average ROUGE-1, ROUGE-2, and ROUGE-L overlap with that OAG sentence is highest, forming a candidate set. Stage two applies Maximum Marginal Relevance (MMR), which balances relevance to the candidate pool against diversity from already-selected sentences, to greedily build an extractive summary of roughly the same length as the OAG. The load-bearing assumption is that ROUGE overlap between an OAG sentence and a judgment sentence is a valid proxy for what a legal expert would call salient, so the selected sentences","core_discovery":"On its own terms, the paper's central claim is that the quality of the transformed extractive gold (TEG) summaries is high enough to treat them as extractive gold-standard references. The evidence is a multi-dimensional comparison against the original abstractive gold (OAG) summaries — legal-entity counts, provision recall, semantic similarity in latent and legal-embedding spaces, lexical overlap, term-distribution distance, divergence from the case document, length, sentence length, and readability — together with a Bradley-Terry instance-level statistical analysis and human grading by law experts. The TEG summaries score comparably or better than OAG on most attributes for five datasets, w","pith_inferences":["A risk the paper does not fully address: because the TEG summaries are selected by lexical overlap, they may systematically favor sentences that mirror the abstractive wording rather than sentences that would be chosen by an expert reading the judgment fresh. A direct test would compare TEG against expert-created extractive summaries on the same cases.","The CivilSum and Australian results suggest that the pipeline's quality hinges on the original summaries being sentential; for phrasal summaries, a sentence-level transform may be fundamentally misaligned, and a clause-level or entity-level extractive target might be more appropriate.","The low legal-entity counts for non-Indian datasets (Australia, BillSum) expose a domain-transfer weakness: the LegalNER model used to evaluate is trained on Indian judgments, so the 'legal richness' comparison likely understates the actual legal information in those TEG summaries. A cross-jurisdiction entity recognizer would give a fairer test.","The paper's approach could be extended to other schemas: instead of ROUGE, one could use semantic similarity (LegalBERT) in the candidate-selection stage, which might improve alignment for phrasal summaries."],"forward_implications":["The seven augmented datasets can be used to train supervised extractive case summarizers without additional human annotation.","TEG summaries enable fair extractive evaluation: system-generated extractive summaries can now be compared against an extractive gold standard rather than only against abstractive references via ROUGE.","The pipeline generalizes to any domain where human-written abstractive summaries exist and judgment documents are available, so new extractive datasets can be created cheaply.","Extractive case summarizers should be designed to prioritize sentences rich in legal entities, since the comparison with LSA shows that domain-blind extractive methods under-deliver on legal information.","The evaluation framework (structural, lexical, semantic, domain) can serve as a template for quality assurance of future gold-standard summaries."],"fun_headline_variants":["AugAbEx turns abstractive gold into extractive gold","Legal summarization: hybrid pipeline yields extractive gold-like references","TEG summaries rival human-written gold in legal domain","From abstractive to extractive: a new legal summarization bridge"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The pipeline's reliability rests on the assumption that a high ROUGE score between an abstractive summary sentence and a judgment sentence means the judgment sentence carries the same legal salience that the expert intended.","fun_headline_variants_meta":{"raw":{"variants":["AugAbEx turns abstractive gold into extractive gold","Legal summarization: hybrid pipeline yields extractive gold-like references","TEG summaries rival human-written gold in legal domain","From abstractive to extractive: a new legal summarization bridge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000743,"raw_usage":{"total_tokens":3152,"prompt_tokens":745,"completion_tokens":2407,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":2338}},"tokens_in":489,"tokens_out":2407,"duration_ms":18849,"temperature":1.0,"reasoning_tokens":2338,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T22:02:30.012685+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier: take a set of judgments and their abstractive gold summaries, run AugAbEx, and have law experts independently pick extractive sentences for the same summaries. If the experts' choices agree with the TEG sentences at no more than chance level, the claim that TEG preserves expert salience is false. Alternatively, if replacing ROUGE scores with random sentence selection produces TEG summaries that score equally well under the paper's own evaluation framework, then the selection step is not doing the work.","supporting_citations":[],"review_version":1}