REVIEW 4 major objections 4 minor 2 references
AugAbEx turns human-written abstractive legal summaries into extractive ones good enough to serve as gold standards for training and evaluating extractive case summarizers.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A ROUGE+MMR pipeline converts abstractive legal gold summaries into extractive sentence summaries across seven datasets, but the main quality check compares the output to the same summaries used to build it.
T0 review reviewed 2026-08-03 challenge →
load-bearing objection Useful dataset resource whose central claim is undermined by circular validation and an abstract that overstates the comparisons. the 4 major comments →
AugAbEx: Bridging Abstractive and Extractive Legal Summarization
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
On its own terms, the paper's central claim is that the quality of the transformed extractive gold (TEG) summaries is high enough to treat them as extractive gold-standard references. The evidence is a multi-dimensional comparison against the original abstractive gold (OAG) summaries — legal-entity counts, provision recall, semantic similarity in latent and legal-embedding spaces, lexical overlap, term-distribution distance, divergence from the case document, length, sentence length, and readability — together with a Bradley-Terry instance-level statistical analysis and human grading by law experts. The TEG summaries score comparably or better than OAG on most attributes for five datasets, w
What carries the argument
The central mechanism is a two-stage augmentation pipeline. Stage one selects, for each sentence in the original abstractive gold (OAG) summary, the top-k judgment sentences whose average ROUGE-1, ROUGE-2, and ROUGE-L overlap with that OAG sentence is highest, forming a candidate set. Stage two applies Maximum Marginal Relevance (MMR), which balances relevance to the candidate pool against diversity from already-selected sentences, to greedily build an extractive summary of roughly the same length as the OAG. The load-bearing assumption is that ROUGE overlap between an OAG sentence and a judgment sentence is a valid proxy for what a legal expert would call salient, so the selected sentences
Load-bearing premise
The pipeline's reliability rests on the assumption that a high ROUGE score between an abstractive summary sentence and a judgment sentence means the judgment sentence carries the same legal salience that the expert intended.
What would settle it
A concrete falsifier: take a set of judgments and their abstractive gold summaries, run AugAbEx, and have law experts independently pick extractive sentences for the same summaries. If the experts' choices agree with the TEG sentences at no more than chance level, the claim that TEG preserves expert salience is false. Alternatively, if replacing ROUGE scores with random sentence selection produces TEG summaries that score equally well under the paper's own evaluation framework, then the selection step is not doing the work.
If this is right
- The seven augmented datasets can be used to train supervised extractive case summarizers without additional human annotation.
- TEG summaries enable fair extractive evaluation: system-generated extractive summaries can now be compared against an extractive gold standard rather than only against abstractive references via ROUGE.
- The pipeline generalizes to any domain where human-written abstractive summaries exist and judgment documents are available, so new extractive datasets can be created cheaply.
- Extractive case summarizers should be designed to prioritize sentences rich in legal entities, since the comparison with LSA shows that domain-blind extractive methods under-deliver on legal information.
- The evaluation framework (structural, lexical, semantic, domain) can serve as a template for quality assurance of future gold-standard summaries.
Where Pith is reading between the lines
- A risk the paper does not fully address: because the TEG summaries are selected by lexical overlap, they may systematically favor sentences that mirror the abstractive wording rather than sentences that would be chosen by an expert reading the judgment fresh. A direct test would compare TEG against expert-created extractive summaries on the same cases.
- The CivilSum and Australian results suggest that the pipeline's quality hinges on the original summaries being sentential; for phrasal summaries, a sentence-level transform may be fundamentally misaligned, and a clause-level or entity-level extractive target might be more appropriate.
- The low legal-entity counts for non-Indian datasets (Australia, BillSum) expose a domain-transfer weakness: the LegalNER model used to evaluate is trained on Indian judgments, so the 'legal richness' comparison likely understates the actual legal information in those TEG summaries. A cross-jurisdiction entity recognizer would give a fairer test.
- The paper's approach could be extended to other schemas: instead of ROUGE, one could use semantic similarity (LegalBERT) in the candidate-selection stage, which might improve alignment for phrasal summaries.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AugAbEx, a pipeline that transforms human-written abstractive gold (OAG) summaries in seven legal case-summarization datasets into extractive summaries (TEG). The transformation selects candidate judgment sentences by ROUGE-1/2/L overlap with each OAG sentence and then applies MMR with λ=0.5 to produce a summary of length similar to the OAG. The authors evaluate TEG against OAG along structural, lexical, semantic, and domain-specific dimensions, compare TEG with LSA-based extractive summaries, and report a small human evaluation. The central claim is that TEG summaries are of sufficient quality to serve as extractive gold-standard summaries for training and evaluating legal summarizers.
Significance. If the central claim could be established, the contribution would be valuable: a low-cost, transparent way to create extractive gold-standard data from existing abstractive datasets, with code and data committed for public release. The evaluation framework is multidimensional and uses the Bradley-Terry model for paired comparison, which is a sound methodological choice. However, the validation as presented is largely circular: TEG sentences are selected for lexical overlap with OAG, and the main quality evidence measures similarity back to that same OAG. The only external human evaluation is very small and not independent of the construction objective. The paper therefore does not currently provide convincing evidence for the gold-standard claim.
major comments (4)
- [Sec. 2 vs. Sec. 5.1–5.4] The validation is circular. Section 2 builds TEG by selecting, for each OAG sentence, the top-k judgment sentences with highest ROUGE-1/2/L overlap. Section 5 then reports high ROUGE scores, high LSA/LegalBERT semantic similarity, and low JSD between TEG and the same OAG used to construct TEG. These results are partly by construction: the selected sentences are the ones lexically closest to OAG. High semantic similarity is plausible but does not demonstrate that TEG sentences are the salient extractive sentences of the judgment. The paper needs an independent gold standard—for example, expert annotations of salient sentences in a held-out sample of judgments—against which to measure TEG quality.
- [Sec. 5.6] The human evaluation is not an independent validation. It covers only 16 summaries, deliberately selected as top- and least-scoring on ROUGE-L from the four datasets where TEG already scores better. The grading scale (Table 9) asks how well TEG carries the information in OAG, not whether TEG sentences are the salient sentences an extractive-gold annotator would choose. The high correlation between expert scores and ROUGE-L/embedding similarity is unsurprising because those are the same signals used to build TEG. A valid study would ask experts to rate the extractive summary against the judgment itself, or to independently identify salient sentences, on a random sample.
- [Sec. 5 and Fig. 3] The parameter k=2 is selected post hoc from the provision-recall curves (Fig. 3) computed on the same datasets used in the main evaluation. This introduces selection bias: the reported results are for a parameter chosen on the evaluation metric. The authors should use a held-out validation split for choosing k, or report results across a range of k values to show robustness.
- [Abstract vs. Sec. 5.5] The abstract states that AugAbEx is compared with 'two baselines and three current state-of-the-art methods' and outperforms all competing methods. The body, however, reports only the LSA baseline (Sec. 5.5). No other baselines or SOTA methods appear anywhere in the full text. This discrepancy must be corrected: either add the missing comparisons or revise the abstract to match the actual scope.
minor comments (4)
- [Sec. 5.1] The paper acknowledges that LegalNER is trained on Indian judgments and underperforms on UK, US, and Australian data (Table 2). This limitation is appropriate, but the text says 'in-depth analysis of ILC summaries failed to reveal a convincing explanation'—consider reporting the analysis or removing the sentence.
- [Sec. 2] λ=0.5 is set without any sensitivity analysis. A brief robustness check over λ values would strengthen the claim that the pipeline is not sensitive to this choice.
- [Conclusion] The conclusion states TEG summaries 'lag behind the original summaries in capturing the legal provisions in the judgment.' This is an honest limitation but sits uneasily with the paper's earlier claim that TEG is 'at par' with OAG; the abstract and conclusion should be aligned in tone.
- [Throughout] Minor typographical and capitalization issues: 'macro-Averaged' in Fig. 3 caption should be 'macro-averaged'; 'legal entites' in Sec. 3.1; 'IndianKanoon' in Sec. 5.6 should be 'Indian Kanoon'.
Circularity Check
TEG quality is validated only against the OAG summaries used to build it; the main metrics use the same ROUGE objective as the sentence-selection step, and the human eval is a small, selected self-consistency check.
specific steps
-
self definitional
[Section 2 (Candidate Sentence Selection) and Section 5.3 (Lexical Level Analysis), Table 5]
"The method finds lexical overlap of each sentence in the original abstractive gold standard summary (OAG) with each sentence in the case document by computing the ROUGE-1, ROUGE-2, and ROUGE-L scores. Next, the average of three scores is calculated for each sentence, based on which the top-k highest scoring sentences from the case document are selected corresponding to each OAG summary sentence. ... First three rows of Table 5 show the lexical overlap between the original abstractive gold standard and transformed extractive summaries in terms of unigrams (Rouge-1), bigrams (Rouge-2) and longes"
TEG sentences are selected to maximize ROUGE-1/2/L overlap with OAG, so reporting high ROUGE-1/2/L between TEG and OAG as evidence of TEG quality is a self-consistency check: the evaluation metric is the same objective used to select the candidate sentences. The scores are therefore partly forced by construction and cannot independently establish that a legal expert would select these same sentences as extractive gold. Since this ROUGE comparison is the paper's main lexical evidence for claiming TEG is 'at par' with OAG, the central validation reduces to checking the pipeline against its own input.
-
self definitional
[Section 5.6 (Human Evaluation), Table 9]
"From each of the four datasets (IN-Jud-Cit, ILC, IN-Abs and UK-Abs) for which the TEG summaries consistently score better than OAG summaries, we select two top-scoring and two least-scoring summaries (ROUGE-L) for human evaluation. We ask the law experts to grade the transformed extractive summary on a Likert scale of 1 - 5 based on how well it carries the information in the original abstractive summary."
The human evaluation does not ask experts whether TEG sentences are the salient sentences an extractive-gold annotator would pick; it asks only how well TEG carries OAG information, i.e., fidelity to the same summaries used to construct TEG. Moreover, the 16 summaries are deliberately chosen as the top- and least-scoring ones from the four datasets where TEG already outperforms OAG on ROUGE-L. Thus the evaluation is a filtered self-consistency check, not an independent test of TEG's status as extractive gold.
-
fitted input called prediction
[Section 5 opening, Fig. 3 and Table 8 (ProvRecall)]
"Our empirical analysis reveals that selecting two candidate sentences enhances recall score of provisions, leading to improved summary quality. This effectiveness is evident in the trends shown in Fig. 3. ... Based on these findings, we set k=2 for all subsequent experiments."
The number of candidate sentences k is tuned on the provision-recall curves (Fig. 3), and the same provision recall is then reported as a domain-level quality metric for TEG (Section 5.1 and Table 8). Choosing k to maximize the evaluation metric and then presenting the resulting high ProvRecall values as evidence of quality is selection-on-the-evaluation-set; the metric is no longer an unbiased characterization of the pipeline.
full rationale
The circularity here is not a self-citation chain. The paper cites Shukla et al. (2022) for the candidate-selection method, but that is an external method. The problem is that construction and evaluation share the same reference object: TEG is built by maximizing ROUGE overlap with OAG, and then TEG is judged by ROUGE, JSD, LSA/LegalBERT similarity, and even a human 'carries OAG information' criterion, all with respect to OAG. These are self-consistency checks. High similarity to OAG is partly by construction because the selected sentences are the ones lexically closest to OAG. The only non-circular comparison is against the LSA baseline, which shows TEG is closer to OAG than LSA, but that again uses OAG as the criterion and does not test extractive salience against an independent gold standard. Additionally, k=2 is tuned on the Fig. 3 provision-recall curves that are later used as evidence of TEG quality. The human evaluation is very small (16 summaries), selected from the four datasets where TEG already scores better, and asks about OAG fidelity rather than independent extractive salience. Therefore the central claim that TEG can serve as extractive gold standard is not independently established; however, the pipeline is transparent and the augmented data may still be useful for training with appropriate caveats. The abstract's statement that AugAbEx is compared with two baselines and three state-of-the-art methods is unsupported by the body, which reports only the LSA baseline; this is a reporting inconsistency rather than circularity. Overall score 8: the main validation reduces to a self-consistency check against the input summaries.
Axiom & Free-Parameter Ledger
free parameters (3)
- k (candidate sentences per OAG sentence) =
2
- lambda (MMR relevance-diversity trade-off) =
0.5
- TEG summary length target =
OAG word count
axioms (5)
- domain assumption ROUGE overlap between OAG summary sentences and judgment sentences identifies sentences carrying the expert's salient content.
- domain assumption Higher legal-entity count and provision recall are valid proxies for summary quality for legal experts.
- domain assumption Semantic similarity scores (LSA, LegalBert) between TEG and OAG are meaningful measures of summary fidelity.
- domain assumption LegalNER trained on Indian judgments generalizes to UK/US/Australian legal text.
- domain assumption The original human-written OAG summaries are correct and trustworthy gold standards.
Cite this review
Pith. "Pith review of AugAbEx: Bridging Abstractive and Extractive Legal Summarization." pith.science (2026). https://pith.science/paper/N5IKCJ5V
@misc{pith2026251112290,
author = {Pith},
title = {Pith review of: AugAbEx: Bridging Abstractive and Extractive Legal Summarization},
year = {2026},
howpublished = {\url{https://pith.science/paper/N5IKCJ5V}},
note = {Machine review of arXiv:2511.12290}
}
read the original abstract
Automatic summarization of legal judgments liberates law professionals from heavy cognitive burden due to the complexity of the language, context-sensitive legal jargon, and the length of the document. Caveats of abstractive summarization for legal documents, revealed in recent studies, have impelled the development of newhybrid and extractive summarization methods, along with datasets in the legal domain. We propose an efficient and elegant pipeline (AugAbEx) to repurpose an existing large case summarization dataset with human-written gold-standard summaries by augmenting it with silver standard extractive summaries ensconcing experts' opinion. Availability of silver summaries along with gold standard human-written summaries will bolster development and evaluation of novel hybrid and extractive case-summarization algorithms in the legal domain. We thoroughly scrutinize the augmented extractive summaries in structural, lexical, and semantic dimensions, within a domain specific framework, to ensure quality. Extensive experiments and statistical test on seven English legal case-summarization datasets demonstrate that the silver standard extractive summaries produced by the proposed pipeline score well across all evaluation dimensions. Comparison of AugAbEx with two baselines and three current state-of-the-art methods reveals that it outperforms all competing methods, and the quality of the summaries produced by the proposed pipeline is superior
Figures
Reference graph
Works this paper leans on
-
[1]
Abdullah, D., Nayak, S., Suri, G., Chali, Y.: Generating query focused summaries without fine-tuning the transformer-based pre-trained models. arXiv preprint arXiv:2303.06230 (2023) Bhattacharya, P., Hiware, K., Rajgaria, S., Pochhi, N., Ghosh, K., Ghosh, S.: A com- parative study of summarization algorithms applied to legal case judgments. In: Azzopardi,...
Pith/arXiv arXiv 2023
-
[2012]
Proceedings 12, pp. 40–52 (2012). Springer Harasta, J., Novotn´ a, T., Savelka, J.: It cannot be right if it was written by ai: On lawyers’ preferences of documents perceived as authored by an llm vs a human. arXiv preprint arXiv:2407.06798 (2024) Kalamkar, P., Agarwal, A., Tiwari, A., Gupta, S., Karn, S., Raghavan, V.: Named entity recognition in indian ...
Pith/arXiv arXiv 2012
This paper was first reviewed by deepseek-v4-flash on August 3, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.