Automatic Text Summarization of Legal Cases: A Hybrid Approach

Varun Pandya

REVIEW 3 major objections 5 minor 26 references

Automatic Text Summarization of Legal Cases: A Hybrid Approach

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An unsupervised k-means and tf-idf hybrid, tested on one 200-page Australian case, reports higher ROUGE F-measures than three existing legal summarizers.

desk verdict Honest but thin student project; the k-means+tf-idf recipe is unremarkable and the single-case, length-matched evaluation doesn't support the 'favourably well' claim. read the letter →

arxiv 1908.09119 v1 pith:TBR4B4TL submitted 2019-08-24 cs.CL cs.IR

Varun Pandya This is my paper · ORCID

classification cs.CLcs.IR

keywords automatictextsummarizationlegaldomaink-meansclusteringtf-idfextractiveROUGEevaluationunsupervised

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that a legal case file can be summarized automatically by an unsupervised, extractive pipeline: preprocess the text, represent sentences as tf-idf vectors, cluster them with k-means, rank sentences inside each cluster by tf-idf weight plus similarity to the case title, and take the top sentences from every cluster. Tested on a single 200-page Australian legal case, the method generated a roughly 150-sentence summary in under three minutes. The reported ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-W F-measures place the hybrid above TextSummarizer, SplitBrain, and ESummarizer on all four metrics. The paper concludes that this cluster-based, coverage-oriented extraction yields a brief close to the attorney's own summary and offers a practical starting point for legal summarization without requiring training data.

What carries the argument

The load-bearing machinery is the cluster-then-rank selection process. Sentences are converted to tf-idf (term frequency-inverse document frequency) vectors, grouped by the k-means clustering algorithm into K similarity-based clusters, and then each sentence is scored as the sum of its normalized tf-idf weight and a title-similarity score that compares only noun forms against the case title. Equal numbers of top-ranked sentences are taken from each cluster, so the summary length is controlled and the final output is intended to represent the whole document with minimal redundancy.

What would settle it

Take a set of, say, twenty legal cases, obtain two or more independent human reference summaries for each, run the hybrid and the three baseline tools with all outputs fixed to a common length, and compare average ROUGE scores; if the hybrid no longer beats the baselines, or if random sentence selection from clusters matches its scores, the reported ranking would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that ranking sentences by normalized tf-idf score plus similarity to the case title, and doing that within k-means clusters rather than over the whole document, makes an extractive summarizer competitive for long legal texts. ROUGE is a set of standard n-gram overlap measures for comparing machine summaries with human reference summaries; on the one case reported, the hybrid achieved ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-W F-measures of 27.88, 5.83, 33.50, and 11.38, each above TextSummarizer, SplitBrain, and ESummarizer. The paper attributes the result to low redundancy: because sentences are drawn from every cluster, the summary covers different parts of the case file instead of concentrating on the most frequent words. It also claims the method is computationally favorable and unsupervised, unlike approaches that need a training corpus.

Load-bearing premise

The load-bearing premise is that ROUGE scores computed against a single lawyer-written summary, with the proposed system's output length manually matched to that summary, provide a fair basis for ranking the methods.

Editorial extensions

If this is right

According to the paper, legal case files can be summarized without labeled training data, which matters because annotated legal corpora are scarce.
If the reported speed and stability hold, several-hundred-page cases can be reduced to usable briefs in minutes, while a graph-based method the paper tried on the same input failed with a memory error.
Because the summary draws equally from every cluster, the method is designed to cover the full case rather than repeat dominant vocabulary, which is the paper's stated reason for its stronger ROUGE results.
The paper's own next steps are automatic cluster-count selection, additional sentence-ranking features, and rhetorical-role structures for more formal legal briefs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implied but untested consequence is that the reported ranking may depend on output length: the proposed system's summary was manually set to the reference length of about 150 sentences, whereas the online baselines likely produced shorter outputs, so a length-matched comparison would be the stronger test.
Because the evaluation uses one case and one human reference summary, the margins are evidence of feasibility rather than general superiority; running the same pipeline across many cases with multiple independent reference summaries would show whether the advantage persists.
A random-control experiment would sharpen the claim: if sentences selected at random from the clusters achieved similar ROUGE scores, then the tf-idf and title ranking, not the clustering, would be doing the work.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take on 1908.09119: it's an honest student project, not a research result. The method is k-means over tf-idf sentence vectors plus a title-similarity rerank, applied to one 200-page Australian case. Nothing in the pipeline is new, but the paper does what it says: it produces a length-controlled extractive summary in under three minutes without training data.

What it does well: the writing is clear, the approach is straightforwardly reproducible from the description, and it cites the relevant legal-NLP background (CaseSummarizer, FLEXICON, etc.). It also includes an explicit limitation: real-world attorney testing hasn't been done. That's good hygiene.

The soft spots are the ones the stress test flags. Section 5 says the system generated about 150 sentences 'to match its length with the provided summary for better comparison.' That's a mild form of peeking at the reference; ROUGE F-measures are length-sensitive. The paper doesn't report the output lengths of TextSummarizer, SplitBrain, or ESummarizer, so if those tools returned shorter summaries (typical for online tools), their recall and F-measures are depressed by length mismatch, not by content selection. Add the single case, single reference, and no error bars, and the table in Section 6 cannot support the conclusion that the hybrid 'performs favourably well.' A ROUGE-2 F of 5.83 also undercuts the claim that the output 'closely resembles' the attorney's summary.

The central problem is not the method - it's the comparison. I don't see this as a fatal flaw in the idea; the idea is fine as a baseline. But the paper overstates what the data show, and the evaluation protocol would need to be rebuilt before the comparative claim means anything.

Who's it for? A workshop in legal NLP or a course project repository. A serious referee? I wouldn't send this to a main-track conference; the contribution is too incremental and the evaluation too thin. If the authors added a few cases, fixed the length matching, and reported confidence intervals, it could be a decent short paper. As is, it's a reasonable technical note.

Recommendation: desk-reject for a research venue, but don't ignore it entirely if you work on legal summarization - it's a clean description of a useable baseline.

Referee Report

3 major / 5 minor

Summary. The paper proposes an unsupervised extractive text summarization method for legal case documents. The approach consists of pre-processing the input case file, embedding sentences with tf-idf vectors, clustering the sentences using k-means, and then selecting the top-ranked sentences from each cluster using a combination of tf-idf score and similarity to the case title. The authors evaluate the method on a single Australian legal case (Rush v Nationwide News Pty Ltd (No 7)) by comparing the automatically generated summary with the attorney-prepared summary using ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-W scores against three online summarization tools (TextSummarizer, SplitBrain, ESummarizer). The reported table shows higher ROUGE values for the proposed method, and the authors conclude that it performs favorably and that the generated summary closely resembles the attorney's summary.

Significance. If the reported comparative advantage were valid, the proposed hybrid would be a lightweight, unsupervised baseline for legal-case summarization that avoids the need for training data. The pipeline is clearly described and simple enough to reproduce, and the paper explicitly addresses the large size of legal documents as a motivating challenge. However, the empirical evidence presented is not sufficient to establish the central claim. The comparison is uncontrolled with respect to output length, is based on a single case and a single reference summary, and lacks any significance testing. The low absolute ROUGE scores further undermine the qualitative claim of close resemblance. The idea is reasonable, but the evaluation must be substantially strengthened before the comparative conclusion can be accepted.

major comments (3)

[Section 5 and Table 1] The proposed system's output length is deliberately matched to the gold summary: Section 5 states that 'approximately 150 sentence summary was generated ... to match its length with the provided summary for better comparison.' The paper reports no output lengths for TextSummarizer, SplitBrain, or ESummarizer. Since ROUGE recall and F-measure are sensitive to candidate length, comparing a length-tuned candidate with length-uncontrolled baselines is not a fair test. The reported advantages in Table 1 could therefore be an artifact of length matching rather than better content selection.
[Section 6 and Table 1] The evaluation uses exactly one legal case (Rush v Nationwide News Pty Ltd (No 7)) and exactly one human reference summary. No error bars, confidence intervals, or significance tests are provided. The differences among the systems in Table 1 are therefore indistinguishable from random variation. The assertion in Section 6 that the proposed method 'performs favourably well' is not supported by a single-case comparison, and the paper's own statement in Section 8 that real-world tests with attorneys have not been conducted does not mitigate this gap.
[Section 8 and Table 1] The conclusion that the generated summary 'closely resembles' the attorney's summary is contradicted by the low absolute ROUGE scores in Table 1 (ROUGE-1 F=27.88, ROUGE-2 F=5.83, ROUGE-L F=33.5). A ROUGE-2 F-measure of 5.83 indicates very little bigram overlap, which does not support a qualitative claim of close resemblance. If this claim is intended, it needs to be backed by human evaluation or a more appropriate similarity metric; otherwise the wording should be moderated.

minor comments (5)

[Throughout] The metric name is ROUGE, not ROGUE; the misspelling appears in the abstract, Sections 5 and 6, and in Table 1.
[Section 4.3] The formula for title similarity has unbalanced parentheses: 'Title_score =( (length of similar words) * 0.1 ) / length of words in title)' has an extra closing parenthesis.
[Section 5] The paper mentions algorithms such as the Elbow method for choosing the optimum number of clusters, but the actual value of K used in the experiments is never reported, which hinders reproducibility.
[Table 1] The table header formatting makes it hard to read, and the name 'ESUMMERIZER' in the table is inconsistent with 'ESummarizer' used in Section 6.
[References] References [11] through [14] are listed but do not appear to be cited in the body of the paper, and reference [3] appears to combine multiple distinct sources.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; the hybrid method's content derivation is self-contained, with only a minor length-matching evaluation concern.

full rationale

I traced the claimed derivation chain: Section 4 defines an unsupervised extractive pipeline (pre-processing, k-means clustering of tf-idf vectors, rank by tf-idf score and title similarity, and extraction of top sentences per cluster); Section 5 applies it to Rush v Nationwide and generates an approximately 150-sentence summary; Section 6 compares ROUGE scores against three online tools. No parameter is fitted to the reference summary's content, no equation defines the predicted ROUGE score in terms of the gold summary, and no load-bearing self-citation or imported uniqueness theorem appears. The author cites only external prior work, and the central scoring is independent of the training data because no training data are used. The only target-dependent element is the explicit length matching in Section 5 ('approximately 150 sentence summary was generated... to match its length with the provided summary for better comparison'), which is an evaluation-fairness risk rather than a circular derivation: it does not force the content selection or the ROUGE advantage by construction. The paper also candidly notes in Section 8 that real-world tests with attorneys have not been conducted, further supporting that the strong comparative claim is under-supported, but under-support is a correctness/evidence concern, not circularity. Therefore the derivation is self-contained and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method rests on hand-chosen parameters and unverified domain assumptions. The 0.1 multiplier in the title score is arbitrary, K is unspecified, and the output length is determined by the gold summary. The claim that clusters represent distinct topics and that equal extraction per cluster ensures coverage is asserted without evidence.

free parameters (3)

Title similarity weight multiplier = 0.1
The formula Title_score = ((length of similar words) * 0.1) / length of title in Section 4.3 has an arbitrary multiplier 0.1, chosen by hand with no justification.
Number of clusters K for k-means
Section 4.2 says the optimum number of clusters can be calculated, but the actual K used for the reported results is not reported, so the central result depends on an undisclosed setting.
Summary length = 150 sentences (approximately)
Section 5 states the generated summary was set to match the reference summary length for better comparison, a data-dependent configuration.

assumptions (4)

domain assumption Euclidean distances between tf-idf vectors group sentences into meaningful topical clusters.
Section 4.2 assumes that k-means on tf-idf vectors clusters similar sentences, which is a standard but unverified assumption for legal text.
ad hoc to paper Sentences similar to the title are more informative for legal briefs.
Section 4.3 introduces title similarity as a ranking feature without evidence from legal corpora.
ad hoc to paper Extracting equal numbers of sentences from each cluster ensures coverage with least redundancy.
Section 4.3 asserts this design choice without testing alternatives or validating the cluster quality.
domain assumption ROUGE metrics validly measure summary quality against a single human-prepared summary.
Section 6 uses ROUGE, a word-overlap metric, without accounting for its known limitations for legal text, where synonyms and paraphrases are common.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automatic Text Summarization of Legal Cases: A Hybrid Approach." pith.science (2026). https://pith.science/paper/TBR4B4TL

@misc{pith2026190809119,
  author       = {Pith},
  title        = {Pith review of: Automatic Text Summarization of Legal Cases: A Hybrid Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TBR4B4TL}},
  note         = {Machine review of arXiv:1908.09119}
}

read the original abstract

Manual Summarization of large bodies of text involves a lot of human effort and time, especially in the legal domain. Lawyers spend a lot of time preparing legal briefs of their clients' case files. Automatic Text summarization is a constantly evolving field of Natural Language Processing(NLP), which is a subdiscipline of the Artificial Intelligence Field. In this paper a hybrid method for automatic text summarization of legal cases using k-means clustering technique and tf-idf(term frequency-inverse document frequency) word vectorizer is proposed. The summary generated by the proposed method is compared using ROGUE evaluation parameters with the case summary as prepared by the lawyer for appeal in court. Further, suggestions for improving the proposed method are also presented.

Figures

Figures reproduced from arXiv: 1908.09119 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages

[1]

With the limited human capacity of consuming information, it becomes paramount to extract only useful information out of the vast amount of unstructured data available

INTRODUCTION The amount of information and data has increased tremendously in every field over the years. With the limited human capacity of consuming information, it becomes paramount to extract only useful information out of the vast amount of unstructured data available. Thus, text summarization plays an important role in extracting useful pieces of in...

work page
[2]

Automatic text summarization is broadly of two types:

BACKGROUND Summarization is a challenging sub-task of the broader text-to-text generation field of natural language processing (NLP)[3]. Automatic text summarization is broadly of two types:

work page
[3]

Here the sentences originally present in the document are used to form the summary, with no new sentences being formed

Extractive Summarization: This method involves selecting the most useful and relevant sentences from the document to provide a summary. Here the sentences originally present in the document are used to form the summary, with no new sentences being formed. Extractive summaries produce a set of the most significant sentences from a document, exactly as they...

work page
[4]

Here the gist of the document is represented using computer generated sentences, having close resemblance to the summary as produced by humans

Abstractive Summarization: In this type of summarization, new sentences are formed which are relevant to the document. Here the gist of the document is represented using computer generated sentences, having close resemblance to the summary as produced by humans. Abstractive text summarization captures the salient features of the text corpus and summary is...

work page
[5]

Fast Legal EXpert CONsultant

EXISTING APPROACHES Different kinds of methods have been proposed for text summarization in the legal domain. The earliest of works in this area include the “Fast Legal EXpert CONsultant” (FLEXICON) system developed by Gelbart and Smith (Gelbartand Smith, 1991a). FLEXICON is keyword-based, referencing against a large database of terms to find important re...

work page 1999
[6]

Pre-processing Pre-processing of the document is a vital part of any text summarization approach

THE PROPOSED SYSTEM The implementation is carried out in three different steps: pre-processing, clustering of similar sentences, extracting top ranked sentences from each cluster 4.1. Pre-processing Pre-processing of the document is a vital part of any text summarization approach. It is the foremost step of the proposed approach which includes removing of...

work page
[7]

sent_values = [sum(tf-idf of all words of sentence) / sum(tf-idf of all words of document)]

Tf-idf score of each sentence: The sum of tf-idf value of all words of the sentence is divided by the sum of tf-idf value for all the words present in the document. sent_values = [sum(tf-idf of all words of sentence) / sum(tf-idf of all words of document)]

work page
[8]

Sentence similarity to title: The sentences similar to the title of the case generally present the important facts related to the case, thus aiding in generating a useful and relevant summary. The similarity score to the title is calculated as: Title_score =( (length of similar words) * 0.1 ) / length of words in title) Here only the noun form words ['NN'...

work page

Show all 26 references

[9]

The legal cases consist of the headnotes and summary within the case file

IMPLEMENTATION For implementation, Auslii (http://austlii.edu.au) was used which consists of a database of Australian legal cases. The legal cases consist of the headnotes and summary within the case file. The legal cases downloaded in pdf format are given as input and the leg...

2019
[10]

The relevance and usefulness of each sentence in the summary changes with the person evaluating them

EVALUATION Summaries are difficult to evaluate because of their subjective nature. The relevance and usefulness of each sentence in the summary changes with the person evaluating them. Hence Lin et al. introduced a set of metrics called the ROUGE package in (Lin, 2004) that pr...

2004
[11]

FUTURE WORK The preliminary results of the proposed system serve as a promising means of providing legal summaries of case files. However, the proposed approach can be further improved by making the following addition:  For determining the optimal number of clusters for k-mea...
[12]

The summary generated closely resembles to the original summary as generated by the attorney of the case and can possibly be used in the court of law after further improvements

CONCLUSIONS The proposed method works favorably well against existing approaches. The summary generated closely resembles to the original summary as generated by the attorney of the case and can possibly be used in the court of law after further improvements. The summary gener...
[13]

Text Summarization Extraction System (TSES) Using Extracted Keywords

Al-Hashemi, R.: “Text Summarization Extraction System (TSES) Using Extracted Keywords.” International Arab Journal of e-Technology 1(4) (June 2010)

2010
[14]

Summarizing court decisions

Marie-Francine Moens: “Summarizing court decisions” Information Processing and Management: an International Journal Volume 43 Issue 6, November, 2007 Pages:1748-1764

2007
[15]

CaseSummarizer: A System for Automated Summarization of Legal Texts

Polsley, Seth, Pooja Jhunjhunwala and Ruihong Huang. “CaseSummarizer: A System for Automated Summarization of Legal Texts.” (COLING (2016))

2016
[16]

Assessing sentence scoring techniques for extractive text summarization

RafaelFerreiraa, Lucianode Souza Cabrala, Rafael DueireLins, Gabriel Pereira e Silvaa, Fred Freitas, George D.C. Cavalcantia, Rinaldo Lima, Steven J. Simske, Luciano Favaroc : “Assessing sentence scoring techniques for extractive text summarization” Expert Systems with Applica...

2013
[17]

Survey on Abstractive Text Summarization

Nithin Raphal ; Hemanta Duwarah ; Philemon Daniel: “Survey on Abstractive Text Summarization” 2018 International Conference on Communication and Signal Processing (ICCSP)

2018
[18]

Multi- document text summarization - a survey

Amol Tandel ; Brijesh Modi ; Priyasha Gupta ; Shreya Wagle ; Sujata Khedkar: “Multi- document text summarization - a survey” (2016 International Conference on Data Mining and Advanced Computing (SAPIENCE))

2016
[19]

Casenote Legal Briefs for Torts, Keyed to Prosser, Wade Schwartz Kelly and Partlett
[20]

Review on determining number of Cluster in K-Means Clustering

Trupti M. Kodinariya Dr. Prashant R. Makwana: “Review on determining number of Cluster in K-Means Clustering” (2013, IJARCSMS)

2013
[21]

ROUGE: a Package for Automatic Evaluation of Summaries

Lin, Chin-Yew. “ROUGE: a Package for Automatic Evaluation of Summaries.” In Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), Barcelona, Spain, July 25 - 26, 2004

2004
[22]

A Review On Automatic Text Summarization Approaches

Yogan, Jaya Kumar and Goh, Ong Sing and Halizah, Basiron and Ngo, Hea Choon and Puspalata, C Suppiah “A Review On Automatic Text Summarization Approaches.” Journal Of Computer Science(2016), 12 (4). pp. 178-190. ISSN 1549-3636

2016
[23]

Klopotek and Slawomir T

Mieczyslaw A. Klopotek and Slawomir T. Wierzchon: Modern Algorithms of Cluster Analysis
[24]

Steven Bird, Ewan Klein and Edward Loper: Natural Language Processing with Python
[25]

Data Algorithms by Mahmoud Parsian
[26]

Identification of Rhetorical Roles for Segmentation and Summarization of a Legal Judgment

M. Saravanan • B. Ravindran : “Identification of Rhetorical Roles for Segmentation and Summarization of a Legal Judgment”(B. Artif Intell Law (2010) ) Authors VARUN PANDYA is currently in third year, pursuing B.Tech in Computer Engineering from Pandit Deendayal Petroleum University

2010

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

With the limited human capacity of consuming information, it becomes paramount to extract only useful information out of the vast amount of unstructured data available

INTRODUCTION The amount of information and data has increased tremendously in every field over the years. With the limited human capacity of consuming information, it becomes paramount to extract only useful information out of the vast amount of unstructured data available. Thus, text summarization plays an important role in extracting useful pieces of in...

work page

[2] [2]

Automatic text summarization is broadly of two types:

BACKGROUND Summarization is a challenging sub-task of the broader text-to-text generation field of natural language processing (NLP)[3]. Automatic text summarization is broadly of two types:

work page

[3] [3]

Here the sentences originally present in the document are used to form the summary, with no new sentences being formed

Extractive Summarization: This method involves selecting the most useful and relevant sentences from the document to provide a summary. Here the sentences originally present in the document are used to form the summary, with no new sentences being formed. Extractive summaries produce a set of the most significant sentences from a document, exactly as they...

work page

[4] [4]

Here the gist of the document is represented using computer generated sentences, having close resemblance to the summary as produced by humans

Abstractive Summarization: In this type of summarization, new sentences are formed which are relevant to the document. Here the gist of the document is represented using computer generated sentences, having close resemblance to the summary as produced by humans. Abstractive text summarization captures the salient features of the text corpus and summary is...

work page

[5] [5]

Fast Legal EXpert CONsultant

EXISTING APPROACHES Different kinds of methods have been proposed for text summarization in the legal domain. The earliest of works in this area include the “Fast Legal EXpert CONsultant” (FLEXICON) system developed by Gelbart and Smith (Gelbartand Smith, 1991a). FLEXICON is keyword-based, referencing against a large database of terms to find important re...

work page 1999

[6] [6]

Pre-processing Pre-processing of the document is a vital part of any text summarization approach

THE PROPOSED SYSTEM The implementation is carried out in three different steps: pre-processing, clustering of similar sentences, extracting top ranked sentences from each cluster 4.1. Pre-processing Pre-processing of the document is a vital part of any text summarization approach. It is the foremost step of the proposed approach which includes removing of...

work page

[7] [7]

sent_values = [sum(tf-idf of all words of sentence) / sum(tf-idf of all words of document)]

Tf-idf score of each sentence: The sum of tf-idf value of all words of the sentence is divided by the sum of tf-idf value for all the words present in the document. sent_values = [sum(tf-idf of all words of sentence) / sum(tf-idf of all words of document)]

work page

[8] [8]

Sentence similarity to title: The sentences similar to the title of the case generally present the important facts related to the case, thus aiding in generating a useful and relevant summary. The similarity score to the title is calculated as: Title_score =( (length of similar words) * 0.1 ) / length of words in title) Here only the noun form words ['NN'...

work page