{"id":"88beb981-4305-4c4c-8fcc-271e911df6f4","arxiv_id":"1908.09119","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A k-means and tf-idf sentence-ranking method scored higher ROUGE values than three online summarizers on one 200-page Australian legal case.","lead":"This paper proposes an automatic extractive summarizer for legal case files that clusters sentences with k-means and ranks them by tf-idf and title similarity. It reports that on one Australian court case, this method scored higher on ROUGE text-overlap metrics than three online summarizers, though the comparison uses only a single case.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The comparative claim rests on a single length-matched evaluation against web tools with unreported output lengths, making the reported ROUGE advantage attributable to evaluation bias rather than method quality.","rationale":"The reader's weakest assumption correctly identifies the evaluation protocol as the pivot of the paper's central claim. My own read of the manuscript confirms this: the only evidence for 'performs favourably well' is the single-case ROUGE table, and the length-matching procedure explicitly described in Section 5 introduces a confound that directly affects the measured scores. The paper is transparent about its preliminary nature, explicitly stating that attorney-based tests were not conducted, which is a limitation rather than a deception. The method's components are standard and the arithmetic in Table 1 is consistent. However, the empirical claim is not supported by the evidence as presented, so the reader's REJECT verdict is appropriate and I see no reason to change it. A controlled multi-case experiment with matched summary lengths and variance reporting would be the decisive test.","tokens_in":5214,"tokens_out":6514,"duration_ms":63182,"concrete_test":"Run the proposed system and all three baselines on a set of at least 10 legal cases from AustLII, forcing each system to produce exactly the same number of sentences (e.g., the reference length and also a fixed 50-sentence length). Use multiple reference summaries per case or bootstrap resampling over sentences, and report ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-W F-measures with 95% confidence intervals. If the proposed system does not maintain its rank order across cases or its confidence intervals overlap the baselines, the comparative claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the proposed k-means plus tf-idf hybrid performs favorably against TextSummarizer, SplitBrain, and ESummarizer (Section 6, Table 1). This claim is load-bearing on the assumption that the ROUGE scores in Table 1 provide a fair ranking. That assumption is insecure in two concrete ways. First, Section 5 states the proposed system generated approximately 150 sentences 'to match its length with the provided summary for better comparison.' The paper reports no output lengths for the three baselines; if their default outputs were shorter (typical for online summarizers), their lower recall and F-measures are an artifact of length mismatch, not of content selection. ROUGE F-measures are length-sensitive, so comparing a length-tuned candidate to length-uncontrolled baselines is not a controlled experiment. Second, the evaluation uses a single case and a single reference summary, so even if lengths were matched, the reported differences could be noise. The paper itself acknowledges that real-world tests with attorneys have not been conducted (Section 8). Without multi-document evaluation and confidence intervals, the favorable comparison is not established. The low absolute ROUGE-2 F-measure of 5.83 also contradicts the conclusion that the summary 'closely resembles' the attorney's summary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an unsupervised extractive text summarization method for legal case documents. The approach consists of pre-processing the input case file, embedding sentences with tf-idf vectors, clustering the sentences using k-means, and then selecting the top-ranked sentences from each cluster using a combination of tf-idf score and similarity to the case title. The authors evaluate the method on a single Australian legal case (Rush v Nationwide News Pty Ltd (No 7)) by comparing the automatically generated summary with the attorney-prepared summary using ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-W scores against three online summarization tools (TextSummarizer, SplitBrain, ESummarizer). The reported table shows higher ROUGE values for the proposed method, and the authors conclude that it performs favorably and that the generated summary closely resembles the attorney's summary.","tokens_in":5498,"tokens_out":4554,"duration_ms":46583,"significance":"If the reported comparative advantage were valid, the proposed hybrid would be a lightweight, unsupervised baseline for legal-case summarization that avoids the need for training data. The pipeline is clearly described and simple enough to reproduce, and the paper explicitly addresses the large size of legal documents as a motivating challenge. However, the empirical evidence presented is not sufficient to establish the central claim. The comparison is uncontrolled with respect to output length, is based on a single case and a single reference summary, and lacks any significance testing. The low absolute ROUGE scores further undermine the qualitative claim of close resemblance. The idea is reasonable, but the evaluation must be substantially strengthened before the comparative conclusion can be accepted.","major_comments":[{"comment":"The proposed system's output length is deliberately matched to the gold summary: Section 5 states that 'approximately 150 sentence summary was generated ... to match its length with the provided summary for better comparison.' The paper reports no output lengths for TextSummarizer, SplitBrain, or ESummarizer. Since ROUGE recall and F-measure are sensitive to candidate length, comparing a length-tuned candidate with length-uncontrolled baselines is not a fair test. The reported advantages in Table 1 could therefore be an artifact of length matching rather than better content selection.","section":"Section 5 and Table 1"},{"comment":"The evaluation uses exactly one legal case (Rush v Nationwide News Pty Ltd (No 7)) and exactly one human reference summary. No error bars, confidence intervals, or significance tests are provided. The differences among the systems in Table 1 are therefore indistinguishable from random variation. The assertion in Section 6 that the proposed method 'performs favourably well' is not supported by a single-case comparison, and the paper's own statement in Section 8 that real-world tests with attorneys have not been conducted does not mitigate this gap.","section":"Section 6 and Table 1"},{"comment":"The conclusion that the generated summary 'closely resembles' the attorney's summary is contradicted by the low absolute ROUGE scores in Table 1 (ROUGE-1 F=27.88, ROUGE-2 F=5.83, ROUGE-L F=33.5). A ROUGE-2 F-measure of 5.83 indicates very little bigram overlap, which does not support a qualitative claim of close resemblance. If this claim is intended, it needs to be backed by human evaluation or a more appropriate similarity metric; otherwise the wording should be moderated.","section":"Section 8 and Table 1"}],"minor_comments":[{"comment":"The metric name is ROUGE, not ROGUE; the misspelling appears in the abstract, Sections 5 and 6, and in Table 1.","section":"Throughout"},{"comment":"The formula for title similarity has unbalanced parentheses: 'Title_score =( (length of similar words) * 0.1 ) / length of words in title)' has an extra closing parenthesis.","section":"Section 4.3"},{"comment":"The paper mentions algorithms such as the Elbow method for choosing the optimum number of clusters, but the actual value of K used in the experiments is never reported, which hinders reproducibility.","section":"Section 5"},{"comment":"The table header formatting makes it hard to read, and the name 'ESUMMERIZER' in the table is inconsistent with 'ESummarizer' used in Section 6.","section":"Table 1"},{"comment":"References [11] through [14] are listed but do not appear to be cited in the body of the paper, and reference [3] appears to combine multiple distinct sources.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper appears to be an early-stage student project. The method is a straightforward combination of k-means and tf-idf, and the evaluation is not at the level expected for a journal publication. The central comparative claim is unsupported because of the length-matching issue and the single-case design. I would consider a substantially revised version with a multi-case evaluation, controlled baseline lengths, and significance testing, but the current manuscript does not meet the bar."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on 1908.09119: it's an honest student project, not a research result. The method is k-means over tf-idf sentence vectors plus a title-similarity rerank, applied to one 200-page Australian case. Nothing in the pipeline is new, but the paper does what it says: it produces a length-controlled extractive summary in under three minutes without training data.\n\nWhat it does well: the writing is clear, the approach is straightforwardly reproducible from the description, and it cites the relevant legal-NLP background (CaseSummarizer, FLEXICON, etc.). It also includes an explicit limitation: real-world attorney testing hasn't been done. That's good hygiene.\n\nThe soft spots are the ones the stress test flags. Section 5 says the system generated about 150 sentences 'to match its length with the provided summary for better comparison.' That's a mild form of peeking at the reference; ROUGE F-measures are length-sensitive. The paper doesn't report the output lengths of TextSummarizer, SplitBrain, or ESummarizer, so if those tools returned shorter summaries (typical for online tools), their recall and F-measures are depressed by length mismatch, not by content selection. Add the single case, single reference, and no error bars, and the table in Section 6 cannot support the conclusion that the hybrid 'performs favourably well.' A ROUGE-2 F of 5.83 also undercuts the claim that the output 'closely resembles' the attorney's summary.\n\nThe central problem is not the method - it's the comparison. I don't see this as a fatal flaw in the idea; the idea is fine as a baseline. But the paper overstates what the data show, and the evaluation protocol would need to be rebuilt before the comparative claim means anything.\n\nWho's it for? A workshop in legal NLP or a course project repository. A serious referee? I wouldn't send this to a main-track conference; the contribution is too incremental and the evaluation too thin. If the authors added a few cases, fixed the length matching, and reported confidence intervals, it could be a decent short paper. As is, it's a reasonable technical note.\n\nRecommendation: desk-reject for a research venue, but don't ignore it entirely if you work on legal summarization - it's a clean description of a useable baseline.","headline":"Honest but thin student project; the k-means+tf-idf recipe is unremarkable and the single-case, length-matched evaluation doesn't support the 'favourably well' claim.","tokens_in":5950,"tokens_out":2124,"would_cite":false,"duration_ms":21318,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An unsupervised k-means and tf-idf hybrid, tested on one 200-page Australian case, reports higher ROUGE F-measures than three existing legal summarizers.","keywords":["automatic text summarization","legal domain","k-means clustering","tf-idf","extractive summarization","ROUGE evaluation","unsupervised summarization"],"falsifier":"Take a set of, say, twenty legal cases, obtain two or more independent human reference summaries for each, run the hybrid and the three baseline tools with all outputs fixed to a common length, and compare average ROUGE scores; if the hybrid no longer beats the baselines, or if random sentence selection from clusters matches its scores, the reported ranking would be refuted.","tokens_in":5025,"feed_emoji":"⚖️","tokens_out":10375,"duration_ms":95445,"temperature":0.7,"pith_summary":"The paper proposes that a legal case file can be summarized automatically by an unsupervised, extractive pipeline: preprocess the text, represent sentences as tf-idf vectors, cluster them with k-means, rank sentences inside each cluster by tf-idf weight plus similarity to the case title, and take the top sentences from every cluster. Tested on a single 200-page Australian legal case, the method generated a roughly 150-sentence summary in under three minutes. The reported ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-W F-measures place the hybrid above TextSummarizer, SplitBrain, and ESummarizer on all four metrics. The paper concludes that this cluster-based, coverage-oriented extraction yields a brief close to the attorney's own summary and offers a practical starting point for legal summarization without requiring training data.","feed_headline":"Cluster-then-rank summary beats three legal summarizers","feed_subtitle":"Unsupervised hybrid scores higher ROUGE on a 200-page case than three tools, in minutes.","key_machinery":"The load-bearing machinery is the cluster-then-rank selection process. Sentences are converted to tf-idf (term frequency-inverse document frequency) vectors, grouped by the k-means clustering algorithm into K similarity-based clusters, and then each sentence is scored as the sum of its normalized tf-idf weight and a title-similarity score that compares only noun forms against the case title. Equal numbers of top-ranked sentences are taken from each cluster, so the summary length is controlled and the final output is intended to represent the whole document with minimal redundancy.","core_discovery":"The central claim is that ranking sentences by normalized tf-idf score plus similarity to the case title, and doing that within k-means clusters rather than over the whole document, makes an extractive summarizer competitive for long legal texts. ROUGE is a set of standard n-gram overlap measures for comparing machine summaries with human reference summaries; on the one case reported, the hybrid achieved ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-W F-measures of 27.88, 5.83, 33.50, and 11.38, each above TextSummarizer, SplitBrain, and ESummarizer. The paper attributes the result to low redundancy: because sentences are drawn from every cluster, the summary covers different parts of the case file instead of concentrating on the most frequent words. It also claims the method is computationally favorable and unsupervised, unlike approaches that need a training corpus.","pith_inferences":["An implied but untested consequence is that the reported ranking may depend on output length: the proposed system's summary was manually set to the reference length of about 150 sentences, whereas the online baselines likely produced shorter outputs, so a length-matched comparison would be the stronger test.","Because the evaluation uses one case and one human reference summary, the margins are evidence of feasibility rather than general superiority; running the same pipeline across many cases with multiple independent reference summaries would show whether the advantage persists.","A random-control experiment would sharpen the claim: if sentences selected at random from the clusters achieved similar ROUGE scores, then the tf-idf and title ranking, not the clustering, would be doing the work."],"forward_implications":["According to the paper, legal case files can be summarized without labeled training data, which matters because annotated legal corpora are scarce.","If the reported speed and stability hold, several-hundred-page cases can be reduced to usable briefs in minutes, while a graph-based method the paper tried on the same input failed with a memory error.","Because the summary draws equally from every cluster, the method is designed to cover the full case rather than repeat dominant vocabulary, which is the paper's stated reason for its stronger ROUGE results.","The paper's own next steps are automatic cluster-count selection, additional sentence-ranking features, and rhetorical-role structures for more formal legal briefs."],"supporting_citations":[{"why":"Supplies the ROUGE evaluation package used to compute the n-gram overlap scores on which the entire method comparison rests.","marker":"[9]"},{"why":"Underpins the paper's contrast between recent approaches that require training data or a corpus and the proposed unsupervised method.","marker":"[10]"},{"why":"Documents prior legal text summarization systems and the difficulty of legal documents, which motivates the specific hybrid design.","marker":"[3]"}],"fun_headline_variants":["Legal summary hybrid beats three tools on ROUGE","Cluster-then-rank tops three legal summarizers","K-means tf-idf summary outranks three baselines","Hybrid method leads legal summarization comparison","Unsupervised clustering improves legal case summaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that ROUGE scores computed against a single lawyer-written summary, with the proposed system's output length manually matched to that summary, provide a fair basis for ranking the methods.","fun_headline_variants_meta":{"raw":{"variants":["Legal summary hybrid beats three tools on ROUGE","Cluster-then-rank tops three legal summarizers","K-means tf-idf summary outranks three baselines","Hybrid method leads legal summarization comparison","Unsupervised clustering improves legal case summaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1464,"prompt_tokens":848,"completion_tokens":616,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":542}},"tokens_in":464,"tokens_out":616,"duration_ms":6245,"temperature":1.0,"reasoning_tokens":542,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:20:51.258126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of, say, twenty legal cases, obtain two or more independent human reference summaries for each, run the hybrid and the three baseline tools with all outputs fixed to a common length, and compare average ROUGE scores; if the hybrid no longer beats the baselines, or if random sentence selection from clusters matches its scores, the reported ranking would be refuted.","supporting_citations":[{"cited_title":"The legal cases consist of the headnotes and summary within the case file","cited_arxiv_id":null,"evidence_quote":"Supplies the ROUGE evaluation package used to compute the n-gram overlap scores on which the entire method comparison rests."},{"cited_title":"The relevance and usefulness of each sentence in the summary changes with the person evaluating them","cited_arxiv_id":null,"evidence_quote":"Underpins the paper's contrast between recent approaches that require training data or a corpus and the proposed unsupervised method."},{"cited_title":"Here the sentences originally present in the document are used to form the summary, with no new sentences being formed","cited_arxiv_id":null,"evidence_quote":"Documents prior legal text summarization systems and the difficulty of legal documents, which motivates the specific hybrid design."}],"review_version":1}