{"id":"8cd43d56-8fd0-477b-9eae-6511e5d1cfef","arxiv_id":"2507.18952","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":1.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A legal document summarization framework is described, but the experiments use four non-legal summarization datasets and generic equations, so the claimed judicial efficiency improvements are not established.","lead":"The paper proposes using large language models and BERT to automatically extract key sentences from legal documents and summarizes expected efficiency gains. A generalist should read it because legal automation is a high-value application, but this version does not supply legal datasets, code, or a concrete method, so the headline gains are unverified.","discovery_kind":"unclear","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central legal-efficiency claim is unsupported because the only summarization experiments use non-legal corpora (WikiLingua, WikiWeb2M, GameWikiSum, FinDSum), contradicting the paper's repeated assertion of 'real-world legal datasets.'","rationale":"I agree with the reader's weakest_assumption. The manuscript's own Section IV-A contradicts its repeated claim of legal datasets, so this is not a matter of outside consensus; it is an internal inconsistency. The central claim of judicial efficiency is untestable without legal-domain evaluation. The workflow metrics in Section V-F are unsupported assertions, but the dataset mismatch alone is sufficient to reject. No code, data splits, or prompts are provided, and the equations in Section III are generic definitions, so reproducibility is also lacking. The reader's REJECT verdict is appropriate, and my review does not change it. I set verdict_should_be to UNCHANGED to reflect that no adjustment is needed.","tokens_in":10997,"tokens_out":3448,"duration_ms":33314,"concrete_test":"Run the same GPT-4 and BERT configurations from Section IV-C on an established legal-summarization benchmark such as BillSum, SCALE, or the LexGLUE EUR-Lex subset, using the reported ROUGE protocol, and also sample 100 documents from each of the four Section IV-A corpora for manual genre classification. If ROUGE scores on legal data fall substantially below Table I, or if fewer than 5 percent of sampled documents are legal texts, the central legal-efficiency claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that the framework automates legal document summarization and improves judicial efficiency—requires that the evaluation corpora be legal in genre and that the efficiency metrics be measured. Neither condition is met. Section IV-A lists only WikiLingua, WikiWeb2M, GameWikiSum, and FinDSum; these are multilingual Wikipedia, multimodal Wikipedia, game-wiki, and financial-document corpora, respectively, with no legal-document subset. Yet the abstract, introduction, and conclusion repeatedly assert 'comprehensive experiments conducted with actual legal datasets.' That direct contradiction means Table I's ROUGE, accuracy, and F1 values are scores on non-legal text, so they provide no evidence for legal-domain transfer. In addition, Section II-B's 'baselines' are not summarization systems (voice recognition, GitHub workflow automation, hyper-automation, testbench generation, defect detection), so there is no comparison against existing summarization methods. Section V-F's workflow gains (15 to 5 hours review time, 65 to 90 percent detection rate, 5 to 15 cases per day) are stated without any measurement protocol, subject pool, or link to the experiments. The load-bearing assumption of legal representativeness is therefore false on the face of the manuscript.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Legal Document Summarization framework that combines automated key-segment extraction and summarization (Eqs. 1 and 2) with machine-learning pattern recognition, and it claims that this automation substantially improves judicial efficiency. Experiments are reported for GPT-4 and BERT on WikiLingua, WikiWeb2M, GameWikiSum, and FinDSum, with ROUGE, accuracy, relevance, completeness, and F1 scores in Table I, and Section V-F reports workflow gains including a 66.67% reduction in review time and a 200% increase in cases processed per day. The abstract, introduction, and conclusion repeatedly state that the method was validated on real-world legal datasets, but the paper describes no legal-document corpus and no experimental protocol for the workflow-efficiency claims.","tokens_in":11230,"tokens_out":3160,"duration_ms":29478,"significance":"If the central claims were substantiated, the paper would address an important applied problem: automated legal summarization could reduce the time lawyers and judges spend on document triage. The problem framing is relevant and the paper identifies a plausible use of LLMs for extraction and summarization. However, the manuscript provides no machine-checked proofs, reproducible code, or complete experimental protocol, and the evidence that connects the experiments to legal documents is absent. The claimed efficiency gains are not tied to any measurement, and several tables report methods never introduced in the methodology. As it stands, the contribution is a set of formal definitions and performance numbers on non-legal corpora, not a demonstrated legal summarization system.","major_comments":[{"comment":"The four datasets used in Table I—WikiLingua, WikiWeb2M, GameWikiSum, and FinDSum—are not legal document corpora, yet the abstract, Section I, and Section VI repeatedly state that experiments were conducted on real-world legal datasets. Since every ROUGE, accuracy, and F1 score in Table I is computed on non-legal text, the central claim that the framework summarizes legal documents with high quality is untested.","section":"Section IV-A and Table I"},{"comment":"The workflow gains reported in Section V-F (review time reduced from 15 to 5 hours, detection rate from 65% to 90%, error rate from 20% to 5%, and cases processed per day from 5 to 15) are stated without any measurement protocol, subject population, document sample, or link to the experiments in Section V-A. Equation (6) defines an efficiency percentage but does not supply the values of Tmanual and Tautomated for these claims, so the 66.67% and 200% figures have no evidentiary basis.","section":"Section V-F and Eq. (6)"},{"comment":"The baselines in Section II-B are not summarization systems (voice recognition, GitHub workflow automation, hyper-automation, testbench generation, and defect detection), so there is no comparison against existing summarization methods. In addition, Tables II and III report results for methods (Rule-Based, Supervised Learning, Deep Learning, Hybrid Approach; SVM, Random Forest, Naive Bayes, Decision Tree, Logistic Regression) that are not described in the methodology, making the numbers in these tables uninterpretable and disconnected from the proposed framework.","section":"Section II-B and Tables II-III"},{"comment":"Equations (1), (2), (3), and (5) are definitions of the desired extraction and summarization behavior, not derivations of a concrete trained system: the relevance threshold theta in Eq. (1) is never specified, the model parameters P and theta are not tied to a training procedure on legal data, and the scoring function score(s, M) is not defined. The claim that models are trained on annotated legal data (Section III-A) is unsupported because Section IV-A lists only non-legal datasets; the formal apparatus therefore does not establish a working method.","section":"Section III-A through III-C, Eqs. (1)-(5)"}],"minor_comments":[{"comment":"Section IV-D states that the experiments use 500 annotated legal documents averaging 2,000 tokens, but no such legal dataset is described in Section IV-A or anywhere else in the paper; this contradicts the dataset list and needs clarification.","section":"Section IV-D"},{"comment":"Table II presents precision, recall, and F1 as percentages while Table III uses decimal values; the units and their relationship should be unified and stated explicitly.","section":"Tables II and III"},{"comment":"The summation notation in Eq. (4) appears as 'mX' and should be rendered as a standard sum over i = 1 to m; the current typesetting makes the loss definition unclear.","section":"Eq. (4)"},{"comment":"Figures 2 and 3 lack axis labels, error bars, and any indication of statistical significance, so the claimed upward trend in pattern recognition performance cannot be verified from the figures alone.","section":"Figures 2 and 3"},{"comment":"Reference formatting is inconsistent (for example, 'V oice' and 'teusz Litwin' are misspelled), and references [38] and [39] are cited in the dataset section but are not datasets used in the summarization evaluation.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claims of legal-domain validation are directly contradicted by the actual datasets listed in Section IV-A, and the workflow-efficiency numbers in Section V-F have no measurement protocol. These are load-bearing issues that cannot be fixed by local revisions; the evaluation would need to be redesigned around a real legal corpus and a fully specified experimental protocol. The paper also reads like an incomplete draft, with Tables II and III disconnected from the methodology. Given the journal's standards, rejection is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honestly, this paper is a non-starter. The central claim—that a Legal Document Summarization framework improves judicial efficiency—rests on experiments that never touch legal text. Section IV-A lists WikiLingua, WikiWeb2M, GameWikiSum, and FinDSum, none of which are legal datasets, while the abstract and introduction repeatedly say \"actual legal datasets.\" That direct contradiction alone sinks the paper.\n\nTo its credit, the paper is readable and the related work shows the authors have surveyed legal NLP. The motivation is reasonable: legal review is expensive and summarization could help. But there is no new algorithm, no new benchmark, and no code or data shipped.\n\nBeyond the dataset mismatch, the workflow improvements in Section V-F—15 to 5 hours review, 65% to 90% detection, 5 to 15 cases per day—are asserted without any measurement protocol or subject pool. They read as illustrative, not measured. The baselines in Section II-B are not summarization systems; they are voice recognition and GitHub automation, so there is no comparison against existing summarizers. The methodology equations are definitions, not derivations: θ in Eq. (1) is never specified, and Eq. (6) merely defines an efficiency ratio. Tables II and III describe models not present in the methodology—there is no Rule-Based or Hybrid Approach in the experiments, and the SVM/Random Forest numbers appear from nowhere.\n\nWho is this for? Readers doing a broad survey of legal AI might skim it, but it will not change anyone's work. It does not deserve peer review as it stands; it should be desk rejected and the authors directed to actually run on legal data.","headline":"A legal-summarization paper whose evaluation uses no legal data and whose workflow numbers are asserted, so the central claim is untested.","tokens_in":11744,"tokens_out":1950,"would_cite":false,"duration_ms":17105,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Legal Document Summarization framework built on GPT-4 and BERT claims to automate key-information extraction from legal texts, reducing review time from 15 to 5 hours and tripling daily case capacity.","keywords":["legal document summarization","judicial efficiency","automation detection","key information extraction","large language models","GPT-4","BERT","ROUGE"],"falsifier":"Run the same GPT-4 and BERT pipelines on a held-out set of actual court opinions or legal briefs with human reference summaries and measure lawyer review time against manual reading; if the review-time reduction does not approach 66.67% or the key-information detection rate does not climb from 65% to 90%, the central efficiency claim is not supported.","tokens_in":10776,"feed_emoji":"⚖️","tokens_out":9174,"duration_ms":91070,"temperature":0.7,"pith_summary":"Legal Document Summarization is a proposed framework for automatically finding and extracting the key information in legal texts so that attorneys can review a summary instead of the full document. The paper claims that combining a relevance-threshold extraction step with GPT-4 or BERT summarization, plus a hybrid key-information detector, produces concise summaries while cutting the average review time from 15 hours to 5 hours. It reports that detection of crucial details rises from 65% to 90%, summary error rates fall from 20% to 5%, and daily case capacity grows from 5 to 15 cases. A sympathetic reader would take the contribution as evidence that today's large language models can take over the triage portion of legal document review, freeing practitioners for analysis and decisions. The experimental evidence is drawn from four public summarization corpora paired with a 500-document annotated set.","feed_headline":"Automated summaries cut legal review time by two-thirds","feed_subtitle":"Paper reports detection rising to 90 percent and daily case capacity tripling from 5 to 15.","key_machinery":"The central mechanism is a formal extraction-summarization loop. Given document $D$, the framework computes $K = \\mathrm{Extract}(D) = \\{k_i \\mid k_i \\in D,\\ \\mathrm{Relevance}(k_i) \\geq \\theta\\}$, a set of sentences whose relevance score passes a threshold $\\theta$, and then generates $S = \\mathrm{Summarize}(K)$, a concise summary parameterized by the trained model. Sentence selection is the same operation in scoring form: $S_{\\mathrm{summary}} = \\arg\\max_S \\sum_{s \\in S} \\mathrm{score}(s,M)$, and efficiency is defined as $E = (T_{\\mathrm{manual}} - T_{\\mathrm{automated}})/T_{\\mathrm{manual}} \\times 100\\%$. This machinery carries the whole argument because every claimed gain, from review time to daily case capacity, is the reported output of this two-stage process running with GPT-4 or BERT and a hybrid key-information detector. The equations specify the pieces, but the actual performance comes from the model choices and the threshold design rather than from an analytic guarantee.","core_discovery":"The paper's central claim is that automated legal summarization is not just a convenience but a workflow transformation: a two-stage pipeline that extracts relevance-scored segments and then compresses them can preserve the content a lawyer needs while dramatically reducing the time spent reading. The reported numbers are concrete: 66.67% less review time (15 to 5 hours), a 38.46% increase in key-information detection (65% to 90%), a 75% drop in summary errors (20% to 5%), and a 200% rise in cases handled per day (5 to 15). The paper attributes these gains to the framework's use of large language models and machine-learned relevance scoring, with GPT-4 outperforming BERT across every dataset and a hybrid detector giving the best precision and recall. If the framework works as described, legal professionals could delegate document triage to software and reserve human judgment for the analytical decisions that follow reading a summary.","pith_inferences":["A direct testable extension would rerun the method on real court opinions with lawyer-written reference summaries; the four public datasets are generic summarization benchmarks, so legal-domain transfer is the unconfirmed step.","The 66.67% time saving is computed from assumed baselines of 15 manual hours and 5 automated hours rather than measured end-to-end workflow data, so the real ratio depends on the verification work that still surrounds automated summaries.","A natural deployment the paper does not spell out is a confidence threshold: route only high-confidence summaries around human review, leaving low-confidence documents to lawyers, which would preserve accuracy while realizing most of the speed gain."],"forward_implications":["Legal staff could cut per-document review from 15 hours to 5 hours, reallocating the saved time to analysis and decision-making.","A key-information detection rise from 65% to 90% would mean fewer overlooked clauses, citations, or case elements during triage.","A summary error-rate drop from 20% to 5% would make automated summaries dependable enough to use as the first-pass review artifact.","Tripling daily case capacity from 5 to 15 would allow a legal team to handle a larger docket without adding review hours."],"supporting_citations":[{"why":"Provides the few-shot learning premise that lets large models summarize with minimal task-specific fine-tuning.","marker":"[1]"},{"why":"Grounds the claim that large language models handle a broad range of tasks, motivating the GPT-4 choice.","marker":"[2]"},{"why":"Supports the use of human-feedback alignment so model summaries follow user intent.","marker":"[3]"},{"why":"Supplies the WikiLingua benchmark used for the paper's multilingual summarization results.","marker":"[34]"},{"why":"Supplies the WikiWeb2M multimodal corpus behind one set of reported ROUGE scores.","marker":"[35]"},{"why":"Provides the GameWikiSum multi-document summarization dataset used in evaluation.","marker":"[36]"},{"why":"Provides the FinDSum long-text and multi-table summarization benchmark that gives the paper its highest scores.","marker":"[37]"}],"fun_headline_variants":["AI legal summaries cut review time by 66%","Legal AI triples cases per day, boosts detection to 90%","GPT-4 legal summaries: 3x caseload, 75% fewer errors","Automated legal docs: 90% key info, 10 hrs saved per case"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the four evaluation corpora, WikiLingua, WikiWeb2M, GameWikiSum, and FinDSum, are close enough to genuine legal documents that their summary quality and speed numbers carry over to real judicial review.","fun_headline_variants_meta":{"raw":{"variants":["AI legal summaries cut review time by 66%","Legal AI triples cases per day, boosts detection to 90%","GPT-4 legal summaries: 3x caseload, 75% fewer errors","Automated legal docs: 90% key info, 10 hrs saved per case"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1299,"prompt_tokens":894,"completion_tokens":405,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":323}},"tokens_in":510,"tokens_out":405,"duration_ms":4971,"temperature":1.0,"reasoning_tokens":323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:04:05.008383+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same GPT-4 and BERT pipelines on a held-out set of actual court opinions or legal briefs with human reference summaries and measure lawyer review time against manual reading; if the review-time reduction does not approach 66.67% or the key-information detection rate does not climb from 65% to 90%, the central efficiency claim is not supported.","supporting_citations":[{"cited_title":"Gamewikisum: a novel large multi- document summarization dataset,","cited_arxiv_id":null,"evidence_quote":"Provides the GameWikiSum multi-document summarization dataset used in evaluation."}],"review_version":1}