{"id":"3c82df5e-c591-427c-90b4-1bfd4007c5f1","arxiv_id":"2505.17059","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A lightweight medical summarization app built by fine-tuning three FLAN-T5-Large models reportedly beats GPT-4 on structured medical report summaries, while GPT-4 wins the question-extraction task and is mixed on conversations.","lead":"Medalyze is an app that adapts three small language models to summarize medical reports, pull health problems from doctor-patient conversations, and find the main question in a medical passage. The authors compare its outputs with GPT-4 and claim their smallest model wins on structured medical reports, but their own tables show GPT-4 wins on other tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"M-Passage 'superiority over GPT-4' rests on a test set that the paper never shows to be disjoint from fine-tuning data; §IX-B's wording suggests it may be derived from the training pool, so Table 7 could measure memorization rather than generalization.","rationale":"The reader's weakest assumption is reasonable, but I rank train/test overlap above the 512-token context issue for the central claim. The reader's own strongest claim is specifically about M-Passage, and M-Passage inputs (20-141 words) fit within 512 tokens, so the context issue does not threaten that pillar. Section IX-C-3 already concedes GPT-4 wins M-Question, and Section IX-C-4 shows mixed M-Conversation results; thus M-Passage is the only result that can justify 'superior performance.' The paper's wording in IX-B ('test dataset is derived from the original training dataset') can be read as an admission that the test set came from the same pool used to fine-tune, and no split is described anywhere. If that is true, the model was evaluated on training references, and the comparison is invalid. The concrete test would resolve by looking at the actual split and checking for duplicates. This does not change the REJECT verdict; it sharpens the reason for it. The system architecture and deployment details are described in good faith, but the central scientific claim depends on the evaluation, and that evaluation is not currently verifiable.","tokens_in":17057,"tokens_out":7749,"duration_ms":76716,"concrete_test":"Release the fine-tuning/test split or the preprocessing code, then run an exact and near-duplicate search (e.g., normalized n-gram Jaccard >= 0.8) between every M-Passage test input/reference and every fine-tuning example. Recompute Table 7 on the confirmed disjoint subset and report per-sample scores and standard errors. If the gap over GPT-4 disappears or drops below noise, the headline claim is an artifact of data leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The pivotal result is Table 7, the only comparison in which the fine-tuned model clearly beats GPT-4 on all four metrics. That table is trustworthy only if the M-Passage test items were not seen during fine-tuning. The paper never states that it held out a test split. Section IX-B says 'the test dataset is derived from the original training dataset,' and Section V describes the Stanford resource as 'the dataset used for training the models.' On the plain reading, the test input/target pairs come from the same pool used for fine-tuning, so the reported BLEU 0.0982, ROUGE-L 0.3728, BERTScore 0.6533, and SpaCy 0.8413 could reflect memorization rather than generalization. Without a released split, code, or duplicate check, the central claim of superiority over GPT-4 is not established. The 512-token context issue is also serious, but it affects M-Conversation; the train/test overlap, if real, attacks the one comparison that actually supports the headline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes Medalyze, an application built around three fine-tuned FLAN-T5-Large models for (1) summarizing structured medical passages, (2) extracting health issues from patient-doctor conversations, and (3) identifying the central question in a medical text. The authors also present a web/mobile platform with a Flask API and YugabyteDB storage, and they compare their models against GPT-4 on BLEU, ROUGE-L, BERTScore, and SpaCy Similarity. The abstract claims superior performance over GPT-4 in domain-specific tasks, citing the experimental evaluation.","tokens_in":17316,"tokens_out":5678,"duration_ms":52635,"significance":"If validated, the headline result—that a 780M-parameter fine-tuned model beats GPT-4 on domain-specific medical summarization—would be practically significant for lightweight, privacy-preserving deployment. The paper also demonstrates a full application pipeline, which is a useful engineering contribution. The authors should be credited for using an external benchmark (the Stanford dataset [38]) and a strong baseline (GPT-4), and for releasing no code or split, which limits reproducibility. However, the central comparative claim is not supported by the evidence as presented: the evaluation appears to use test items derived from the training pool, the abstract overstates results that the paper's own tables contradict, and the handling of long conversations beyond the model's context window is unexplained. These issues are load-bearing for the paper's main contribution.","major_comments":[{"comment":"The paper states that 'the test dataset is derived from the original training dataset' (Section IX-B), and Section V describes the Stanford resource as 'the dataset used for training the models.' This implies the M-Passage test items in Table 7 come from the same pool used for fine-tuning. If so, the reported BLEU 0.0982, ROUGE-L 0.3728, BERTScore 0.6533, and SpaCy 0.8413 could reflect memorization rather than generalization, invalidating the comparison with GPT-4. The authors must specify an explicit train/test split (e.g., disjoint subsets of the Stanford data), report the number of test samples, and ideally release the split or code. Without a held-out test set, the central claim of superiority over GPT-4 is not established.","section":"Section IX-B, Table 7"},{"comment":"The abstract claims 'superior summarization performance over GPT-4 in domain-specific tasks,' but Tables 8 and 9 show GPT-4 outperforming M-Question on all metrics (e.g., BLEU 0.0911 vs. 0.0084, ROUGE-L 0.2564 vs. 0.1938, BERTScore 0.6524 vs. 0.5552) and M-Conversation on BERTScore (0.5428 vs. 0.4707). The paper itself acknowledges in Section IX-C-3 that 'GPT-4 performance is superior across all metrics' for M-Question. The abstract therefore overstates the findings; the data support superiority only on the M-Passage task, and even that is contingent on the unresolved train/test overlap issue. The abstract must be revised to reflect the evidence.","section":"Abstract, Section IX-C"},{"comment":"The M-Conversation model is evaluated on long inputs of 2992 to 3050 words (Section VII-B-2), while Table 5 lists FLAN-T5-Large's context window as 512 tokens. The paper never describes how such long inputs are handled: whether they are truncated, chunked, or processed with a sliding window. If inputs are truncated, the model sees only a prefix of the conversation, so its outputs cannot be meaningfully compared with GPT-4's full-context outputs; if they are not truncated, the model cannot process them at all given its context limit. The authors must report the exact input processing procedure and discuss its effect on the conversational summarization results.","section":"Section VII-B-2, Section IX-C-4, Table 5"},{"comment":"The evaluation reports average scores over the test samples with no error bars, confidence intervals, significance tests, or even the number of samples per task. The per-sample figures (Figs. 7–9) show considerable fluctuation, yet no statistical analysis is provided. For the headline M-Passage comparison, at minimum the authors should report the test set size, the variance across samples, and bootstrap confidence intervals for each metric, so the reader can assess whether the reported differences are actually robust.","section":"Section VII-B, Section IX-B"}],"minor_comments":[{"comment":"The subsection heading 'Fine-turning process' contains a typo; it should be 'Fine-tuning process'.","section":"Section VI"},{"comment":"The metric descriptions use 'the original text' ambiguously; the comparisons are against the human-written reference summary, not the original input text.","section":"Section VII"},{"comment":"The BLEU score for the GPT-4 Conversation row is reported as 0.0000; clarify whether this is exactly zero or rounded from a small nonzero value.","section":"Table 6"},{"comment":"The phrase 'due to it's lightweight' uses 'it's' incorrectly; it should be 'its'.","section":"Section VI"},{"comment":"References [40] and [43] appear to be the same paper (Van Veen et al.); duplicate entries should be consolidated.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as an application/system description rather than a rigorous evaluation of a new method. The central claim is not supported by the presented evidence, and the methodological flaws (train/test overlap, unhandled context-length limits, and internally inconsistent abstract) would require substantial new experiments and a careful rewrite to become acceptable. I would not encourage a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The key thing to know: the paper's headline claim—that fine-tuned FLAN-T5-Large beats GPT-4 on medical summarization—is supported by only one of the three task comparisons, and that one is compromised. Section IX-B explicitly says the test dataset is derived from the original training dataset. On the plain reading, the M-Passage test items were seen during fine-tuning, so Table 7's BLEU 0.0982 vs 0.0032 could be memorization, not generalization. The stress-test note got this right, and it is a load-bearing flaw, not a minor omission.\n\nWhat the paper does well: the system design is clearly described—three specialized models, web/mobile front end, Flask API, YugabyteDB—and the authors are honest about where GPT-4 wins. Table 8 shows GPT-4 ahead on all four M-Question metrics, and Table 9 shows GPT-4 ahead on BERTScore for M-Conversation. They don't cherry-pick those numbers away. The hyperparameter details and hardware constraints are also transparent.\n\nThe soft spots, in order of severity:\n\n1. Train/test overlap. If the M-Passage test set comes from the same pool as fine-tuning, the one favorable comparison is meaningless. This needs an explicit held-out split and a duplicate check against the Stanford source.\n\n2. Context window. FLAN-T5-Large has a 512-token limit, yet M-Conversation inputs run to 3,050 words. The paper never says whether inputs were truncated, chunked, or somehow processed. Without that, even the GPT-4 comparison on that task is apples-to-oranges.\n\n3. No error bars, significance tests, or released code/data. The per-sample plots show high variance, so the averaged scores in Table 7 may not be robust. Given the small, undisclosed test size, a few outliers could flip the conclusion.\n\n4. Abstract overclaim. \"Superior performance over GPT-4\" is contradicted by the paper's own tables for two of three tasks. The abstract should be narrowed to the M-Passage result—and only after the split issue is settled.\n\nThe M-Question extraction task is also under-evaluated; it is just scored with generic summarization metrics, not checked for whether the extracted question is actually the clinically relevant one.\n\nWho this is for: readers interested in building lightweight, locally deployed medical NLP tools will find the system architecture useful. But the empirical claims should not be cited until the evaluation is redone on a clean split.\n\nMy recommendation: send it to peer review, because the practical question—can a 780M-parameter model beat GPT-4 on a narrow clinical task at a fraction of the cost—is worth a careful look. But a serious referee should demand a held-out test set, a description of input-length handling, and significance testing. Without those, the paper should not be accepted.","headline":"The M-Passage claim of beating GPT-4 likely rests on a test set drawn from the training pool; without a properly held-out split, the paper's central result is not established.","tokens_in":17841,"tokens_out":1936,"would_cite":false,"duration_ms":20783,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Medalyze claims that a 780M-parameter fine-tuned FLAN-T5-Large summarizes structured medical passages more faithfully than GPT-4, and ships that capability as a locally deployable web and mobile app.","keywords":["Medalyze","medical text summarization","FLAN-T5-Large","fine-tuning","GPT-4 comparison","patient-doctor conversation extraction","medical question identification","privacy-preserving healthcare NLP"],"falsifier":"Inspect the deployed pipeline's tokenized input lengths on the reported test samples: if any long M-Conversation input is truncated to 512 tokens before generation while GPT-4 received the full conversation, the comparison is not on equal terms. A second check is to hash the training and test inputs to detect overlap between fine-tuning and evaluation samples.","tokens_in":16896,"feed_emoji":"🩺","tokens_out":8451,"duration_ms":75585,"temperature":0.7,"pith_summary":"Medalyze is an application built around three specialized FLAN-T5-Large models for medical text: one summarizes structured reports, one extracts health issues from patient-doctor conversations, and one identifies the central question in a passage. The paper's central claim is that fine-tuning a lightweight model on domain data yields summaries that align better with human-written references than GPT-4 does, at lower computational cost and with local, privacy-preserving deployment. The strongest evidence is on structured medical passages, where the fine-tuned model beats GPT-4 on all four reported metrics (BLEU, ROUGE-L, BERTScore, and SpaCy similarity). The paper's own tables also show GPT-4 winning the question-extraction task and one of the two semantic metrics on conversations, so the across-the-board wording in the abstract overstates what the evaluation supports.","feed_headline":"780M-parameter model beats GPT-4 on medical summaries","feed_subtitle":"Fine-tuned FLAN-T5-Large beats GPT-4 on structured medical summaries and runs locally.","key_machinery":"The named central object is FLAN-T5-Large, a roughly 780-million-parameter text-to-text transformer with a 512-token context window, chosen because its sequence-to-sequence design fits summarization. Its fine-tuned variants, M-Passage, M-Conversation, and M-Question, are the actual engines of the system. The argument is carried by the fine-tuning step: exposure to medical passages teaches the model to compress technical text while retaining key terms and meaning, which the paper calls the reason for the high semantic-overlap scores. The evaluation machinery is the four-metric battery of BLEU, ROUGE-L, BERTScore, and SpaCy similarity, each measuring a different level of overlap with human-written reference summaries.","core_discovery":"The discovery, stated on the paper's terms, is that a 780-million-parameter FLAN-T5-Large model fine-tuned on medical passages can outperform GPT-4 in summarizing structured medical reports. On the M-Passage task it reaches BLEU 0.0982 versus 0.0032, ROUGE-L 0.3728 versus 0.0644, BERTScore 0.6533 versus 0.5453, and SpaCy similarity 0.8413 versus 0.7810. The paper interprets these gaps as evidence that domain specialization plus a sequence-to-sequence architecture can beat scale for this class of task. For the two auxiliary tasks the comparison is not uniformly favorable: GPT-4 scores higher on question extraction and on BERTScore for conversations, while the fine-tuned model keeps a slight edge on lexical and SpaCy measures there.","pith_inferences":["The clearest reading of the results is a division of labor: small domain-tuned models may be preferable for formal, terminology-dense texts, while larger autoregressive models remain stronger at extracting and rewording open-ended questions; a hybrid system could route each input type to the better model.","A necessary engineering step the paper leaves implicit is handling FLAN-T5-Large's 512-token limit for long conversations; testing chunk-and-summarize or hierarchical pipelines would show whether M-Conversation's scores hold on full dialogues.","Because all four metrics reward overlap with reference text rather than clinical factuality, an expert review of generated summaries for factual errors and hallucinated findings would be a stronger test than any of the reported scores.","Fine-tuning on each institution's own report templates is a plausible cheap improvement: domain-tuned summarization should track the format and terminology of the texts it will actually receive."],"forward_implications":["If the M-Passage result is correct, structured medical reports can be summarized by a 780M-parameter local model with closer alignment to human references than a frontier general-purpose model.","Because Medalyze runs locally and stores summaries in its own database, patient text does not have to be sent to an external cloud service, which supports privacy-preserving use in healthcare.","The same fine-tuning recipe produces working models for extracting health issues from dialogues and identifying the main question in a passage, with task-specific trade-offs the paper reports.","The deployment architecture, with API endpoints, web and mobile front ends, and a single-node database, makes real-time summarization available to non-specialist users.","Medalyze is explicitly a comprehension aid rather than a diagnostic tool, so its summaries are positioned as support for clinical decisions, not as medical advice."],"supporting_citations":[{"why":"Supplies the open dataset and human-written target summaries used for fine-tuning and evaluation; also documents a large seq2seq model being competitive at low parameter count, motivating FLAN-T5-Large.","marker":"[38]"},{"why":"Background evidence that adapted LLMs can outperform medical experts in clinical text summarization, setting up the comparison standard for a small fine-tuned model against GPT-4.","marker":"[40]"},{"why":"Earlier application of FLAN-T5 variants to clinical text tasks, cited to justify the architecture selection.","marker":"[36]"},{"why":"Reports open-source models approaching commercial LLMs in medical evidence summarization, contextualizing the GPT-4 comparison.","marker":"[46]"}],"fun_headline_variants":["Medalyze: 780M-parameter model beats GPT-4 on medical summaries","Fine-tuned FLAN-T5-Large outperforms GPT-4 on structured reports","Lightweight medical summarizer edges out GPT-4 in domain task","Medalyze app: small model wins on M-Passage, not all tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes that the fine-tuned models saw the same complete input text GPT-4 saw and that the test samples were not part of fine-tuning, since the paper does not describe how FLAN-T5-Large's 512-token input limit was handled for conversations up to 3,050 words.","fun_headline_variants_meta":{"raw":{"variants":["Medalyze: 780M-parameter model beats GPT-4 on medical summaries","Fine-tuned FLAN-T5-Large outperforms GPT-4 on structured reports","Lightweight medical summarizer edges out GPT-4 in domain task","Medalyze app: small model wins on M-Passage, not all tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1490,"prompt_tokens":869,"completion_tokens":621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":534}},"tokens_in":485,"tokens_out":621,"duration_ms":6303,"temperature":1.0,"reasoning_tokens":534,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:46:25.795838+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the deployed pipeline's tokenized input lengths on the reported test samples: if any long M-Conversation input is truncated to 512 tokens before generation while GPT-4 received the full conversation, the comparison is not on equal terms. A second check is to hash the training and test inputs to detect overlap between fine-tuning and evaluation samples.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the open dataset and human-written target summaries used for fine-tuning and evaluation; also documents a large seq2seq model being competitive at low parameter count, motivating FLAN-T5-Large."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Background evidence that adapted LLMs can outperform medical experts in clinical text summarization, setting up the comparison standard for a small fine-tuned model against GPT-4."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier application of FLAN-T5 variants to clinical text tasks, cited to justify the architecture selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports open-source models approaching commercial LLMs in medical evidence summarization, contextualizing the GPT-4 comparison."}],"review_version":1}