{"id":"5f9f72b4-8975-45d1-a089-46cb20dc986f","arxiv_id":"2505.20320","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RAG-based retrieval of 4,000 words matched whole-note LLM classification on post-operative complication AUROC at a fraction of the token cost, but equivalence is claimed from non-significant p-values rather than a proper non-inferiority test.","lead":"The paper tests whether feeding an LLM only the top 4,000 words retrieved from a clinical note can match the performance of feeding the entire note, for detecting post-operative complications. Across three models, AUROC was statistically indistinguishable, while token cost dropped by more than 90%, though one model lost substantial recall.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central 'no sacrifice in accuracy' claim is undercut by Table I: GPT4o recall drops 0.90→0.71 and LLaMA 0.86→0.73; only AUROC was significance-tested (DeLong), so equivalence in recall is asserted, not shown.","rationale":"Good-faith reading: the paper's contribution is an empirical comparison of two ingestion strategies. The token-cost saving is quantified (Section IV-A: roughly 172M vs 13.2M tokens, $430 vs $33) and the code is public, which is real evidence. The central claim, however, is not just cost savings but 'same performance' / 'without sacrificing classification accuracy.' The reported numbers themselves put that in doubt: recall falls by 19 and 13 percentage points for GPT4o and LLaMA. Since the task is classification of surgical complications, recall is arguably the most important metric; missing complications is a safety failure, not just a statistical nuisance. The statistical evidence only covers AUROC via DeLong; the paper's blanket 'p>0.05' for all metrics is not backed by tests. The reader's verdict already flags this in its rationale, but its named weakest assumption is negative-cohort label noise. That label-noise concern is plausible and worthwhile, but the recall drop is more direct and load-bearing: even with perfect labels, the observed recall gap undermines the equivalence claim. A conditional verdict remains appropriate if the authors add the missing tests and soften the claim; if the significance tests show recall degradation, the conclusion should move to a cost-recall trade-off rather than equivalence. Therefore my read leaves the reader's CONDITIONAL verdict unchanged, with agreement partial: same broad concern, different sharpest point.","tokens_in":7260,"tokens_out":3787,"duration_ms":33422,"concrete_test":"Compute paired significance tests for recall (and precision/F1) on the same patients, e.g., McNemar's test for binary complication detection comparing A2-RAG vs A1-Long Context separately for GPT4o and LLaMA, plus 95% bootstrap confidence intervals for the recall difference. Pre-specify a non-inferiority margin (e.g., recall drop no more than 0.05) and test whether the one-sided 95% CI excludes a drop larger than that margin. If GPT4o's 0.19 recall deficit is statistically significant or exceeds the margin, the equivalence claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that RAG matches whole-text performance while cutting tokens, with 'no statistically significant differences' across AUROC, precision, recall, and F1 (Abstract; Section V-A). The only significance test reported is DeLong on AUROC (Section IV-B; Table II). For recall, Table I shows GPT4o 0.90→0.71 and LLaMA 0.86→0.73; these are large absolute drops in a surgical-complication detection task, where recall is the safety-critical metric. Non-significance of AUROC differences does not imply equivalence of recall, and no p-values or confidence intervals are given for precision, recall, or F1, so the abstract's blanket 'p>0.05' claim is unsupported. Unless the recall deficit is shown to lie within a pre-specified non-inferiority margin, the headline conclusion 'without sacrificing classification accuracy' (and 'no observed drop in ... recall' in Section VI) is contradicted by the reported point estimates. The cost reduction itself is credible, but the performance-equivalence claim is the load-bearing part and it is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two strategies for classifying post-operative complications from clinical notes: whole-text ingestion into large-context LLMs (GPT4o, LLaMA 8B, Mistral 12B) and a RAG pipeline that chunks notes, embeds them in FAISS, and retrieves a top-4,000-word (also described as 4,000-token) context. On a cohort of 2,294 patients (996 M&M-confirmed complications and 1,298 patients with no M&M registry entry), the authors report AUROC, precision, recall, F1, and PR-AUC for both strategies, along with a DeLong test for AUROC differences. They claim no statistically significant performance differences and a cost reduction of over 90%, concluding that RAG can match whole-text accuracy while cutting resource use. The paper includes a code repository and cost/time projections.","tokens_in":7462,"tokens_out":2676,"duration_ms":24554,"significance":"If the equivalence claim were established, the result would be practically valuable: hospitals could reduce API and compute costs by roughly an order of magnitude without measurable loss in discriminative performance. The study has notable strengths: it uses real clinical data across multiple sites within one health system, evaluates three different LLMs, reports a concrete cost analysis, and makes code available. However, the central claim is an equivalence claim, and the statistical evidence provided is incomplete: non-significant DeLong p-values for AUROC do not establish equivalence, and no tests or confidence intervals are reported for precision, recall, or F1. The observed recall drops for GPT4o (0.90 to 0.71) and LLaMA (0.86 to 0.73) are large in absolute terms and directly contradict the conclusion that there was 'no observed drop in accuracy or recall.' These issues are load-bearing because the paper's headline is that RAG preserves accuracy, not merely that AUROC is not significantly different.","major_comments":[{"comment":"The central claim that RAG matches whole-text performance on all reported metrics is not supported by the presented statistical analysis. The only significance test is the DeLong test applied to AUROC (Section IV-B, Table II). No p-values, confidence intervals, or non-inferiority tests are reported for precision, recall, or F1, yet the Abstract and Section VI assert that these metrics showed no statistically significant differences and that there was 'no observed drop in accuracy or recall.' The point estimates in Table I directly contradict that assertion for recall: GPT4o drops from 0.90 to 0.71 and LLaMA from 0.86 to 0.73 under RAG. Non-significance of an AUROC difference does not imply equivalence of recall, and the paper should either (a) pre-specify an equivalence/non-inferiority margin, report confidence intervals for all metrics, and conduct appropriate tests (e.g., bootstrap or DeLong-style tests for recall differences), or (b) substantially soften the conclusion to claim only comparable AUROC with a clear statement of the observed recall trade-off. As written, the headline conclusion is not established by the evidence provided.","section":null},{"comment":"","section":"Section III-A"},{"comment":"","section":"Section IV-A and Section III-B"}],"minor_comments":[{"comment":"","section":"Abstract"},{"comment":"","section":"Section III-B"},{"comment":"","section":"Section III-C"},{"comment":"","section":"Section IV-A"},{"comment":"","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful empirical comparison and a credible cost-reduction story, but the central equivalence claim is not supported by the statistical methods used. The recall drops in Table I are too large to wave away with 'no statistically significant differences' without proper non-inferiority testing or confidence intervals. The negative-cohort definition is another concern that the authors should address. I believe these issues are fixable within the scope of the manuscript, but they require real analytical work, not just editing. If the authors can add pre-specified equivalence margins, confidence intervals, and a discussion of label quality, the paper could become a solid contribution. If not, the claims should be substantially narrowed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this before reading: the paper's core empirical finding is a head-to-head comparison of RAG against whole-text ingestion for post-operative complication classification, across three LLMs (GPT4o, LLaMA, Mistral), with code and a real cost saving of over 90%. That part is solid and worth taking seriously. The abstract's blanket claim that accuracy is not sacrificed, however, is not supported by the numbers in Table I.\n\nWhat's genuinely new: prior work applies RAG to clinical tasks, but not this direct equivalence comparison on a surgical complication detection task with open and proprietary models. The DeLong tests on AUROC show no significant difference, and the cost projection is credible. The authors also share their pipeline on GitHub, which is a plus.\n\nThe soft spots are real and central. Only AUROC gets a significance test. For recall, GPT4o drops from 0.90 to 0.71 and LLaMA from 0.86 to 0.73 under RAG. In a safety-critical complication detection task, that is not a trivial decline, and the paper offers no non-inferiority margin or confidence intervals to show these drops are acceptable. The abstract's 'p > 0.05 for all metrics' is unsupported because precision, recall, and F1 were never tested. Also, the negative cohort is defined as 'no M&M registry entry,' which depends on residents flagging and adjudicating complications; if unrecorded complications correlate with note content, that could bias the RAG/whole-text comparison. The retrieval constants (4,000 words, 512-word chunks) are hand-chosen with no sensitivity analysis, so we don't know how robust the equivalence claim is to those choices.\n\nThese are fixable problems, not fatal ones. The cost-efficiency result is likely to hold up, and the AUROC non-inferiority might too. But the paper currently overstates what it shows. A serious referee should ask for: non-inferiority tests or CIs on all metrics, per-metric significance tests, a sensitivity analysis on retrieval parameters, and a clearer discussion of the label-noise risk in the negative cohort.\n\nWho is this for? People building clinical NLP pipelines in cost-constrained settings, and researchers working on retrieval-based classification. It deserves peer review, but as a conditional—the equivalence claim needs to be either proven with proper bounds or softened appropriately.","headline":"Useful head-to-head RAG vs whole-text comparison with credible cost savings, but the 'no sacrifice' conclusion overreaches: recall drops are real and equivalence is asserted, not tested.","tokens_in":8057,"tokens_out":1311,"would_cite":false,"duration_ms":11747,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieval-augmented generation matches whole-text LLM classification of surgical complications while cutting token use by more than 90%.","keywords":["retrieval-augmented generation","large language models","clinical NLP","surgical complication classification","FAISS","token efficiency","long-context LLMs","text classification"],"falsifier":"Run both pipelines on a cohort where every complication status is confirmed by blinded manual chart review rather than registry absence, and check whether RAG's parity with whole-text ingestion survives; if AUROC or F1 diverges on gold-standard labels, the paper's equivalence claim is an artifact of registry noise.","tokens_in":7032,"feed_emoji":"🩺","tokens_out":6263,"duration_ms":48119,"temperature":0.7,"pith_summary":"The paper asks whether a retrieval-augmented generation (RAG) pipeline can classify post-operative complications as accurately as an LLM that reads the entire clinical note, and answers yes. Over 2,294 surgical patients, three LLMs (GPT4o, LLaMA, and Mistral) showed no statistically significant differences in AUROC, precision, recall, or F1 between RAG and whole-text ingestion, with DeLong test $p$-values all above $0.05$. The RAG arm used at most 4,000 retrieved words per patient instead of the complete 30-day note history, cutting token usage from roughly 172 million to 13.2 million tokens and API cost from about $430 to $33 for GPT4o, a reduction of more than 90%. If the finding holds, clinical NLP classifiers can be run at roughly a tenth of the inference cost without sacrificing average discriminative performance.","feed_headline":"RAG matches full-note LLMs while cutting token use 90%","feed_subtitle":"Three LLMs classify surgical complications as well from 4,000 retrieved words as from entire clinical notes.","key_machinery":"The load-bearing mechanism is a vector-retrieval pipeline: each clinical note is split into 512-word chunks, converted to dense vector embeddings, and stored in a FAISS index; a targeted classification query retrieves the top $N$ chunks most semantically similar to the question, which are concatenated in their original order and kept under a 4,000-token limit before being passed to the LLM classifier. This machinery prunes irrelevant text before the model sees it, so the LLM's attention is concentrated on high-yield segments. It is what converts a long-document problem into a short-context classification problem, and it is the component whose token savings produce the reported cost reduction.","core_discovery":"The paper's central claim is that for identifying surgical complications in clinical text, most of a note is redundant: feeding only the top semantically relevant segments to an LLM preserves discriminative performance relative to feeding the whole document. In the RAG arm, notes from the prior 30 days are split into chunks of up to 512 words, embedded, indexed in FAISS, and the top chunks are concatenated in original order under a 4,000-token budget. On the same 2,294-patient cohort, GPT4o-RAG reached AUROC 0.67 and F1 0.61 versus 0.66 and 0.61 for GPT4o-Long Context; LLaMA reached 0.63 and 0.60 versus 0.63 and 0.61; Mistral reached 0.58 and 0.60 versus 0.57 and 0.61. The RAG method also improved per-patient inference speed for the open-source models (19% for Mistral, 23% for LLaMA). The authors conclude that RAG is a scalable, model-agnostic way to lower cost and latency in clinical NLP without a measurable accuracy trade-off.","pith_inferences":["The paper does not ablate the retrieval step; an immediate testable extension is to compare semantic retrieval against random chunks of equal size, which would show whether the parity is caused by retrieval or simply by the LLM's robustness to shorter input.","Because the negative cohort is defined by absence from a registry that depends on resident flagging and manual adjudication, the comparison could be distorted if unrecorded complications cluster in long or short notes; a gold-standard chart-review cohort would test this.","The authors leave open whether the 4,000-token budget is task-dependent; for complications that require synthesizing evidence spread across many notes, the required budget may grow, and a scaling curve of performance versus token budget would make the trade-off explicit.","The equivalence of whole-text and RAG suggests that the LLM's decision is driven by local evidence rather than global narrative; if true, harder clinical questions that require cross-note reasoning may show a larger performance gap."],"forward_implications":["Hospitals can run the same complication-classification task for roughly one-tenth of the API cost of whole-note ingestion, with no statistically significant change in AUROC, precision, recall, or F1.","The pipeline transfers across model families: parity held for a proprietary large model and two open-source models of different sizes, so the savings do not depend on one vendor's model.","The open-source models ran faster in RAG mode, so the approach can reduce both dollar cost and latency at scale.","Because the same FAISS index can be queried with different prompts, one indexed corpus could support multiple classification tasks without re-embedding the notes.","The authors propose extending the framework to ICD coding, radiology report classification, medication extraction, and real-time triage, where complication signals are likely concentrated in a few text regions."],"supporting_citations":[{"why":"Defines the retrieval-augmented generation framework that the RAG arm instantiates.","marker":"[7]"},{"why":"Supplies the FAISS similarity-search library used to index chunk embeddings and retrieve top-N segments.","marker":"[12]"},{"why":"Provides the prompt-engineering approach adapted for the LLM classification query.","marker":"[16]"},{"why":"Motivates the hypothesis that long contexts can introduce noise, so shorter retrieved context may suffice.","marker":"[4]"},{"why":"Earlier clinical RAG application that the study extends to surgical complication classification.","marker":"[5]"}],"fun_headline_variants":["Less is equal: RAG keeps clinical LLM accuracy at 4k words","4k words beat full notes in clinical LLM classification","RAG: 90% token cut, same clinical NLP accuracy","Shortened context, same results: RAG for clinical tasks","Clinical LLMs: top segments match full notes, minus 90% tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The negative cohort is defined as patients with no entry in the surgical complication registry, so any complication that was never flagged and adjudicated is silently counted as a true negative, and that labeling noise could differ systematically between long and short notes.","fun_headline_variants_meta":{"raw":{"variants":["Less is equal: RAG keeps clinical LLM accuracy at 4k words","4k words beat full notes in clinical LLM classification","RAG: 90% token cut, same clinical NLP accuracy","Shortened context, same results: RAG for clinical tasks","Clinical LLMs: top segments match full notes, minus 90% tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000286,"raw_usage":{"total_tokens":1697,"prompt_tokens":973,"completion_tokens":724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":631}},"tokens_in":589,"tokens_out":724,"duration_ms":5784,"temperature":1.0,"reasoning_tokens":631,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:35:03.414490+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run both pipelines on a cohort where every complication status is confirmed by blinded manual chart review rather than registry absence, and check whether RAG's parity with whole-text ingestion survives; if AUROC or F1 diverges on gold-standard labels, the paper's equivalence claim is an artifact of registry noise.","supporting_citations":[{"cited_title":"Retrieval-augmented generation for knowledge-intensive nlp tasks,","cited_arxiv_id":null,"evidence_quote":"Defines the retrieval-augmented generation framework that the RAG arm instantiates."},{"cited_title":"Evaluating prompt engineering on GPT-3.5’s per- formance in USMLE-style medical calculations and clinical scenarios generated by GPT-4,","cited_arxiv_id":null,"evidence_quote":"Provides the prompt-engineering approach adapted for the LLM classification query."}],"review_version":1}