{"id":"464fa184-6708-4970-b376-ecbe24e91109","arxiv_id":"2506.19512","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Query-dependent-k ranked list truncation performs comparably to, but not consistently better than, fixed-k retrieval in an attributed clinical RAG pipeline, with the best system still below the organizer baseline.","lead":"An RAG pipeline for answering patient questions from electronic health records compares fixed-k retrieval with query-dependent-k truncation strategies, including two new heuristics. The reported gains over fixed-k are small, about 0.01 to 0.02 in overall score on a 20-case development set, and the best pipeline does not beat the shared task organizer's baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that query-dependent-k retrieval 'shows benefits' over fixed-k is not supported by the paper's own Table 2: surprise and elbow tie or nearly tie fixed-k=10 on strict F1 and overall score, and the conclusion itself concedes 'comparable performance.'","rationale":"The reader's weakest-assumption analysis focused on the 20-case dev set being used both for configuration selection and final scoring, which is a real and important limitation that the paper itself flags in Section 2.1's footnote. My concern is more direct: even within that dev set, the reported numbers do not support the abstract's claim of benefits. Table 2 shows surprise and elbow performing essentially identically to fixed-k=10, with the largest gap being 0.01 in overall score. Table 1 shows fixed-k=10 outperforming surprise in retrieval F1, and matching elbow. The paper's conclusion explicitly uses the word 'comparable' rather than 'better,' so the abstract overstates the finding. This is not a disagreement with the shared-task community or a demand for a different method; it is an internal inconsistency between the central claim and the evidence in the same paper. The existence of code and the honest limitations section are real strengths, which is why this is a revision issue rather than a rejection. The reader's conditional verdict already requires removing the overclaim and making the dev-set limitation central; my analysis reinforces that same requirement without moving the verdict. I mark agreement as partial because the reader's stated weakest assumption emphasizes representativeness of the dev set, whereas the more load-bearing problem is that the benefit claim fails on the dev set itself.","tokens_in":12972,"tokens_out":4200,"duration_ms":50659,"concrete_test":"Use the official ArchEHR-QA test set (100 case studies) and the released code to run three configurations under identical one-shot prompting and 200-token limits: (a) the submitted surprise pipeline, (b) fixed-k=10, and (c) fixed-k=15. Repeat each run 3 times to account for LLM non-determinism. Compute strict F1, relevance, and overall score with the official script, then perform a paired bootstrap (e.g., 10,000 resamples over case studies) on the per-case overall score difference between surprise and the best fixed-k. If the 95% confidence interval includes zero or the fixed-k mean is higher, the abstract's 'benefits' claim is unsupported and should be replaced with the 'comparable performance' wording already used in the conclusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's claim—'The experimental results show the benefits of our strategy in producing factual and relevant answers when compared to a fixed-k'—requires that query-dependent-k improve the full pipeline over fixed-k. The reported data do not show this. In Table 2, under the same query formulation, one-shot prompting, and 200-token limit, surprise achieves strict F1 0.37 and overall 0.36; elbow achieves 0.37 and 0.35; fixed-k=10 achieves 0.37 and 0.35. Fixed-k=15 achieves 0.35 and 0.35. Thus the best query-dependent-k configuration is indistinguishable from fixed-k=10 on F1 and ahead by only 0.01 on overall score. The retrieval-level Table 1 tells the same story: fixed-k=10 has strict F1 0.50 and lenient F1 0.58, while surprise has 0.42 and 0.49, and elbow has 0.50 and 0.55. The paper's own conclusion in Section 4 states that 'query-dependent-k strategies like surprise, elbow, and autocut* methods for different types of queries in the dataset showed comparable performance to fixed-k approaches.' This directly contradicts the abstract's 'benefits' claim. Moreover, all experiments in Section 3 use only the 20-case development set, as the manuscript itself notes in the footnote to Section 2.1, and the same dev set was used to select the pipeline configuration (query, prompt, token limit, attribution weights, and retrieval strategy). No error bars, significance tests, or held-out test set results are reported. Given the acknowledged non-determinism of the LLM even at temperature 0.001, differences of 0.01 in overall score are within replication noise. The load-bearing assertion—that query-dependent-k is beneficial—therefore fails even on the paper's own evidence, independent of the additional generalizability concern about the dev set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents heiDS, the authors' system for the ArchEHR-QA 2025 shared task. The pipeline embeds clinical note sentences with bge-large-en-v1.5, retrieves the top-k most similar sentences for a query formed from both the patient and clinician questions, and then prompts an LLM (LLaMA-3.3-70B or Mixtral-8x7B) to generate answers with inline attributions. The central methodological claim is that replacing a fixed retrieval cut-off k with a query-dependent-k strategy (the existing surprise and autocut methods, plus two newly proposed methods, autocut* and elbow) improves the factualness and relevance of generated answers compared to fixed-k retrieval. The paper reports retrieval-level and pipeline-level experiments on the 20-case development set, including ablations of query formulation, prompting style, token limit, and attribution parameters.","tokens_in":13319,"tokens_out":3628,"duration_ms":40208,"significance":"If the claimed benefit of query-dependent-k were supported, the paper would offer a practical contribution: avoiding manual tuning of k in clinical RAG pipelines while maintaining or improving answer quality. The paper has concrete strengths: the code is publicly available, the official ArchEHR-QA evaluation script is used, the ablations in Appendices D-F and G systematically examine query formulation, prompting, token limits, and attribution weights, and the authors are candid about several limitations, including LLM non-determinism and the lack of held-out evaluation in the paper. However, the reported numbers do not demonstrate the central benefit claim. On the 20-case dev set, surprise and elbow match or nearly match fixed-k=10 on strict F1 (0.37 vs. 0.37) and differ by only 0.01 on overall score (0.36 vs. 0.35). The paper's own Section 4 states that the query-dependent-k strategies 'showed comparable performance to fixed-kapproaches,' which directly undercuts the abstract's 'benefits' claim. The contribution is therefore best characterized as a careful system description with a negative or neutral result, and the claims need to be realigned with the evidence.","major_comments":[{"comment":"The abstract claims that 'The experimental results show the benefits of our strategy in producing factual and relevant answers when compared to a fixed-k.' This is not supported by Table 2: surprise achieves strict F1=0.37 and overall=0.36, elbow achieves 0.37 and 0.35, fixed-k=10 achieves 0.37 and 0.35, and fixed-k=15 achieves 0.35 and 0.35. The largest difference is 0.01 on the overall score, and Section 4 itself concludes that query-dependent-k strategies 'showed comparable performance to fixed-kapproaches.' The abstract and conclusion should be revised to state comparable performance with the practical benefit of avoiding manual k tuning, rather than claiming empirical superiority.","section":"Abstract and Section 4"},{"comment":"All experiments described in Section 3 use only the 20-case development set, and the same development set was used to select the query formulation, prompting style, token limit, retrieval strategies, and attribution parameters. With n=20, differences of 0.01-0.02 in overall score are well within the expected sampling noise, and no error bars, confidence intervals, significance tests, or results on the held-out test set are reported. This selection-on-evaluation-set loop makes the reported advantage of surprise over fixed-k=10 (0.36 vs. 0.35 overall) not interpretable as evidence for the central claim. The authors should either report bootstrap confidence intervals, perform a significance test, add test-set results, or explicitly weaken the claim to 'comparable performance' without implying statistical superiority.","section":"Section 3, footnote 2 and Tables 1-2"},{"comment":"The retrieval-level results do not support the benefit claim either. Fixed-k=10 has strict F1=0.50 and lenient F1=0.58; surprise has strict F1=0.42 and lenient F1=0.49; elbow has strict F1=0.50 and lenient F1=0.55; autocut and autocut* are substantially worse. Thus the query-dependent-k strategies do not outperform fixed-k=10 on retrieval F1, and the choice of surprise as the submitted pipeline appears to be based on criteria other than the retrieval metrics reported in Table 1. The paper should discuss this discrepancy explicitly, since it weakens the motivation for preferring query-dependent-k over fixed-k in this dataset.","section":"Table 1"},{"comment":"The two newly proposed methods, autocut* and elbow, are described only informally: autocut* 'inspects how much each similarity score decreases compared to the previous score' and elbow 'locates the elbow where the transition from high to low relevance occurs.' No equations, pseudocode, or precise decision rules are provided for the 'significant changes' threshold or the elbow-location criterion. Since these methods are introduced as contributions and their results are reported in Tables 1 and 2, the lack of algorithmic detail prevents reproducibility and makes it impossible to determine whether the reported behavior is tied to the actual method or to implementation choices. Please provide precise definitions, including any hyperparameters or stopping criteria.","section":"Section 2.4"}],"minor_comments":[{"comment":"The title contains a spacing typo: 'From Fixed-kto Query-dependent-kfor Retrieval Augmented Generation' should read 'From Fixed-k to Query-dependent-k for Retrieval Augmented Generation.'","section":"Title"},{"comment":"In the row for autocut* with LLaMA, the model name is written as 'LLaMA-3.37B'; this should be 'LLaMA-3.3-70B' to match the rest of the table.","section":"Table 2"},{"comment":"The text states that 'Table 5 summarizes the top 10 configurations by strict F1-score,' but Table 5 lists only five configurations. Please either include all ten rows or correct the description.","section":"Appendix G"},{"comment":"The bullet list in Section 2.2 says the decisions are 'supported by experiments' and then lists choices, but the actual experiment results for query formulation, prompting, and token limit appear only in Appendices D, E, and F. Please add cross-references at the bullet items to make the support explicit.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"This is a shared-task system description rather than a methodological advance, and the main empirical claim is not supported by the paper's own tables. The paper could be acceptable after major revision if the abstract and conclusion are brought in line with the 'comparable performance' finding, if the selection-on-dev-set issue is addressed with appropriate caveats or additional evaluation, and if the two new retrieval strategies are specified precisely enough to be reproduced. The public code and systematic ablations are genuine strengths and should be credited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the heiDS ArchEHR-QA paper. The short version: this is a solid system description that overclaims in the abstract. Query-dependent-k is not shown to be better than fixed-k; its own Table 2 and the conclusion say comparable, and the differences are within noise on a 20-case dev set.\n\nWhat's actually new: two named heuristics, autocut* and elbow. Both are straightforward: autocut* truncates where consecutive similarity scores drop significantly; elbow uses the classic clustering elbow on the score curve. They are not deep, but they are well-motivated and easy to reproduce. The paper also does honest ablations on query wording, one-shot vs zero-shot prompting, token limits, and attribution weights, and the limitations section flags non-determinism and dev-set-only evaluation. Code is released. That is credible work for a shared-task report.\n\nThe main problem is the gap between the abstract—'show the benefits of our strategy'—and the reported numbers. In Table 2, surprise achieves strict F1 0.37/overall 0.36, elbow 0.37/0.35, and fixed-k=10 0.37/0.35. On retrieval (Table 1), fixed-k=10 has strict F1 0.50, elbow 0.50, surprise 0.42. So the best query-dependent-k is statistically indistinguishable from fixed-k=10. There are no error bars or significance tests; 20 cases is small; and the same dev set was used to tune query, prompt, token limit, attribution weights, and the retrieval strategy itself. That selection loop alone could explain a 0.01 gain. The conclusion of the paper says 'comparable performance,' which is the accurate claim. The abstract should say that, with the practical caveat that query-dependent-k saves manual k tuning. That is still a useful contribution, but it is a convenience argument, not an accuracy win.\n\nWho gets value: people building RAG for clinical notes, and anyone interested in RLT heuristics beyond surprise and autocut. It fits a workshop or system-description venue. My recommendation: send to peer review, but require the claim to be fixed. With that change, the paper is fine. Without it, the central assertion is misleading.","headline":"A competent shared-task writeup with two simple truncation heuristics, but the abstract's 'benefits' claim is contradicted by the paper's own tables and conclusion.","tokens_in":13933,"tokens_out":3393,"would_cite":false,"duration_ms":36295,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Choosing the retrieval cut-off per question, rather than fixing a top-k, keeps clinical RAG answers factual and relevant while removing manual tuning.","keywords":["retrieval augmented generation","query-dependent-k","ranked list truncation","clinical question answering","evidence attribution","electronic health records","extreme value theory","ArchEHR-QA"],"falsifier":"Run the official ArchEHR-QA evaluation on the 100-case test set for each retrieval strategy with the same one-shot prompt and LLaMA-3.3-70B model. If the best fixed-k strategy (k = 10, 15, or 20) beats every query-dependent-k strategy on strict F1 and overall relevance by a margin larger than the development-set spread, the paper's comparable-performance claim collapses.","tokens_in":12730,"feed_emoji":"🩺","tokens_out":7293,"duration_ms":73657,"temperature":0.7,"pith_summary":"The paper tries to establish that a retrieval-augmented question-answering pipeline for electronic health records does not need a globally fixed number k of evidence sentences: letting the retrieval score distribution decide where to cut the ranked list per query can produce answers that are as factual and relevant as any fixed top-k. The authors enter the ArchEHR-QA 2025 shared task with a system that retrieves clinically similar sentences, truncates the list with a query-dependent-k strategy, and has an LLM generate answers citing the retrieved sentences. They contribute two parameter-free cut-off heuristics, autocut* and elbow, alongside the existing surprise and autocut methods. On the 20-case development set, the best configuration—surprise truncation with LLaMA-3.3-70B and post-retrieval attribution—reaches strict precision 0.62, recall 0.26, and F1 0.37, with query-dependent-k methods showing comparable performance to fixed-k while avoiding manual k tuning. The practical motivation is that real clinical notes are too long for full-context prompting, and per-query cut-offs adapt to variable note lengths.","feed_headline":"Per-query retrieval cut-offs match fixed-k in clinical RAG","feed_subtitle":"Two parameter-free truncation methods, elbow and autocut*, remove manual k tuning while keeping factual, attributed answers.","key_machinery":"The machinery that carries the argument is the query-dependent-k ranked list truncation (RLT) strategy: a mechanism that inspects the ranked similarity scores between the query and candidate clinical sentences and decides where to cut. Surprise uses extreme value theory, fitting a generalized Pareto distribution to the score tail and truncating at a threshold from Pickands (1975) and Bahri et al. (2023). Autocut cuts at the first discontinuity in the score decline. The paper's new variants, autocut*, which cuts where the drop from one score to the next is significant relative to previous drops, and elbow, which finds the elbow in the score curve as in clustering, make the selection fully automatic. These strategies are paired with a cosine-similarity semantic index over sentence embeddings and either post-retrieval attribution, where the LLM cites sentence ids inline, or post-generation attribution, where a weighted lexical, fuzzy, and semantic similarity mapping with a threshold connects answer sentences back to evidence.","core_discovery":"The central discovery, on the paper's own terms, is that per-query ranked list truncation works: replacing a fixed top-k with a truncation point derived from the shape of similarity scores yields comparable retrieval and final-answer quality without per-corpus tuning. In the submitted pipeline, the surprise method fits a generalized Pareto distribution to the retrieval scores and sets a threshold, so each patient question receives its own k. The two new heuristics, autocut* and elbow, do the same without any parameters. The paper reports that LLaMA-3.3-70B with post-retrieval attribution and surprise truncation scores strict F1 0.37 and overall relevance 0.35, equal to the best fixed-k pipelines on the development set, while post-generation attribution lags behind with best strict F1 0.27. It also acknowledges that the best pipeline does not beat the organizer baseline, which feeds all note sentences to the model, and explains the gap by prompt sensitivity and the recall cost of retrieval.","pith_inferences":["A natural next test is whether the elbow and autocut* cut-offs transfer to other retrieval corpora, such as legal or scientific documents, because they look only at the geometry of similarity scores; the paper demonstrates them only on 20 development cases.","The reported gap to the organizer baseline is attributed to prompt and context-size differences; an untested corollary is that combining the one-shot prompt with a larger per-query context budget could recover the recall loss and close that gap.","The new heuristics could serve as a cheap first-stage filter in front of a re-ranker, truncating the candidate list by elbow before re-ranking the survivors, which would avoid the cost of re-ranking all candidates.","Because the same 20-case development set was used both to select the best configuration and to compute the final scores, the comparable-performance result is not decisive on its own; stable conclusions require the 100-case test set."],"forward_implications":["A per-query retrieval cut-off removes the need to hand-tune k, so the same pipeline can be applied to case studies with widely different numbers of clinical sentences, from a few up to 54 in the development set and 74 in the test set.","The best operating point is high precision at the cost of recall: strict precision reaches 0.62 while recall is only 0.26, meaning false attributions are rare but relevant evidence is often missed.","The two new parameter-free heuristics, autocut* and elbow, are drop-in replacements for fixed-k at the retrieval stage, with elbow reaching retrieval F1 0.50 under both strict and lenient variants in the paper's tables.","Post-retrieval attribution outperforms post-generation attribution in this setup, with best strict F1 0.37 versus 0.27, suggesting that asking the LLM to cite retrieved sentences inline is more effective than attributing after generation.","In real-world settings where full clinical notes cannot fit into the LLM context, a retrieval step is unavoidable, and query-dependent-k methods make that step self-tuning."],"supporting_citations":[{"why":"Supplies the surprise ranked-list truncation method used in the submitted pipeline.","marker":"Bahri et al. 2023"},{"why":"Provides the generalized Pareto extreme-value theory on which the surprise truncation threshold is built.","marker":"Pickands 1975"},{"why":"Introduces ranked list truncation for LLM-based reranking, the context in which surprise-based cut-offs are applied here.","marker":"Meng et al. 2024"},{"why":"Defines the ArchEHR-QA shared task, its development and test splits, and the strict and lenient evaluation criteria.","marker":"Soni and Demner-Fushman 2025b"},{"why":"Provides the PhysioNet dataset of case studies, clinical note excerpts, and annotations used in all experiments.","marker":"Soni and Demner-Fushman 2025a"},{"why":"Supplies the FAISS index used for semantic search over sentence embeddings.","marker":"Johnson et al. 2019"},{"why":"Supports the paper's explanation of the gap to the organizer baseline through prompt sensitivity.","marker":"Salinas and Morstatter 2024"},{"why":"Motivates the retrieval step by showing that LLMs can be distracted by irrelevant context.","marker":"Shi et al. 2023"}],"fun_headline_variants":["Per-question retrieval cut-offs match fixed-k in clinical RAG","Parameter-free truncation: autocut* and elbow for EHR QA","Query-dependent k: no tuning needed, same quality as fixed-k","Adaptive retrieval truncation matches fixed-k in clinical RAG","Two new methods find per-question k automatically for RAG"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's conclusions rest on the assumption that 20 development case studies, used both to pick the best configuration and to produce the final numbers, are representative enough to show that per-query retrieval cut-offs work as well as fixed ones.","fun_headline_variants_meta":{"raw":{"variants":["Per-question retrieval cut-offs match fixed-k in clinical RAG","Parameter-free truncation: autocut* and elbow for EHR QA","Query-dependent k: no tuning needed, same quality as fixed-k","Adaptive retrieval truncation matches fixed-k in clinical RAG","Two new methods find per-question k automatically for RAG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000285,"raw_usage":{"total_tokens":1654,"prompt_tokens":894,"completion_tokens":760,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":671}},"tokens_in":510,"tokens_out":760,"duration_ms":8003,"temperature":1.0,"reasoning_tokens":671,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:06:18.485439+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the official ArchEHR-QA evaluation on the 100-case test set for each retrieval strategy with the same one-shot prompt and LLaMA-3.3-70B model. If the best fixed-k strategy (k = 10, 15, or 20) beats every query-dependent-k strategy on strict F1 and overall relevance by a margin larger than the development-set spread, the paper's comparable-performance claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the FAISS index used for semantic search over sentence embeddings."}],"review_version":1}