{"id":"72ad4233-17a0-4795-b495-f46036961958","arxiv_id":"2505.24040","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM relevance judgments align with physician trainees on only about 45 to 66 percent of sentences, and pruning contexts to physician-labeled relevant sentences improves LLM and trainee accuracy.","lead":"Researchers built a new dataset, MedPAIR, with sentence-level relevance labels from 36 physician trainees on 1,300 medical QA cases. Comparing these labels to LLM attention and self-reports shows models often miss or over-weight the same clinical details physicians consider key, and pruning to physician-relevant sentences improves QA accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal claim that physician-guided pruning improves QA accuracy lacks a random-removal control; the gains may stem from removing any hard-to-process sentences, not from specific relevance labels.","rationale":"I agree with the reader that the weakest point is the causal interpretation of the pruning results. The dataset contribution and the descriptive alignment finding (Table 2) are solid and useful; the concordance numbers are unlikely to be an artifact. The key gap is the absence of a control condition. Because the removed sentences differ systematically from retained ones (higher perplexity, shorter length), the observed gains could be driven by removing hard text rather than by the specific relevance labels. The fact that all three pruning methods improve accuracy in Table 4 weakens the claim that expert labels are uniquely valuable. The SR metric's interpretation further relies on S+ being sufficient, which the paper's own physician post-pruning accuracy (67.2% on 248 QAs) contradicts. These are addressable by adding a random-removal baseline and by softening the causal language. The reader's CONDITIONAL verdict is therefore appropriate; no change to the verdict is needed, and my concern aligns with the reader's stated weakest assumption.","tokens_in":26664,"tokens_out":10246,"duration_ms":104730,"concrete_test":"Run a random-removal baseline on the same 1,300 QAs: for each QA, randomly delete exactly the same number of sentences as the physician labels mark low/irrelevant, repeated over at least 100 random seeds, and compute the distribution of accuracy gains for each model and for a fresh physician cohort on the 248-QA subset. If the physician-label gain falls within the 95% interval of the random-removal gains, the claim that expert relevance labels specifically cause the improvement is rejected. Additionally, test the S+ sufficiency assumption by asking a fresh cohort of physician trainees to answer using only S+; if accuracy is materially below 100%, the SR-based interpretation of correct-to-incorrect flips as 'spurious reliance' must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 reports that removing physician-labeled low/irrelevant sentences improves accuracy for most models and frames this as evidence that expert relevance labels reduce distraction. However, the experiment only compares the full context S with the high-relevance subset S+, with no control condition that removes the same number of sentences at random or matched on length/perplexity. This is load-bearing because Table 1 shows the removed sentences are systematically shorter and have higher perplexity than high-relevance sentences; any pruning that removes the most difficult-to-process text might improve LLM accuracy independently of clinical relevance. The concern is sharpened by Table 4, where three different pruning criteria (physician labels, ContextCite, GPT-4o self-report) all yield accuracy gains on most datasets, so the improvement does not appear specific to the expert labels. The SR metric in Section 3.2 compounds the issue: it interprets a correct-to-incorrect flip after pruning as reliance on spurious S- information, but such a flip can also occur if S+ alone is insufficient for the model to answer correctly. Table 7 shows physician trainee accuracy on S+ is only 67.2% on the 248-QA subset, so the assumed sufficiency of S+ is not met even for humans. Without a random-removal baseline, the paper's central causal claim that physician relevance labels drive the accuracy improvement is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MedPAIR, a dataset of 1,300 medical QA pairs with sentence-level relevance labels collected from 36 physician trainees, alongside relevance scores from three open-source LLMs via ContextCite and from GPT-4o via self-reporting. The authors measure concordance between physician labels and LLM relevance labels (Table 2), report that concordance is below two-thirds for every model, and then study what happens when low/irrelevant sentences identified by physician trainees are removed before answering (Section 4.3, Figure 3, Table 7). They also define a Spurious Rate (SR) metric that counts correct-to-incorrect flips after pruning (Section 3.2). The paper's main claims are that LLMs are poorly aligned with physician relevance estimates and that filtering out physician-labeled irrelevant sentences improves accuracy for both physician trainees and LLMs.","tokens_in":26859,"tokens_out":7242,"duration_ms":67941,"significance":"If the concordance measurement is taken at face value, MedPAIR is a valuable community resource: it provides a new kind of annotation—sentence-level physician relevance judgments for multiple medical QA benchmarks—and it makes both the annotations and the evaluation code publicly available. The concordance table is a simple, honest descriptive result, and the paper does not fit parameters to produce its main quantities. The causal pruning claim, however, is not currently supported by the experiments, because the improvement is measured only against the full context and not against a random-removal control. The dataset contribution is significant enough to warrant publication after the analysis is corrected or reframed.","major_comments":[{"comment":"The central causal claim that pruning low/irrelevant sentences improves accuracy is not supported without a control. The experiment compares the full context S with the physician-selected subset S+ only; there is no condition that removes the same number of sentences at random or matched on length/perplexity. This matters because Table 1 shows the removed sentences are systematically shorter and have higher perplexity than high-relevance sentences, and Table 4 shows that three different pruning criteria (physician labels, ContextCite, GPT-4o self-report) all produce gains on most datasets, with particularly large gains on MedXpertQA. The gains may therefore be a context-length or text-difficulty effect rather than a consequence of physician relevance labels. Please add a random-removal baseline (with multiple seeds) and a length/perplexity-matched removal baseline, and report per-dataset deltas with error bars.","section":"§4.3 / Figure 3 / Table 7"},{"comment":"The Spurious Rate definition and the interpretation of correct-to-incorrect flips assume that S+ alone is sufficient for a human to answer correctly. That assumption is not met: Table 7 reports physician trainee accuracy of only 67.2% on the S+ version of the 248-QA subset, so a model can flip from correct to incorrect after pruning simply because S+ is insufficient. Please report human accuracy on S+ for the full 1,300-QA set, or on the same QA subset used for LLM evaluation, and separate cases where S+ is sufficient from cases where it is not; otherwise SR conflates spurious reliance with information insufficiency.","section":"§3.2 / §4.3 / Table 7"},{"comment":"There is a direct inconsistency in the description of the annotation labels. Section 3.1.1 states that each QA is annotated by at least three physician trainees, while Appendix B states that after excluding annotations from labelers who answered incorrectly, each item received between one and three valid annotations. The final 'majority-vote' labels are therefore sometimes single-annotator or two-annotator decisions, not true majority votes. Because all downstream comparisons use these labels as ground truth, please report the distribution of valid labels per QA, the inter-annotator agreement (e.g., Fleiss kappa on the trinary labels), and the number of QAs at each valid-label count.","section":"§3.1.1 / Appendix B"},{"comment":"The ContextCite concordance measurement in Table 2 depends on a matching procedure that is not fully specified. For each QA, k is set to the number of physician 'high' labels, and the k highest ContextCite sentences are labeled 'high'; the remaining sentences are then assigned to 'low' or 'irrelevant' based on score order, but the cutoff between low and irrelevant is not defined. This procedure forces the model to have exactly k high labels and can inflate or deflate agreement relative to the model's natural threshold. Please specify the low/irrelevant boundary, report sensitivity to the choice of k and to that boundary, and consider a rank-based metric that does not depend on the human count.","section":"§3.3 / Table 2"},{"comment":"The dataset is constructed only from QAs on which physician trainees answered correctly (2,918 of 6,224 labels; 1,300 final QAs). As a result, the relevance labels and all accuracy comparisons are conditioned on physician correctness. The paper's claims about misalignment and pruning should be explicitly scoped to QAs where physician trainees are correct, and the discussion should address whether the conclusions plausibly extend to cases in which physicians themselves err.","section":"§4.1 / Figure 1"}],"minor_comments":[{"comment":"Table 4 reports only percentage gains without standard deviations, confidence intervals, or significance markers; please add uncertainty estimates.","section":"Table 4"},{"comment":"The text says 'in round 2, physician trainee only annotated 248 QAs,' but Table 7 appears to report LLM results on the full 1,300 QAs; please clarify exactly which subset each model group was evaluated on in Round 2.","section":"Figure 3 / §4.3"},{"comment":"Please report the decoding parameters (temperature, top-p, number of samples) used for GPT-4o self-reported labels and for the ContextCite generation runs, because Section 2.2 argues that LLM labels are sensitive to stochastic decoding.","section":"§3.1.2"},{"comment":"Section 3.1 says 2,000 QA pairs were sampled, but the final dataset is 1,300; please state the sampling fractions and explain the 104-QA exclusion earlier in the main text rather than only in Section 4.1.","section":"§3.1 / §4.1"},{"comment":"Minor typos: 'MedPair' in the Table 1 caption should be 'MedPAIR', and 'an 58.6%' in Appendix E.2 should be '58.6%'.","section":"Table 1 / Appendix E.2"}],"recommendation":"major_revision","confidential_remarks":"The dataset itself is a useful contribution and the concordance measurements are straightforward, but the pruning claim needs a proper control condition and the S+ sufficiency assumption needs to be addressed. These are fixable with additional experiments and a more careful framing; I would not reject the manuscript. The authors should be encouraged to present the paper primarily as a dataset and measurement contribution, with the pruning result as an exploratory analysis rather than a causal claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the MedPAIR dataset is the real contribution here. A first large-scale sentence-level physician relevance benchmark for medical QA is worth having, and the concordance numbers showing LLM relevance estimates diverge from trainee labels on a third to half of sentences are a solid, reproducible measurement. The pruning-improvement story, though, is oversold. The paper never compares against random sentence removal. Since the removed sentences are systematically shorter and higher-perplexity (their own Table 1), any pruning that cuts hard-to-process text might improve accuracy regardless of whether the labels are clinically meaningful. Table 4 reinforces that worry: every pruning criterion, including the models' own self-reports, improves accuracy on most datasets, so the effect does not look specific to physician labels.\n\nThe SR metric has a related problem. It interprets a correct-to-incorrect flip after pruning as evidence of spurious reliance, but that inference only holds if the remaining high-relevance sentences are sufficient to answer correctly. Their own Table 7 shows physician trainees answer correctly on only 67.2% of the pruned subset, so S+ is not sufficient even for the humans. A flip could just mean the pruned context is not enough to answer from.\n\nTwo smaller but real issues: the human baseline in Figure 3/Table 7 is computed on a 248-QA subset while the LLM accuracy is on the full 1,300, so the 'improves both trainees and LLMs' comparison is apples to oranges. And the LLM relevance measures are not comparable across models – ContextCite for open-source, self-report for GPT-4o – so the concordance table conflates model differences with method differences.\n\nNone of this sinks the dataset or the descriptive alignment finding. The annotation is careful, the instructions are concrete, and the Limitations section frankly concedes that ContextCite and self-reported labels may not capture relevance well. The causal claim about physician-guided pruning is what needs work. A random-removal baseline, matched for sentence length and perplexity, would directly address it, along with reporting all models on the same QA subset.\n\nWho should read this: anyone building or evaluating medical QA systems, and people working on context attribution or human-AI alignment. It deserves a serious referee. I'd recommend sending it to peer review with a clear request for the random-removal control and consistent evaluation subsets.","headline":"A genuinely new and useful relevance-label dataset for medical QA, but the headline pruning-accuracy claim needs a random-removal control before it can be trusted.","tokens_in":27469,"tokens_out":2735,"would_cite":true,"duration_ms":26721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Correct medical answers can hide flawed reasoning; pruning to physician-relevant sentences exposes the gap.","keywords":["MedPAIR","medical question answering","relevance alignment","sentence-level annotation","spurious rate","physician trainees","large language models","context attribution"],"falsifier":"Run the same pruning experiment but remove an equal number of sentences selected at random instead of by physician labels; if random pruning produces the same or larger accuracy gains on the same 1,300 questions, then the reported improvements do not depend on the content of the relevance labels.","tokens_in":26417,"feed_emoji":"🩺","tokens_out":6445,"duration_ms":60329,"temperature":0.7,"pith_summary":"The paper tries to establish that correct answers in medical question answering do not reveal whether a model reasoned the way a clinician would: large language models systematically disagree with physician trainees about which sentences in a patient case are relevant. It introduces MedPAIR, a dataset of 1,300 clinical QA pairs with sentence-level relevance labels from 36 physician trainees, and measures how often model-assigned relevance matches those labels. The central result is that when the context is pruned to only the sentences physician trainees judged highly relevant, accuracy improves for the trainees and for every tested language model. The paper reads this as evidence that models often lean on spurious or distracter information, and it proposes a Spurious Rate metric to quantify how often a correct answer flips to wrong after such pruning.","feed_headline":"Doctor-relevant sentences lift both human and AI medical scores","feed_subtitle":"New benchmark of 1,300 clinical questions shows pruning to physician-relevant context boosts accuracy for both.","key_machinery":"The load-bearing object is the majority-vote sentence relevance label: each sentence in a patient vignette is classified by physician trainees as high, low, or irrelevant, and these labels define the set $S^+$ that physicians deem sufficient to answer the question. The Spurious Rate is then the fraction of questions a model answers correctly on the full context but incorrectly on $S^+$ alone, with the numerator counting correct-to-wrong flips and the denominator counting all correct answers on the full context. The paper also maps numerical context-attribution scores to the same three categories by keeping the top-$k$ sentences where $k$ is the number of physician-relevant sentences, which is what makes model-human concordance comparable.","core_discovery":"MedPAIR records, for each of 1,300 multiple-choice medical questions drawn from four existing benchmarks, sentence-level trinary labels (high relevance, low relevance, irrelevant) assigned by physician trainees and aggregated by majority vote. Comparing those labels with the relevance estimates produced by four large language models—three via a context-attribution score and one via self-reported sentence judgments—the paper finds agreement on only about 45 to 66 percent of sentences marked highly relevant by clinicians. When the patient case is cut down to the physician-relevant sentences, accuracy rises for both the human labelers and every tested model, with gains as large as roughly 20 percentage points on the hardest benchmark; the Spurious Rate, which counts the fraction of originally correct answers that become wrong after pruning, ranges from about 2 to 18 percent across models. The paper concludes that exam-style accuracy overstates model competence and that expert-labeled relevance exposes a previously unmeasured misalignment.","pith_inferences":["The pruning experiment lacks a random-removal control, so part of the reported gain could come from shorter context alone; comparing physician label pruning with equal-length random pruning on the same questions would settle this.","The labeled set only includes questions physician trainees answered correctly, so the measured disagreement is conditional on trainee success; alignment on cases that trainees themselves miss is left open.","A testable extension is to train a model to predict the physician relevance labels rather than prune at inference time; if the labels carry causal signal, such training should improve accuracy on full unpruned cases.","The near-flat position profile of the attribution scores relative to expert labels suggests the attribution estimator itself may rank sentences weakly, so comparing two attribution methods on the same cases could separate model misalignment from estimator noise."],"forward_implications":["Correct answers on medical QA benchmarks can overstate reasoning quality, because a model may reach the right answer by leaning on sentences physicians mark as irrelevant.","Filtering out physician-irrelevant sentences is itself a low-cost intervention that raises accuracy for both humans and models, so context curation should be evaluated alongside model improvements.","The Spurious Rate gives a numeric handle on whether a model's correct answers are robust to distractor removal, making it a candidate diagnostic for clinically deployable systems.","The largest gains appear on the benchmark with the longest cases and the largest option set, suggesting that relevance misalignment matters most in information-dense clinical scenarios."],"supporting_citations":[{"why":"Supplies the context-attribution method that generates sentence-level relevance scores for the open-weight models.","marker":"[18]"},{"why":"Provides two of the four medical QA benchmarks (JAMA Clinical Challenge and Medbullets) used as data sources.","marker":"[11]"},{"why":"Provides the precision-medicine QA subset used as the third data source.","marker":"[31]"},{"why":"Provides the fourth, expert-level medical QA benchmark used as a data source.","marker":"[69]"},{"why":"The closed large language model whose self-reported relevance labels and accuracy are evaluated.","marker":"[44]"},{"why":"One of the open-weight models whose answers and attribution-based relevance scores are evaluated.","marker":"[27]"},{"why":"Another open-weight model whose answers and attribution-based relevance scores are evaluated.","marker":"[48]"}],"fun_headline_variants":["Physician-relevant context boosts medical QA accuracy for humans and AI","Pruning to physician-relevant sentences improves medical AI answers","Expert-relevant context improves both human and AI medical scores","Cutting irrelevant sentences lifts medical QA accuracy for doctors and AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the sentences physician trainees mark as highly relevant are by themselves sufficient to answer the question correctly, so any model that is correct on the full case but wrong after pruning must have been relying on something doctors would call irrelevant.","fun_headline_variants_meta":{"raw":{"variants":["Physician-relevant context boosts medical QA accuracy for humans and AI","Pruning to physician-relevant sentences improves medical AI answers","Expert-relevant context improves both human and AI medical scores","Cutting irrelevant sentences lifts medical QA accuracy for doctors and AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000561,"raw_usage":{"total_tokens":2649,"prompt_tokens":916,"completion_tokens":1733,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1663}},"tokens_in":532,"tokens_out":1733,"duration_ms":12443,"temperature":1.0,"reasoning_tokens":1663,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:37:01.271359+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pruning experiment but remove an equal number of sentences selected at random instead of by physician labels; if random pruning produces the same or larger accuracy gains on the same 1,300 questions, then the reported improvements do not depend on the content of the relevance labels.","supporting_citations":[{"cited_title":"ContextCite: Attributing Model Generation to Context","cited_arxiv_id":null,"evidence_quote":"Supplies the context-attribution method that generates sentence-level relevance scores for the open-weight models."},{"cited_title":"Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions","cited_arxiv_id":null,"evidence_quote":"Provides two of the four medical QA benchmarks (JAMA Clinical Challenge and Medbullets) used as data sources."}],"review_version":1}