{"id":"2416e87c-577c-4d00-8bcf-43d0a25f7bb3","arxiv_id":"2608.01012","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Mid-stream alignment on next-step clinical decisions (MedUPS) improves LLM next-step accuracy on uncommon cases, but the size of the gain depends substantially on which LLM judge does the scoring.","lead":"This paper introduces MedUPSQA, a dataset of 21,874 mid-stream clinical decision points built from 5,535 real uncommon-case reports, and MedUPS, a training method that aligns language models to predict the next clinical step as a case unfolds. Generalists should read it because it reframes medical LLM evaluation from final diagnoses to the sequential decisions clinicians actually make, and reports that a targeted training objective can matter more than model size.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Judge dependence undermines the 'alignment over scale' claim: the same model family provides reward, metric, and data filter, and re-scoring the strongest checkpoint with GPT-5.6 moves accuracy by 9.2 points, more than the 5.8-point headline margin.","rationale":"The reader's weakest-assumption identification matches my own: the primary evaluation metric and the training reward are produced by the same LLM-judge family, and the paper's own cross-judge re-scoring shows the effect is large enough to reverse the central 'alignment over scale' margin. The paper is unusually transparent about this, explicitly declining to claim judge-independent ordering and listing clinician validation as the most needed follow-up. However, the abstract and headline results still assert that smaller aligned models surpass larger frontier models, and that claim is not supported by the evidence as presented. My recommended verdict remains CONDITIONAL: the dataset and the consistent alignment gains across three backbones are real contributions, and the SFT-vs-GRPO comparison on identical pairs is a sound way to isolate the optimizer. But the scale-claim must be conditioned on independent clinician-validated evaluation. I do not see an additional load-bearing concern beyond this judge-dependence issue; the paper's other limitations (single corpus, contamination risk, single-seed training, unvalidated chunking) are acknowledged and do not separately threaten the central argument. Therefore no change to the reader's conditional verdict is warranted.","tokens_in":16466,"tokens_out":3050,"duration_ms":34739,"concrete_test":"Have two independent clinicians, blinded to model identity and to all LLM-judge verdicts, grade a stratified sample of at least 200 predictions (or the full 500) from the fixed evaluation pool for clinical equivalence to the reference continuation, using the same binary equivalence definition as Figure A2. Compute clinician-judged accuracies for the key comparison set: base Qwen3.6-27B, MedUPS-Qwen3.6-27B, GPT-5.6-sol, and GPT-5.6-luna. If the clinician-judged margin of MedUPS-Qwen3.6-27B over GPT-5.6-sol is not positive (or is smaller than the 9.2-point judge gap), then the 'alignment over scale' claim fails. A necessary intermediate step is to re-score the entire Table 3 under GPT-5.6 and Opus 5 to test ordering stability, but only clinician labels can settle whether the metric measures clinical next-step competence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that alignment matters more than scale rests entirely on an LLM-judge evaluation that is neither validated against clinician labels nor, by the paper's own numbers, judge-independent. In Section 5 ('How much depends on the judge'), re-scoring the strongest checkpoint's 500 predictions with GPT-5.6 lowers accuracy from 67.2 to 58.0, a swing of 9.2 points that exceeds the 5.8-point lead of MedUPS-Qwen3.6-27B over GPT-5.6-sol under the DeepSeek judge. Since only the strongest checkpoint was re-scored, the full Table 3 ordering under a second judge family is unknown, so the 'smaller aligned model surpasses frontier models' conclusion is not established. The problem is compounded by the pipeline's self-referentiality: DeepSeek-R1 generates questions and answers, a DeepSeek-family construction judge filters which 47.1% of candidate decision points survive, DeepSeek-Chat supplies the GRPO reward, and DeepSeek-Chat is also the primary evaluation judge. The retained supervision is therefore by construction what a DeepSeek-family judge considers equivalent, and an aligned policy can raise scores by conforming to that judge's equivalence notion even when clinical decisions are unchanged. The paper is transparent about this in Section 4 and Limitations, but transparency does not resolve the core issue: the magnitude of the reported gains, and especially the cross-scale ordering, is judge-relative until clinician-validated scoring is performed. The error analysis does not close this gap, since its failure attribution is performed by an LLM rater instructed to err against labeling failures as clinical, and only for the strongest aligned model.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MedUPSQA, a dataset of 21,874 mid-stream clinical decision points derived from 5,535 real case reports, and MedUPS, a GRPO-based alignment framework that trains LLMs to predict the next clinical step from an accumulating case prefix. The authors segment case reports into chronological chunks, generate questions and candidate answers with DeepSeek-R1, filter them with an LLM judge, and train with a four-criterion DeepSeek-Chat rubric reward. Evaluation uses a fixed 500-item stratified test pool scored by a binary DeepSeek-Chat equivalence judge. The paper reports that GRPO alignment improves next-step accuracy over the base backbones (e.g., Qwen3.6-27B from 55.2 to 66.7), that SFT on the same pairs also improves all backbones, and that the aligned 27B model outperforms GPT-5.6 variants under the paper's judge. The paper includes a judge-robustness analysis, an error attribution study, and explicit limitations acknowledging judge dependence.","tokens_in":16805,"tokens_out":5075,"duration_ms":52694,"significance":"If the reported gains reflect genuine clinical decision quality, the dataset and the mid-stream alignment objective would be a valuable contribution to medical LLM research, addressing a real gap between terminal-diagnosis benchmarks and the sequential, information-limited nature of clinical care. The paper is commendably transparent: it releases data, code, and checkpoints, reports bootstrap CIs, compares GRPO against SFT on identical pairs, and explicitly quantifies judge disagreement. However, the central 'alignment over scale' claim rests on an LLM judge from the same model family as the training reward and the data-construction filter, and the paper's own alternate-judge rescoring shows a 9.2-point swing that exceeds the claimed 5.8-point margin over a frontier model. The contribution is therefore promising but not yet established as a statement about clinical competence or model ordering.","major_comments":[{"comment":"The paper reports that rescoring MedUPS-Qwen3.6-27B with GPT-5.6 lowers accuracy from 67.2 to 58.0, a 9.2-point change larger than the 5.8-point margin by which this model leads GPT-5.6-sol under the DeepSeek judge. Only the strongest checkpoint was rescored, so the full Table 3 ordering under a second judge family is unknown. The abstract claim that 'smaller models surpassing larger, frontier models we evaluate' is therefore not supported by the evidence. The paper explicitly says 'we do not claim that the open-versus-frontier ordering is judge-independent,' but the abstract and introduction still present the scale comparison without the same caveat. This load-bearing claim needs to be either supported by rescoring all models in Table 3 or downgraded to a judge-conditional observation.","section":"§5, 'How much depends on the judge'"},{"comment":"The GRPO reward and the primary evaluation metric are produced by the same judge family (DeepSeek-Chat), and the construction-time filter also uses a DeepSeek-family judge. Thus the retained supervision is, by construction, what a DeepSeek-family judge considers equivalent to the reference continuation, and an aligned policy can increase its score by conforming to that judge's equivalence notion without changing clinical decisions. The 30-instance manual prompt-development check is explicitly not an independent validation. This design makes it impossible to interpret the reported gains as clinical improvement without external validation. The claim that 'the target carries signal independently of the optimizer' is weakened because the target itself is filtered by the same judge family. A clinician-scored subset, even a few hundred items, is needed to anchor the metric.","section":"§3.3, §4, and Limitations"},{"comment":"The error attribution is performed by an LLM rater instructed to 'resolve doubt against calling a failure clinical,' so the resulting 2.0% category-E rate is a lower bound by construction and is not clinician-verified. The paper uses this analysis to argue that the headline accuracies 'understate clinical competence,' but that inference is not supported: an unvalidated rater cannot establish that the LLM judge's verdicts are conservative. This is important because it is used to soften the judge-dependence concern. External clinician review of a stratified failure sample would be needed to support this interpretation.","section":"§5, 'Error analysis'"}],"minor_comments":[{"comment":"Typo: 'target framwork carries signal' should be 'target framework carries signal.' Also, 'with 95% CI' in the abstract is vague; the CIs are bootstrap intervals over a fixed 500-item pool, not over training runs, as correctly explained in Section 4.","section":"Abstract"},{"comment":"The prompt text contains typos ('wether', 'wich') that are acknowledged as reproduced verbatim. This is acceptable, but please note in a caption that typos are intentional and not OCR artifacts, since readers may otherwise file correction requests.","section":"Appendix C, Figure A3"},{"comment":"Typo: 'strenghten' should be 'strengthen.' Also, the sentence beginning 'however, due to the open, worldwide access...' is grammatically broken and should be revised for clarity.","section":"Limitations"},{"comment":"The bootstrap procedure uses only 5 resamples (seeds 1001–1005). This is a small number; 1000 would give more stable intervals. Please justify or increase the number of resamples.","section":"§4, Evaluation protocol"},{"comment":"The dataset construction uses GPT-4o for chunking and DeepSeek-R1 for generation. Please clarify whether the chunking and generation prompts are released with the dataset, as stated, and specify the exact model versions/dates for reproducibility.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is unusually honest about its limitations, which is a strength. However, the headline claim of 'alignment over scale' is not supported by the paper's own judge-robustness numbers, and the primary metric is not externally validated. I believe this can be fixed within scope by (a) reframing the paper's main claim as a dataset and training framework contribution with judge-conditional results, or (b) adding a clinician-validated evaluation subset and/or rescoring the full model table under an alternate judge. Because the dataset and the mid-stream objective are potentially useful to the community, I recommend major revision rather than rejection. The authors should also ensure the released artifacts include all judge prompts and exact model snapshots so that independent validation is feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing before you read it. The real contribution is the target and the data: next-step prediction along real patient trajectories, trained without final-diagnosis labels, on 21,874 decision points from 5,535 case reports. The headline claim — that mid-stream alignment matters more than scale — is judge-relative, and the paper's own numbers show it: re-scoring the best checkpoint with GPT-5.6 drops accuracy by 9.2 points, more than the 5.8-point margin that motivated the claim.\n\nWhat the paper does well: it isolates optimizer from target by training GRPO and SFT on identical pairs, and it gets a consistent alignment gain across three backbones — that effect is probably real. The error analysis is honest, with a 'clinically wrong' category they deliberately treat as a lower bound; only 2% of decisions land there, and on Action questions none of the failures were clinical errors. The judge-robustness analysis is well done: agreement rates, Cohen's kappa, and the asymmetric rejection pattern. They release dataset, code, and checkpoints. The limitations section is candid, not pro forma.\n\nSoft spots. The circularity is real and it is what the scale claim rests on: DeepSeek-R1 generates the questions and answers, a DeepSeek-family judge filters which instances survive, DeepSeek-Chat supplies the GRPO reward, and DeepSeek-Chat is the evaluation judge. An aligned policy can gain by conforming to the judge's equivalence notion. The authors acknowledge this, but acknowledgment doesn't turn the 'alignment over scale' conclusion into an established result — it stays conditional. Second, only the strongest checkpoint was re-scored with alternate judges, so the full model ordering under a second judge family is unknown. Third, there are no clinician labels: the 30-instance prompt review was done by the authors and the paper says it is not independent validation. Smaller issues — unablated rubric weights, untuned SFT recipe, single corpus — are all disclosed.\n\nThe stress-test note lands, but it doesn't expose a hidden flaw; the paper already tells you this. The abstract is more confident than the body supports, and the body mostly walks it back.\n\nWho it's for: people building clinical LLM evaluation, dense supervision, or working on LLM-judge bias. It deserves a serious referee; I would send it to review with the expectation that a clinician-validated subset is needed before the scale claim can be stated cleanly.","headline":"Useful new dataset and a clean GRPO-vs-SFT comparison, but the 'alignment over scale' claim is judge-dependent by the paper's own re-scoring (9.2-point swing vs a 5.8-point margin) and stays conditional pending clinician labels.","tokens_in":17333,"tokens_out":4686,"would_cite":true,"duration_ms":41092,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mid-stream alignment lifts next-step clinical accuracy up to 11 points","keywords":["clinical decision support","large language models","mid-stream clinical reasoning","next-step prediction","reinforcement learning","GRPO","uncommon cases","LLM-as-a-judge"],"falsifier":"Take the fixed 500-item evaluation pool and score it with a clinician-validated reference standard, or with a judge family never used in training. If MedUPS-Qwen3.6-27B no longer beats its own base model or GPT-5.6, or if the margin collapses, the central claim is falsified.","tokens_in":16343,"feed_emoji":"🩺","tokens_out":6545,"duration_ms":60696,"temperature":0.7,"pith_summary":"This paper claims that medical LLM evaluation and training should focus on the next clinical step in a patient's trajectory rather than the final diagnosis. The authors introduce MedUPSQA, 21,874 mid-stream decision points derived from 5,535 real case reports, and MedUPS, an alignment framework that uses GRPO to reward models for predicting what the report records next. Across three backbones, the objective raises next-step accuracy by 6.6 to 11.4 percentage points, and under the paper's judge a 27B aligned model scores above substantially larger frontier models. A fair reader would take away that targeted intermediate supervision can matter more than model scale for uncommon-case reasoning, while noting the authors' own caveat that the metric is judge-produced and not clinician-validated.","feed_headline":"Mid-stream alignment lifts next-step clinical accuracy up to 11 points","feed_subtitle":"Training on next steps, not final diagnoses, lets a 27B model beat larger frontier LLMs under the paper's judge.","key_machinery":"The core mechanism is converting free-text case presentations into chronologically ordered clinical chunks, generating for each prefix a next-step question and a chain-of-thought answer, filtering instances with an LLM judge that checks clinical equivalence to the continuation, and aligning policies with Group Relative Policy Optimization (GRPO) under a four-criterion rubric reward (correct, specific, grounded, complete). This yields dense, trajectory-aware supervision without any final-diagnosis labels.","core_discovery":"The central discovery is that dense, label-free supervision at the level of the next action—generated from real case presentations and filtered by an LLM equivalence judge—can be optimized with GRPO to improve mid-stream reasoning on uncommon cases. The paper shows the target carries signal independent of the optimizer (SFT on identical pairs also improves every backbone), that GRPO is the stronger learning signal at the largest tested scale, and that mid-stream alignment moves a 9B model above its unaligned 27B counterpart. The authors state plainly that the reward and primary metric are produced by the same model family and are not validated against clinician labels, and they quantify that","pith_inferences":["An implication the paper leaves implicit is that the judge-dependence check was run only on the strongest checkpoint; if a full-table re-scoring shows similar sensitivity, the 'alignment over scale' conclusion could be an artifact of reward hacking rather than a general property of the objective.","The same chunk-and-predict framing could transfer to other longitudinal clinical tasks—treatment escalation, discharge planning, ICU management—where the next action is the object of interest.","Because the construction filter retains only instances an LLM judge accepts, the training distribution is by construction biased toward judge-recognizable equivalences; this may limit what the models learn about clinically defensible alternatives that the single report did not record.","The mixed GRPO-versus-SFT results across backbones suggest the advantage of RL over imitation may depend on base-policy strength; a per-backbone tuned SFT baseline would be needed to separate objective effects from recipe fit."],"forward_implications":["If the central claim holds, medical LLM benchmarks should measure sequential next-step decisions rather than terminal diagnoses.","Training on mid-stream decisions could make smaller open models viable for on-premises hospital deployment, easing privacy and audit constraints.","The released dataset and checkpoints allow independent, clinician-grounded validation to test whether the gains survive a human-judged reference standard.","The paper's error analysis suggests action-question failures are mostly reference under-specification, motivating evaluation with multiple acceptable next steps.","The alignment-over-scale result, if it persists under a second judge family, would argue for investing in task-specific alignment rather than model size for uncommon cases."],"supporting_citations":[{"why":"Supplies the CUPCase corpus of uncommon case reports that MedUPSQA extends.","marker":"Perets et al. 2025"},{"why":"Provides GRPO, the reinforcement-learning optimizer used to align the policies.","marker":"Shao et al. 2024"},{"why":"Establishes the clinical stakes of diagnostic error that motivate mid-stream supervision.","marker":"Ball, Miller, and Balogh 2015"},{"why":"Documents diagnostic delay and misdiagnosis in rare diseases, motivating the uncommon-case focus.","marker":"Faye et al. 2024"},{"why":"Represents the exam-style, final-diagnosis-only benchmarks the paper contrasts with.","marker":"Liévin et al. 2024"},{"why":"MedCaseReasoning is the closest prior work on reasoning supervision from case reports and a main point of comparison.","marker":"Wu et al. 2025b"}],"fun_headline_variants":["Next-step training lifts clinical accuracy by 11 points","Smaller model beats larger one via next-step alignment","MedUPS: Training on next actions, not final diagnoses","LLM judge reward boosts next-step accuracy up to 11 pts","Mid-stream supervision improves LLM clinical decisions"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"Both the training reward and the evaluation metric come from DeepSeek-family LLM judges that are not validated against clinician labels, so the size of the reported gains and the claim that alignment beats scale are only as strong as the judge's notion of clinical equivalence.","fun_headline_variants_meta":{"raw":{"variants":["Next-step training lifts clinical accuracy by 11 points","Smaller model beats larger one via next-step alignment","MedUPS: Training on next actions, not final diagnoses","LLM judge reward boosts next-step accuracy up to 11 pts","Mid-stream supervision improves LLM clinical decisions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000662,"raw_usage":{"total_tokens":2922,"prompt_tokens":862,"completion_tokens":2060,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":1982}},"tokens_in":606,"tokens_out":2060,"duration_ms":17906,"temperature":1.0,"reasoning_tokens":1982,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:34:26.327687+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the fixed 500-item evaluation pool and score it with a clinician-validated reference standard, or with a judge family never used in training. If MedUPS-Qwen3.6-27B no longer beats its own base model or GPT-5.6, or if the margin collapses, the central claim is falsified.","supporting_citations":[],"review_version":1}