Artificial Intelligence · arXiv:2610.17690 v2 · submitted 13 Oct 2026
MedEvo: Medical Reasoning Evolution through Knowledge Navigation
The paper says
“Medical multimodal reasoning models are typically trained on teacher-generated rationales, discarding the valid reasoning in students' own failed trajectories, while general-domain correction methods offer little guidance for errors in image reading, anatomical context, or differential diagnosis.”
more of the abstract
We propose MedEvo, which uses medical knowledge, namely a tag taxonomy and error-type diagnosis, to navigate both trajectory repair and data selection. For each failed rollout, a gold-blind auditor locates the first consequential error and provides a minimal, answer-free hint; the same student retains the valid prefix and regenerates the erroneous suffix. After each round, MedEvo identifies capability gaps across medical tags on a held-out feedback set and prioritizes related questions for new rollouts and repairs. With supervised fine-tuning (SFT) alone, MedEvo raises Qwen3.5-9B mean accuracy across seven benchmarks from 65.57% to 69.12% (47.40% to 54.50% on the multimodal track of MedXpertQA (MedXpertQA-MM)), surpassing teacher-distillation SFT (66.93%). It also raises Qwen3.5-35B and Qwen3.5-397B mean accuracy from 67.91% to 73.04% and from 71.83% to 74.95%, respectively. Our analyses show that failed trajectories contain reasoning worth preserving, that repair success does not imply learning value, and that independent feedback is key to sustained improvement.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 14 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Solid empirical paper with a clear dual-use concern in the feedback calibration—deserves review with reproducibility conditions. This line is wrong
2 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at §3.2.4, Appendix A.4
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The learnability gate thresholds (D_repair in [0.35,0.85], Δ_repair in [0.02,0.30]) are calibrated on the 5,000-question feedback set. The same feedback set is used for measuring capability gaps and guiding question selection across rounds (§3.3.2, Eq. 14-15). This dual role creates an unmeasured risk: threshold tuning may exploit peculiarities of the feedback set, and the subsequent selection mechanism, driven by the same set, may compound those peculiarities by preferentially retrieving questions whose repairs the calibrated thresholds admit.
more of objection 1
The paper reports no split-set validation (e.g., calibrate on 60% of the feedback set and evaluate threshold stability on the held-out 40%) and no sensitivity sweep (e.g., shifting both bounds by ±0.05). Given that removing feedback causes accuracy to fall below the R1 baseline (Table 5), the feedback mechanism is load-bearing; therefore, the dual-use risk is also load-bearing. The authors acknowledge that thresholds are 'calibrated on the feedback set' but do not test their robustness. This gap should be addressed either by additional experiments or by an explicit discussion of why the risk is limited in this setting.
The paper says, at §5.4, Fig. 5a
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.96):
“Evaluating open-source auditors and separating the effects of trajectory length from question difficulty would clarify these dependencies.”
Pith says: Objection 2 of 2 · major
The trajectory-selection experiment compares the 'longest-8k' questions with a random-8k subset. The longest-8k set has lower repair success but yields higher downstream accuracy. This is presented as evidence that repair success and learning value differ. However, the comparison conflates trajectory length with question difficulty or other confounds.
more of objection 2
The paper acknowledges this in §7.1: 'it does not isolate trajectory length from question difficulty.' While this caveat is noted, the central claim of the experiment—that 'learning-based trajectory selection' is favored—would be strengthened by controlling for difficulty or by comparing questions of similar difficulty that differ in trajectory length. As written, the result is suggestive but not conclusive.
5 minor objections
The paper says, at Table 7
Pith says: minor objection 1 of 5
The human audit reports Cohen's kappa and bootstrap CIs but does not state the total number of trajectories audited. Please add the sample size to the table caption or the main text.
The paper says, at §4.3, Table 3
Pith says: minor objection 2 of 5
The right panel of Table 3 reports 95% bootstrap CIs. State explicitly that these are percentile intervals or the bootstrap method used (e.g., 'percentile bootstrap with 10,000 resamples').
show 3 more
The paper says, at Fig. 6d
Pith says: minor objection 3 of 5
The 'Heuristic' baseline (HHD-inspired hint recovery) is compared with MedEvo across difficulty levels. The caption notes that Heuristic outperforms MedEvo at L5, but the main text in §5.4 only mentions that MedEvo exceeds Heuristic at L1–L4. This asymmetry should be discussed: the hardest questions (L5) may benefit from an HHD-style approach more than from MedEvo's current design.
The paper says, at Appendix D.3
Pith says: minor objection 4 of 5
The audit prompt is long and contains many detailed instructions. Consider providing a succinct summary of the key constraints (e.g., answer-blindness, evidence tracing) in the main text to improve readability.
The paper says, at §3.2.1
Pith says: minor objection 5 of 5
Equation (3) defines entropy with natural log, but the paper does not specify the base of the logarithm in Eq. (10) for the loss. Clarify that ℒ_t also uses natural log (nats) throughout.
What would settle it
Re-run Qwen3.5-9B with two variants: (a) threshold sensitivity: shift D_repair bounds by ±0.05 and Δ_repair bounds by ±0.01/±0.05 (e.g., [0.30,0.90], [0.40,0.80]; [0.01,0.35], [0.03,0.25])—report seven-benchmark mean for each variant. (b) Split validation: randomly split the 5,000 feedback questions into calibration (60%) and hold-out (40%); calibrate thresholds on the calibration split only; run three Rounds of MedEvo using these thresholds; report seven-benchmark mean and the hold-out-feedback-set accuracy. If the mean shifts by more than ±0.5 pp across any variant, the thresholds are a material source of variability.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 14 Oct 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This is good work. MedEvo combines several known ideas—prefix preservation, self-training, hint-based distillation—and adds a medical knowledge structure (tag taxonomy and error-type diagnosis) to guide where to repair and what to learn next. The results are clear: mean accuracy on seven medical multimodal benchmarks goes from 65.57% to 69.12% on Qwen3.5-9B, with similar gains on 35B and 397B. The experimental design is thorough: bootstrap confidence intervals, three independent runs at 9B, a human audit with Cohen's kappa up to 0.94, and sensible ablations that show the contribution of each component. The results look reproducible, and the paper's insights (e.g., repair success != learning value) are useful.
The main novelty is the coupling of medical knowledge navigation (tag taxonomy and error-type diagnosis) with trajectory repair and feedback-driven data selection. Combining error diagnosis, prefix retention, and capability-gap feedback into one loop is new in the medical domain.
The main soft spot is that the learnability gate thresholds and the feedback-driven question selection both use the same 5,000-question feedback set. The paper calibrates the thresholds (D_repair in [0.35,0.85], Delta_repair in [0.02,0.30]) on that set and also measures tag-level error rates on it to guide question selection each round. This dual use creates a real risk: threshold tuning may exploit peculiarities of the feedback set, and the selection mechanism can then compound those peculiarities by preferentially retrieving questions whose repairs the calibrated thresholds admit. The paper reports no split-set validation or sensitivity sweep. The authors acknowledge the thresholds are calibrated on the feedback set but do not test robustness. It is a genuine weakness, though it does not undercut the headline gains—those are measured on held-out benchmarks. Still, the claim that the feedback mechanism is key is weakened without empirical defense of the threshold stability. Separately, the auditor is proprietary (GPT-5.6-sol) and code+data are not yet released, which limits immediate reproducibility. The paper itself states this limitation.
Overall this paper deserves a serious referee. I would recommend conditional acceptance contingent on release of code and data for independent verification. Adding a split-set validation or sensitivity analysis for the thresholds would strengthen the work, but should not block review.
The strongest objection
The paper uses one fixed 5,000-question feedback set for two distinct purposes: (1) calibrating the learnability gate thresholds D_repair, Δ_repair (§3.2.4, Appx A.4), which determine which repairs enter training, and (2) measuring capability gaps to guide question selection across rounds (§3.3.2, Eq. 14–15). This dual use creates an unmeasured risk: threshold tuning may exploit peculiarities of the feedback set, and the selection mechanism—driven by the same set—can then compound those peculiarities by preferentially retrieving questions whose repairs the calibrated thresholds admit. The paper reports no split-set validation (e.g., calibrate on 60%, evaluate threshold stability on held-out 40% of the feedback set) and no sensitivity sweep (e.g., shift both bounds by ±0.05). Without such checks, the headline gains on the seven held-out benchmarks could partly reflect thresholds that overfit the feedback set rather than genuinely selecting learnable repairs. The ablation in Table 5 shows that removing feedback causes accuracy to fall below the R1 baseline (67.22→66.67), confirming the centrality of the feedback mechanism; this makes the dual-use risk load-bearing. The authors acknowledge thresholds are 'calibrated on the feedback set' but do not test robustness, which is a concrete gap that a sensitivity analysis would close.
The reader identified threshold generalization as the weakest assumption; this concern agrees with that diagnosis but sharpens it by highlighting the dual role of the same feedback set (calibration + election) as a compounding factor the reader didn't explicitly discuss. The reader's CONDITIONAL ver dict already requires robustness checks; this stress-test provides a specific test to address the gap. Since the concern is squarely within the reader's identified weakest area and does not change the overall assessment, UNCHANGED is appropriate.
The referee's report
If the reported gains are robust, the paper makes several contributions. It demonstrates that preserving the valid prefix and providing a minimal, error-type-conditioned hint yields better downstream accuracy than full rewriting or generic hints. It shows that repair success and learning value can diverge, motivating learnability-based screening. It provides evidence that independent feedback (a fixed held-out set) is more effective than training-error feedback for guiding data evolution. The method scales to 35B and 397B models and outperforms teacher-distillation SFT on the same backbone. The experimental design is thorough: seven benchmarks, three independent runs, bootstrap confidence intervals, human annotation with inter-annotator agreement metrics, and component ablations. The paper also releases prompts, training configurations, and an explicit information-access matrix, supporting reproducibility.
Pith's reading of the paper (a machine's summary, not the authors')
The paper argues that common training methods for medical multimodal reasoning models discard valuable valid reasoning contained in a student's own failed trajectories, replacing them with teacher-written solutions. The authors propose MedEvo, which integrates a medical tag taxonomy and a 20-type error codebook to localize the first consequential error in a failed rollout, preserve the correct prefix, provide a minimal answer-free hint, and let the same student regenerate the erroneous suffix. Across multiple rounds, feedback on a held-out set identifies capability gaps across medical tags and error types, steering subsequent question selection and repair effort toward remaining weaknesses. The central thesis is that correcting a response and constructing useful supervision are different objectives, and that preserving valid reasoning, filtering repairs for learnability, and using independent feedback each meaningfully affect what a student learns.
What carries the argument. The central machinery is Knowledge Navigation, instantiated through a medical tag taxonomy (four axes: modality, anatomy, disease, difficulty) and an error codebook (four parent types with five subtypes each, totalling twenty). Within a trajectory, token-level entropy guides the auditor to a candidate window; the gold-blind auditor then identifies the first consequential error, selects a safe rollback boundary, and constructs a minimal answer-free hint by instantiating the recovery operation associated with the diagnosed error type. Across rounds, tag-level error rates and error-type distributions on a fixed feedback set determine question-selection priorities and repair-order rules. A learnability gate (screening ratios $D_{\text{repair}} \in [0.35, 0.85]$ and $\Delta_{\text{repair}} \in [0.02, 0.30]$) filters repairs for student-relative difficulty before admitting them as supervised fine-tuning targets.
Core claim. The core discovery is that failed medical reasoning trajectories contain substantial valid reasoning that can be salvaged, and that structured medical knowledge can navigate both within-trajectory repair and cross-round data evolution to produce consistent gains. MedEvo achieves this by using a gold-blind auditor (accessible to the paper only through a proprietary model) that diagnoses errors and supplies a recovery hint without ever seeing the reference answer; the student then regenerates only the erroneous suffix while keeping its own valid prefix verbatim. On Qwen3.5-9B, mean accuracy across seven medical multimodal benchmarks rises from 65.57% to 69.12%, surpassing teacher-distillation SFT (66.93%), and the improvement extends to 35B (73.04%) and 397B (74.95%) models. The analysis further shows that repair success does not imply learning value: trajectories that are harder to repair can yield larger downstream accuracy gains, and feedback from an independent held-out set, rather than from training errors, is essential for sustained improvement.
The premise it rests on. The learnability gate thresholds ($D_{\text{repair}} \in [0.35, 0.85]$ and $\Delta_{\text{repair}} \in [0.02, 0.30]$) are calibrated on the held-out feedback set before final evaluation; if these thresholds do not generalize to new data distributions or if the feedback set is not representative of the target benchmarks, the filtering of repairs may introduce bias that inflates the reported gains.
If the paper is right (Pith's inferences, not the authors')
Medical vision-language models can be improved without teacher-generated rationales or reinforcement learning, using only the student's own repaired trajectories and structured feedback.
Preserving the valid prefix of a failed trajectory and the student's own regeneration of the suffix yields better downstream accuracy than full rewriting with a teacher solution.
Feedback from an independent held-out set, rather than from training errors, prevents overfitting to the training distribution and generalizes better to unseen benchmarks.
The repair workload (number of questions requiring repair) decreases over successive rounds while benchmark accuracy increases, suggesting that feedback-driven selection effectively targets the student's remaining weaknesses.
The method's dependence on a proprietary auditor is a practical limitation; if open-source auditors can replicate the error diagnosis and hint construction, the approach becomes more broadly accessible and auditable.
The learnability gate thresholds, calibrated on a single 5,000-question feedback set, may not be robust across different distributions of medical images or question difficulties, and the paper does not empirically test sensitivity to these threshold choices.
The finding that lower repair success can yield higher downstream accuracy suggests that learning value and repairability are in tension: over-easy repairs may not teach the model much, while harder-to-fix errors signal deeper gaps that, once addressed, produce larger gains.
The method's framework, structured domain knowledge plus failure-guided data evolution, could transfer to other expert domains such as legal reasoning or engineering diagnosis where a taxonomy of tasks and error types is available.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Medical multimodal reasoning models are typically trained on teacher-generated rationales, discarding the valid reasoning in students' own failed trajectories, while general-domain correction methods offer little guidance for errors in image reading, anatomical context, or differential diagnosis. We propose MedEvo, which uses medical knowledge, namely a tag taxonomy and error-type diagnosis, to navigate both trajectory repair and data selection. For each failed rollout, a gold-blind auditor locates the first consequential error and provides a minimal, answer-free hint; the same student retains the valid prefix and regenerates the erroneous suffix. After each round, MedEvo identifies capability gaps across medical tags on a held-out feedback set and prioritizes related questions for new rollouts and repairs. With supervised fine-tuning (SFT) alone, MedEvo raises Qwen3.5-9B mean accuracy across seven benchmarks from 65.57% to 69.12% (47.40% to 54.50% on the multimodal track of MedXpertQA (MedXpertQA-MM)), surpassing teacher-distillation SFT (66.93%). It also raises Qwen3.5-35B and Qwen3.5-397B mean accuracy from 67.91% to 73.04% and from 71.83% to 74.95%, respectively. Our analyses show that failed trajectories contain reasoning worth preserving, that repair success does not imply learning value, and that independent feedback is key to sustained improvement.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-14
The authors' main claim, as the classifier picked it out 0.99
“Our analyses show that failed trajectories contain reasoning worth preserving, that repair success does not imply learning value, and that independent feedback is key to sustained improvement.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.61 · human participants 0.07 · controlled comparison 0.93 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Medical multimodal reasoning models are typically trained on teacher-generated rationales, discarding the valid reasoning in students' own failed trajectories, while general-domain correction methods offer little guidance for errors in image reading, anatomical context, or differential diagnosis. | background 0.52 | 0.07 | 0.04 | asserted 1.00 |
| We propose MedEvo, which uses medical knowledge, namely a tag taxonomy and error-type diagnosis, to navigate both trajectory repair and data selection. | contribution 0.81 | 0.77 | 0.07 | asserted 0.99 |
| For each failed rollout, a gold-blind auditor locates the first consequential error and provides a minimal, answer-free hint; the same student retains the valid prefix and regenerates the erroneous suffix. | method 1.00 | 0.88 | 0.04 | asserted 0.99 |
| After each round, MedEvo identifies capability gaps across medical tags on a held-out feedback set and prioritizes related questions for new rollouts and repairs. | method 0.99 | 0.94 | 0.06 | asserted 0.97 |
| With supervised fine-tuning (SFT) alone, MedEvo raises Qwen3.5-9B mean accuracy across seven benchmarks from 65.57% to 69.12% (47.40% to 54.50% on the multimodal track of MedXpertQA (MedXpertQA-MM)), surpassing teacher-distillation SFT (66.93%). | result 1.00 | 0.96 | 0.16 | asserted 1.00 |
| It also raises Qwen3.5-35B and Qwen3.5-397B mean accuracy from 67.91% to 73.04% and from 71.83% to 74.95%, respectively. | result 1.00 | 0.94 | 0.06 | asserted 1.00 |
| Our analyses show that failed trajectories contain reasoning worth preserving, that repair success does not imply learning value, and that independent feedback is key to sustained improvement. | interpretation 0.94 | 0.95 | 0.76 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.93
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 5,958 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (7)
Figures are the authors', reproduced from the paper.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment