Pith.

Artificial Intelligence · arXiv:2610.17690 v2 · submitted 13 Oct 2026

MedEvo: Medical Reasoning Evolution through Knowledge Navigation

Xiaoliang Wang, Zhuo Chen, Chuanyang Zheng, Quanbin Wang, Jiangwei Lao, Yingao Liu, Jiajia Liu, Jinming Cao

show 6 more authors

Honglin Lin, Zhihong Lu, Hao Jin, Jian Wang, Jinjie Gu, Lijun Wu

The paper says

“Medical multimodal reasoning models are typically trained on teacher-generated rationales, discarding the valid reasoning in students' own failed trajectories, while general-domain correction methods offer little guidance for errors in image reading, anatomical context, or differential diagnosis.”

more of the abstract

We propose MedEvo, which uses medical knowledge, namely a tag taxonomy and error-type diagnosis, to navigate both trajectory repair and data selection. For each failed rollout, a gold-blind auditor locates the first consequential error and provides a minimal, answer-free hint; the same student retains the valid prefix and regenerates the erroneous suffix. After each round, MedEvo identifies capability gaps across medical tags on a held-out feedback set and prioritizes related questions for new rollouts and repairs. With supervised fine-tuning (SFT) alone, MedEvo raises Qwen3.5-9B mean accuracy across seven benchmarks from 65.57% to 69.12% (47.40% to 54.50% on the multimodal track of MedXpertQA (MedXpertQA-MM)), surpassing teacher-distillation SFT (66.93%). It also raises Qwen3.5-35B and Qwen3.5-397B mean accuracy from 67.91% to 73.04% and from 71.83% to 74.95%, respectively. Our analyses show that failed trajectories contain reasoning worth preserving, that repair success does not imply learning value, and that independent feedback is key to sustained improvement.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 14 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Solid empirical paper with a clear dual-use concern in the feedback calibration—deserves review with reproducibility conditions. This line is wrong

2 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at §3.2.4, Appendix A.4

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The learnability gate thresholds (D_repair in [0.35,0.85], Δ_repair in [0.02,0.30]) are calibrated on the 5,000-question feedback set. The same feedback set is used for measuring capability gaps and guiding question selection across rounds (§3.3.2, Eq. 14-15). This dual role creates an unmeasured risk: threshold tuning may exploit peculiarities of the feedback set, and the subsequent selection mechanism, driven by the same set, may compound those peculiarities by preferentially retrieving questions whose repairs the calibrated thresholds admit.

more of objection 1

The paper reports no split-set validation (e.g., calibrate on 60% of the feedback set and evaluate threshold stability on the held-out 40%) and no sensitivity sweep (e.g., shifting both bounds by ±0.05). Given that removing feedback causes accuracy to fall below the R1 baseline (Table 5), the feedback mechanism is load-bearing; therefore, the dual-use risk is also load-bearing. The authors acknowledge that thresholds are 'calibrated on the feedback set' but do not test their robustness. This gap should be addressed either by additional experiments or by an explicit discussion of why the risk is limited in this setting.

The paper says, at §5.4, Fig. 5a

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.96):

“Evaluating open-source auditors and separating the effects of trajectory length from question difficulty would clarify these dependencies.”

Pith says: Objection 2 of 2 · major

The trajectory-selection experiment compares the 'longest-8k' questions with a random-8k subset. The longest-8k set has lower repair success but yields higher downstream accuracy. This is presented as evidence that repair success and learning value differ. However, the comparison conflates trajectory length with question difficulty or other confounds.

more of objection 2

The paper acknowledges this in §7.1: 'it does not isolate trajectory length from question difficulty.' While this caveat is noted, the central claim of the experiment—that 'learning-based trajectory selection' is favored—would be strengthened by controlling for difficulty or by comparing questions of similar difficulty that differ in trajectory length. As written, the result is suggestive but not conclusive.

5 minor objections

The paper says, at Table 7

Pith says: minor objection 1 of 5

The human audit reports Cohen's kappa and bootstrap CIs but does not state the total number of trajectories audited. Please add the sample size to the table caption or the main text.

The paper says, at §4.3, Table 3

Pith says: minor objection 2 of 5

The right panel of Table 3 reports 95% bootstrap CIs. State explicitly that these are percentile intervals or the bootstrap method used (e.g., 'percentile bootstrap with 10,000 resamples').

show 3 more

The paper says, at Fig. 6d

Pith says: minor objection 3 of 5

The 'Heuristic' baseline (HHD-inspired hint recovery) is compared with MedEvo across difficulty levels. The caption notes that Heuristic outperforms MedEvo at L5, but the main text in §5.4 only mentions that MedEvo exceeds Heuristic at L1–L4. This asymmetry should be discussed: the hardest questions (L5) may benefit from an HHD-style approach more than from MedEvo's current design.

The paper says, at Appendix D.3

Pith says: minor objection 4 of 5

The audit prompt is long and contains many detailed instructions. Consider providing a succinct summary of the key constraints (e.g., answer-blindness, evidence tracing) in the main text to improve readability.

The paper says, at §3.2.1

Pith says: minor objection 5 of 5

Equation (3) defines entropy with natural log, but the paper does not specify the base of the logarithm in Eq. (10) for the loss. Clarify that ℒ_t also uses natural log (nats) throughout.

What would settle it

Re-run Qwen3.5-9B with two variants: (a) threshold sensitivity: shift D_repair bounds by ±0.05 and Δ_repair bounds by ±0.01/±0.05 (e.g., [0.30,0.90], [0.40,0.80]; [0.01,0.35], [0.03,0.25])—report seven-benchmark mean for each variant. (b) Split validation: randomly split the 5,000 feedback questions into calibration (60%) and hold-out (40%); calibrate thresholds on the calibration split only; run three Rounds of MedEvo using these thresholds; report seven-benchmark mean and the hold-out-feedback-set accuracy. If the mean shifts by more than ±0.5 pp across any variant, the thresholds are a material source of variability.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 14 Oct 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic partly agrees 39 s · 43,125 tokens read, 4,346 written
  3. Editor wrote the verdict above 27 s · 44,087 tokens read, 2,360 written
  4. Referee major revision: 2 major objections, 5 minor 35 s · 43,586 tokens read, 3,275 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This is good work. MedEvo combines several known ideas—prefix preservation, self-training, hint-based distillation—and adds a medical knowledge structure (tag taxonomy and error-type diagnosis) to guide where to repair and what to learn next. The results are clear: mean accuracy on seven medical multimodal benchmarks goes from 65.57% to 69.12% on Qwen3.5-9B, with similar gains on 35B and 397B. The experimental design is thorough: bootstrap confidence intervals, three independent runs at 9B, a human audit with Cohen's kappa up to 0.94, and sensible ablations that show the contribution of each component. The results look reproducible, and the paper's insights (e.g., repair success != learning value) are useful.

The main novelty is the coupling of medical knowledge navigation (tag taxonomy and error-type diagnosis) with trajectory repair and feedback-driven data selection. Combining error diagnosis, prefix retention, and capability-gap feedback into one loop is new in the medical domain.

The main soft spot is that the learnability gate thresholds and the feedback-driven question selection both use the same 5,000-question feedback set. The paper calibrates the thresholds (D_repair in [0.35,0.85], Delta_repair in [0.02,0.30]) on that set and also measures tag-level error rates on it to guide question selection each round. This dual use creates a real risk: threshold tuning may exploit peculiarities of the feedback set, and the selection mechanism can then compound those peculiarities by preferentially retrieving questions whose repairs the calibrated thresholds admit. The paper reports no split-set validation or sensitivity sweep. The authors acknowledge the thresholds are calibrated on the feedback set but do not test robustness. It is a genuine weakness, though it does not undercut the headline gains—those are measured on held-out benchmarks. Still, the claim that the feedback mechanism is key is weakened without empirical defense of the threshold stability. Separately, the auditor is proprietary (GPT-5.6-sol) and code+data are not yet released, which limits immediate reproducibility. The paper itself states this limitation.

Overall this paper deserves a serious referee. I would recommend conditional acceptance contingent on release of code and data for independent verification. Adding a split-set validation or sensitivity analysis for the thresholds would strengthen the work, but should not block review.

The strongest objection

The paper uses one fixed 5,000-question feedback set for two distinct purposes: (1) calibrating the learnability gate thresholds D_repair, Δ_repair (§3.2.4, Appx A.4), which determine which repairs enter training, and (2) measuring capability gaps to guide question selection across rounds (§3.3.2, Eq. 14–15). This dual use creates an unmeasured risk: threshold tuning may exploit peculiarities of the feedback set, and the selection mechanism—driven by the same set—can then compound those peculiarities by preferentially retrieving questions whose repairs the calibrated thresholds admit. The paper reports no split-set validation (e.g., calibrate on 60%, evaluate threshold stability on held-out 40% of the feedback set) and no sensitivity sweep (e.g., shift both bounds by ±0.05). Without such checks, the headline gains on the seven held-out benchmarks could partly reflect thresholds that overfit the feedback set rather than genuinely selecting learnable repairs. The ablation in Table 5 shows that removing feedback causes accuracy to fall below the R1 baseline (67.22→66.67), confirming the centrality of the feedback mechanism; this makes the dual-use risk load-bearing. The authors acknowledge thresholds are 'calibrated on the feedback set' but do not test robustness, which is a concrete gap that a sensitivity analysis would close.

The reader identified threshold generalization as the weakest assumption; this concern agrees with that diagnosis but sharpens it by highlighting the dual role of the same feedback set (calibration + election) as a compounding factor the reader didn't explicitly discuss. The reader's CONDITIONAL ver dict already requires robustness checks; this stress-test provides a specific test to address the gap. Since the concern is squarely within the reader's identified weakest area and does not change the overall assessment, UNCHANGED is appropriate.

The referee's report

If the reported gains are robust, the paper makes several contributions. It demonstrates that preserving the valid prefix and providing a minimal, error-type-conditioned hint yields better downstream accuracy than full rewriting or generic hints. It shows that repair success and learning value can diverge, motivating learnability-based screening. It provides evidence that independent feedback (a fixed held-out set) is more effective than training-error feedback for guiding data evolution. The method scales to 35B and 397B models and outperforms teacher-distillation SFT on the same backbone. The experimental design is thorough: seven benchmarks, three independent runs, bootstrap confidence intervals, human annotation with inter-annotator agreement metrics, and component ablations. The paper also releases prompts, training configurations, and an explicit information-access matrix, supporting reproducibility.

Pith's reading of the paper (a machine's summary, not the authors')

The paper argues that common training methods for medical multimodal reasoning models discard valuable valid reasoning contained in a student's own failed trajectories, replacing them with teacher-written solutions. The authors propose MedEvo, which integrates a medical tag taxonomy and a 20-type error codebook to localize the first consequential error in a failed rollout, preserve the correct prefix, provide a minimal answer-free hint, and let the same student regenerate the erroneous suffix. Across multiple rounds, feedback on a held-out set identifies capability gaps across medical tags and error types, steering subsequent question selection and repair effort toward remaining weaknesses. The central thesis is that correcting a response and constructing useful supervision are different objectives, and that preserving valid reasoning, filtering repairs for learnability, and using independent feedback each meaningfully affect what a student learns.

What carries the argument. The central machinery is Knowledge Navigation, instantiated through a medical tag taxonomy (four axes: modality, anatomy, disease, difficulty) and an error codebook (four parent types with five subtypes each, totalling twenty). Within a trajectory, token-level entropy guides the auditor to a candidate window; the gold-blind auditor then identifies the first consequential error, selects a safe rollback boundary, and constructs a minimal answer-free hint by instantiating the recovery operation associated with the diagnosed error type. Across rounds, tag-level error rates and error-type distributions on a fixed feedback set determine question-selection priorities and repair-order rules. A learnability gate (screening ratios $D_{\text{repair}} \in [0.35, 0.85]$ and $\Delta_{\text{repair}} \in [0.02, 0.30]$) filters repairs for student-relative difficulty before admitting them as supervised fine-tuning targets.

Core claim. The core discovery is that failed medical reasoning trajectories contain substantial valid reasoning that can be salvaged, and that structured medical knowledge can navigate both within-trajectory repair and cross-round data evolution to produce consistent gains. MedEvo achieves this by using a gold-blind auditor (accessible to the paper only through a proprietary model) that diagnoses errors and supplies a recovery hint without ever seeing the reference answer; the student then regenerates only the erroneous suffix while keeping its own valid prefix verbatim. On Qwen3.5-9B, mean accuracy across seven medical multimodal benchmarks rises from 65.57% to 69.12%, surpassing teacher-distillation SFT (66.93%), and the improvement extends to 35B (73.04%) and 397B (74.95%) models. The analysis further shows that repair success does not imply learning value: trajectories that are harder to repair can yield larger downstream accuracy gains, and feedback from an independent held-out set, rather than from training errors, is essential for sustained improvement.

The premise it rests on. The learnability gate thresholds ($D_{\text{repair}} \in [0.35, 0.85]$ and $\Delta_{\text{repair}} \in [0.02, 0.30]$) are calibrated on the held-out feedback set before final evaluation; if these thresholds do not generalize to new data distributions or if the feedback set is not representative of the target benchmarks, the filtering of repairs may introduce bias that inflates the reported gains.

If the paper is right (Pith's inferences, not the authors')

Medical vision-language models can be improved without teacher-generated rationales or reinforcement learning, using only the student's own repaired trajectories and structured feedback.

Preserving the valid prefix of a failed trajectory and the student's own regeneration of the suffix yields better downstream accuracy than full rewriting with a teacher solution.

Feedback from an independent held-out set, rather than from training errors, prevents overfitting to the training distribution and generalizes better to unseen benchmarks.

The repair workload (number of questions requiring repair) decreases over successive rounds while benchmark accuracy increases, suggesting that feedback-driven selection effectively targets the student's remaining weaknesses.

The method's dependence on a proprietary auditor is a practical limitation; if open-source auditors can replicate the error diagnosis and hint construction, the approach becomes more broadly accessible and auditable.

The learnability gate thresholds, calibrated on a single 5,000-question feedback set, may not be robust across different distributions of medical images or question difficulties, and the paper does not empirically test sensitivity to these threshold choices.

The finding that lower repair success can yield higher downstream accuracy suggests that learning value and repairability are in tension: over-easy repairs may not teach the model much, while harder-to-fix errors signal deeper gaps that, once addressed, produce larger gains.

The method's framework, structured domain knowledge plus failure-guided data evolution, could transfer to other expert domains such as legal reasoning or engineering diagnosis where a taxonomy of tasks and error types is available.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Medical multimodal reasoning models are typically trained on teacher-generated rationales, discarding the valid reasoning in students' own failed trajectories, while general-domain correction methods offer little guidance for errors in image reading, anatomical context, or differential diagnosis. We propose MedEvo, which uses medical knowledge, namely a tag taxonomy and error-type diagnosis, to navigate both trajectory repair and data selection. For each failed rollout, a gold-blind auditor locates the first consequential error and provides a minimal, answer-free hint; the same student retains the valid prefix and regenerates the erroneous suffix. After each round, MedEvo identifies capability gaps across medical tags on a held-out feedback set and prioritizes related questions for new rollouts and repairs. With supervised fine-tuning (SFT) alone, MedEvo raises Qwen3.5-9B mean accuracy across seven benchmarks from 65.57% to 69.12% (47.40% to 54.50% on the multimodal track of MedXpertQA (MedXpertQA-MM)), surpassing teacher-distillation SFT (66.93%). It also raises Qwen3.5-35B and Qwen3.5-397B mean accuracy from 67.91% to 73.04% and from 71.83% to 74.95%, respectively. Our analyses show that failed trajectories contain reasoning worth preserving, that repair success does not imply learning value, and that independent feedback is key to sustained improvement.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-14

The authors' main claim, as the classifier picked it out 0.99

“Our analyses show that failed trajectories contain reasoning worth preserving, that repair success does not imply learning value, and that independent feedback is key to sustained improvement.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.61 · human participants 0.07 · controlled comparison 0.93 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Medical multimodal reasoning models are typically trained on teacher-generated rationales, discarding the valid reasoning in students' own failed trajectories, while general-domain correction methods offer little guidance for errors in image reading, anatomical context, or differential diagnosis. background 0.52 0.07 0.04 asserted 1.00
We propose MedEvo, which uses medical knowledge, namely a tag taxonomy and error-type diagnosis, to navigate both trajectory repair and data selection. contribution 0.81 0.77 0.07 asserted 0.99
For each failed rollout, a gold-blind auditor locates the first consequential error and provides a minimal, answer-free hint; the same student retains the valid prefix and regenerates the erroneous suffix. method 1.00 0.88 0.04 asserted 0.99
After each round, MedEvo identifies capability gaps across medical tags on a held-out feedback set and prioritizes related questions for new rollouts and repairs. method 0.99 0.94 0.06 asserted 0.97
With supervised fine-tuning (SFT) alone, MedEvo raises Qwen3.5-9B mean accuracy across seven benchmarks from 65.57% to 69.12% (47.40% to 54.50% on the multimodal track of MedXpertQA (MedXpertQA-MM)), surpassing teacher-distillation SFT (66.93%). result 1.00 0.96 0.16 asserted 1.00
It also raises Qwen3.5-35B and Qwen3.5-397B mean accuracy from 67.91% to 73.04% and from 71.83% to 74.95%, respectively. result 1.00 0.94 0.06 asserted 1.00
Our analyses show that failed trajectories contain reasoning worth preserving, that repair success does not imply learning value, and that independent feedback is key to sustained improvement. interpretation 0.94 0.95 0.76 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.93

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 5,958 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (7)
Figure 1 of the paper
Figure 1. Medical multimodal reasoning across model scales. Mean accuracy on seven benchmarks. MedEvo improves 9B/35B/397B Qwen3.5 models. B: billion parameters; Hindsight Hint Distillation (HHD) [1]. local recovery decisions and learning decisions reflecting the student’s remaining weaknesses. Medical tags locate a failure in its clinical context, while error types identify the reasoning operation that needs correction. We call this navigation because these signals direct a path from an observed failure to a targeted intervention, at two connected levels: within a trajectory and across…
Figure 2 of the paper
Figure 2. Local trajectory repair. A minimal, answer-blind hint guides the student to recover from a failed reasoning step while retaining its supported prefix. Localization and admission checks are detailed in § 3.2.1 and § 3.2.4. soning while distinguishing structural efficiency from token length. JOLT makes student compatibility explicit: even a higher-performing privileged teacher can induce an unhelpful student update, motivating joint teacher–student training with regularization toward the student’s current policy [20]. MedEvo diagnoses failures with a medical codebook, screening…
Figure 3 of the paper
Figure 3. Medical multimodal data. Top: modality and difficulty; bottom: examples. Electrocardiogram (ECG); magnetic resonance cholangiopancreatography (MRCP); positron emission tomography/computed tomography (PET/CT); magnetic resonance imaging (MRI). Examples include both anatomical images and physiological recordings. 3 Method Knowledge Navigation. We define Knowledge Navigation as a feedback-guided process that uses structured domain knowledge and failure diagnosis to choose where to intervene, how to recover, and what to learn next. MedEvo instantiates this process with a medical tag…
Figure 4 of the paper
Figure 4. Knowledge Navigation in MedEvo. (a) Medical tags organize questions. (b) The current student generates fresh rollouts. (c) Knowledge-guided trajectory navigation localizes errors and provides recovery hints through a gold-blind auditor; the student preserves the valid prefix and regenerates the suffix. (d) Admitted trajectories update the training set, retaining valid historical targets; the next model trains from 𝑀0. (e) Fixed-set feedback guides question selection for the next round. diagnosed disease. Across rounds, tag-level accuracy on the feedback set identifies capability…
Figure 5 of the paper
Figure 5. Selection, hints, and feedback (Qwen3.5-9B). (a) Longest-8k versus random-8k questions: lower repair success can yield higher downstream accuracy. (b) Hint-count ablation uses the same longest-8k subset and run configuration as (a), with five continuation attempts; 𝑘 = 1 reproduces its 66.78% single-hint baseline. Three hints perform best. (c,d) Training accuracy and seven-benchmark mean with training-error or fixed-set feedback. Feedback: 5,000 held-out questions. 9B 35B 397B Model size (log scale) 65 70 75 Seven-benchmark mean (%) 69.12 73.04 74.95 65.57 67.91 71.83 (a)…
Figure 6 of the paper
Figure 6. Scale, uncertainty, and difficulty. (a) Mean accuracy after R3 for 9B/35B and R2 for 397B. (b) Repair acceptance with/without entropy localization; pointwise 95% Wilson intervals use 45,000 attempts per setting. (c) Error rates and step counts by min–max-normalized step-entropy bin; the diagonal is 𝑦 = 100𝑥. (d) Accuracy on a modality-stratified 2,000-question subset of the feedback set; counts use the right axis. Heuristic is a training-free HHD-inspired hint-recovery baseline (Appendix A.6). Further analyses
Figure 7 of the paper
Figure 7. shows twenty subtypes across four families. Bars normalize rates by subtype R0 values; labels retain reported percentages. All subtypes are below R0 at R3. The knowledge-gap family has an R3/R0 ratio of approximately 0.89, compared with 0.72–0.75 for the other families, indicating a smaller relative reduction in the reported breakdown, conditional on a consistent counting denominator across rounds. Rates remain separate by subtype. V1 Missed key finding V2 Spurious finding V3 Location / laterality V4 Sequence / phase V5 Cross-image alignment 0 0.5 1.0 1.5 Reported rate / R0 ×0.68…

Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment