Pith. sign in

REVIEW 4 major objections 5 minor 4 references

Medical AI models are 'geometrically blind,' and this paper claims that reinforcement learning on three self-generated spatial puzzles — scale, jigsaw, anomaly — cures it, beating leading models by over 40% on a new perception benchmark.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:08 UTC pith:PD3XRNFI

load-bearing objection A thoughtful method and a genuinely useful pilot study, but the headline 40% result is measured with a benchmark that is effectively the training reward on an in-distribution subset, so the central claim of 'curing geometric blindness' is not supported by the primary evidence. the 4 major comments →

arxiv 2601.23220 v2 pith:PD3XRNFI submitted 2026-01-30 cs.CV cs.AI

Med-Scout: Curing MLLMs' Geometric Blindness in Medical Perception via Geometry-Aware RL Post-Training

Anglin Liu , Ruichao Chen , Yi Lu , Hongxia Xu , Jintai Chen This is my paper
classification cs.CV cs.AI
keywords geometric blindnessmultimodal large language modelsmedical image perceptionreinforcement learningGRPOdense geometric rewardspatial groundingradiology VQA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper targets a specific failure mode it calls geometric blindness: multimodal large language models (MLLMs) describe medical images fluently but get spatial facts wrong — missing a lesion when it appears in the full view after being found in a crop, failing to update positions when the image is rotated, and ignoring obvious pasted-in structural artifacts. The authors argue the cause is training objectives that reward linguistic plausibility rather than geometric fidelity, and that the cure is reinforcement learning on verifiable geometric proxy tasks built from unlabeled images: hierarchical scale localization, topological jigsaw reconstruction, and anomaly consistency detection, scored by a dense (continuous) reward instead of pass/fail. The paper reports that this post-training raises average scores on its new Med-Scout-Bench from roughly 30–40% to 70–90% across four open-source backbones, surpassing proprietary GPT-5 and Gemini-3-Flash, and that the gains transfer to standard radiological VQA and report generation. The claim matters because if correct, it offers an annotation-free route to spatial grounding in clinical AI, addressing a core reliability gap.

Core claim

On its own terms, the paper discovers that geometric blindness in medical MLLMs is measurable and curable: in a pilot study models lose over 20% accuracy when lesions found in crops must be found again in global views, ignore a 180° rotation in 80% of cases, and miss pasted-in structural artifacts in over 90%. Med-Scout converts these deficits into three machine-verifiable tasks — scale-level and bounding-box prediction for local patches, reordering 2×2 shuffled quadrants, and locating a foreign patch in a 4×4 grid — and optimizes them with GRPO under a dense geometric reward. After training on ~97k samples, all four backbones improve from roughly 30–40% to 70–90% on Med-Scout-Bench, and six

What carries the argument

The load-bearing mechanism is the Dense Geometric Reward (DGR) inside the GRPO reinforcement loop: total reward R = R_acc + R_fmt + I_CoT·R_reason, where R_acc is continuous per task — average IoU over predicted patch boxes for scale localization, element-wise index match for jigsaw, and an exponential Euclidean-distance decay for anomaly location — so the policy earns partial credit for near-correct answers instead of a binary 0/1. This dense gradient, applied through three complementary proxy tasks, is what the paper claims steers models from semantic mimicry to spatial grounding.

Load-bearing premise

The headline improvement is computed on a benchmark carved from the same 108k-sample perturbation pool used for training and measured with the very reward function that drives the RL updates, so the 40% number chiefly shows that the model fits its training objective; the claim that this is a general cure for geometric blindness rests on the smaller external-benchmark gains.

What would settle it

Take Med-Scout-trained models and a matched baseline, apply geometric perturbations from a different generator (different grid granularities, unseen modalities, novel distortion types such as organ transposition), and have expert radiologists score the outputs instead of the DGR reward: if the advantage evaporates, or if the factual-vs-counterfactual energy gap fails to widen on new report pairs, the cure does not generalize and the 40% is a training-loop artifact.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If geometric blindness is cured as claimed, open-source MLLMs post-trained with Med-Scout can surpass proprietary models like GPT-5 and Gemini-3-Flash on geometric perception without any expert annotation.
  • The three proxy tasks together — removing any one degrades results — provide a recipe for instilling spatial awareness that transfers to radiological VQA and report generation.
  • Dense geometric rewards outperform sparse binary rewards on external benchmarks, and RL generalizes where SFT overfits, so reward design is central to the claimed cure.
  • The widened energy gap between factual and spatially-inverted report pairs indicates the model's probability mass shifts away from geometric hallucinations, a property relevant to clinical reliability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • We infer that the headline 40% gain measures within-distribution optimization, since Med-Scout-Bench is a 10% sample of the same perturbation pipeline used for training and is scored by the identical reward function; the genuinely out-of-distribution evidence is the smaller but consistent gains on external VQA benchmarks.
  • If the mechanism is what the paper says, the same proxy-task template — scale localization, layout reconstruction, anomaly detection — can be applied to other structured-imaging domains such as pathology whole-slide images (where magnification changes are natural scale tasks) or ultrasound, using only unlabeled data.
  • A direct testable extension: apply Med-Scout to a much larger backbone (70B+) and to held-out modalities; the paper predicts the geometric-alignment principles are model-size agnostic and modality-general.
  • One could also probe whether the energy-gap analysis holds for adversarial perturbations beyond left-right spatial inversions, e.g., organ swaps or scale distortions, which would tell whether the model learned general geometric constraints or narrow positional priors.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Med-Scout, a reinforcement-learning post-training framework aimed at reducing 'geometric blindness' in medical MLLMs. It defines three automatically generated proxy tasks—hierarchical scale localization, topological jigsaw reconstruction, and anomaly consistency detection—and optimizes models with a dense geometric reward (DGR) via GRPO. The authors also release Med-Scout-Bench, a 10,800-sample benchmark constructed from the same 108,000-case pool used for training. They report that Med-Scout improves internal benchmark scores by over 40% relative to baselines and yields consistent, albeit smaller, gains on radiological VQA, medical VQA, and report generation benchmarks. The paper includes ablations on task difficulty, reward type, proxy task composition, and an SFT-vs-RL comparison.

Significance. If the external gains are robust, the proposed framework is a promising low-cost approach to improving geometric grounding in medical MLLMs without expert annotations. The three proxy tasks and the dense reward design are reasonable and the authors provide a useful open-source benchmark and a detailed appendix. The paper also contains a valuable negative result: SFT can match internal benchmark scores without transferring to external tasks, which the authors correctly interpret as evidence that internal scores alone do not measure generalization. However, the central quantitative claim—that Med-Scout 'cures' geometric blindness—is currently supported primarily by an in-distribution metric that is exactly the training reward, and the external evidence is modest and without statistical confidence intervals. The validity of the benchmark as an independent measure needs to be established before the headline claim can be accepted.

major comments (4)
  1. [§4.3, §5.2, Appendix A.1] The headline improvement of over 40% is measured on Med-Scout-Bench, which is a 10% random subset of the same 108,000-case pool used for RL training, and the DGR score is computed by the same reward functions (Eqs. 2–5) optimized during training. This makes the primary result a measure of reward optimization on the training distribution, not an independent test of geometric competence. The paper's own SFT-vs-RL comparison in Appendix C.7 (Tables 10–11) demonstrates the danger: SFT reaches internal scores comparable to RL (e.g., 82.2 vs. 84.4 for Qwen3-VL-4B) yet shows no external gains. The authors should either construct a benchmark with source images disjoint from training and a metric not used as a reward, or substantially temper the 'curing geometric blindness' claim to focus on the external generalization results.
  2. [§5.3, Tables 1–2] The external benchmark gains, which are the only independent evidence for generalization, are modest (most improvements are between 0.2 and 4.2 points) and are reported without error bars, multiple seeds, or significance tests. Given the small magnitudes, some gains (e.g., Lingshu-7B on SLAKE, +0.2; on MIMIC-CXR CIDEr, +0.3) may be within run-to-run variability. The authors should provide confidence intervals or a statistical comparison (e.g., bootstrap or paired tests) across multiple seeds to support the claim of consistent and meaningful generalization.
  3. [§5.6, Figure 5 (Right)] The authors use the positive correlation between Med-Scout-Bench scores and external benchmark accuracy to argue that the internal benchmark is a 'reliable indicator' of broader clinical visual reasoning. However, since the Med-Scout-Bench score is the training reward, this correlation is largely a reflection of training progress. It does not establish benchmark validity. The SFT results in Appendix C.7 directly contradict the interpretation: SFT achieves high internal scores without external improvement, so the correlation is not a reliable indicator when the model is not optimized against the reward. The authors should validate the benchmark using models that did not train on it, or at least discuss this confound.
  4. [Appendix D] The section titled 'Proof of True Grounding via Energy Landscapes' claims that an increased energy gap between factual and counterfactual reports (Gap ≈ 0.69 vs. 0.06) is a 'theoretical certificate' that Med-Scout internalized geometric constraints. This is not a proof; it is an empirical observation on a probe set of 800 examples, with no statistical analysis or discussion of how the probe set relates to training data. The theoretical formalism (Eqs. 8–9) describes an ideal behavior but does not by itself certify the trained model. The authors should reframe this as a qualitative analysis and provide proper error bars or a larger probe set, or remove the 'certificate' language.
minor comments (5)
  1. [Algorithm 3 vs §A.2.3] In §A.2.3 the reference slice offset is described as 'z±5', but Algorithm 3, line 8, reads 'I.z±1'. Please make the values consistent.
  2. [§3 Pilot Study] The pilot study is based on 200 filtered samples per task, but the selection criteria and the exact prompts used are not described. Adding these details and confidence intervals would help the reader assess the robustness of the three findings.
  3. [§5.2 / Table 5] The benchmark scores are reported as 'scaled by a factor of 100' and the average is a simple mean across tasks. Consider reporting per-task standard errors, especially since Task C for several baselines is very low (e.g., 8.6–9.2), which makes the average sensitive to task composition.
  4. [§5.4 / Table 1] The 'Generalization' column of Table 1 includes benchmarks that are not purely radiological; consider renaming to 'Broader Medical VQA' for clarity.
  5. [Appendix B.3.2] The reward curves are visually described but not quantified. Reporting the converged reward values in a small table would be more precise than the qualitative statements.

Circularity Check

2 steps flagged

Med-Scout-Bench's DGR metric is the RL training reward on a same-pipeline subset; the headline >40% gain is therefore not independent evidence of curing geometric blindness.

specific steps
  1. self definitional [Section 4.2 (Eqs. 1-7), Section 4.3, Section 5.1 (Evaluation Metrics)]
    "For Med-Scout-Bench, we report the DGR score computed directly via the reward functions defined in Section 4.2. ... The total reward R is composed of three components: R=R acc +R fmt +I CoT · Rreason (1)"

    The benchmark metric (DGR) is, by the paper's own definition, the same dense reward optimized by GRPO during training (Eqs. 2-7). Therefore the reported gains on Med-Scout-Bench (e.g., Qwen3-VL-8B from 39.7 to 83.6) measure how well the model learned to maximize its training objective on a held-in-distribution subset; they do not independently establish a general geometric faculty. The claim 'Med-Scout significantly mitigates geometric blindness' rests on a score that is the training reward by construction.

  2. other [Section 4.3; Appendix A.1; Appendix C.7 (Tables 10-11)]
    "From this pool, we sampled a high-quality subset of 10,800 cases (10%) as the benchmark. ... The rest of the data used for training and validation (N=97,200) follows the same pattern, ensuring that the way the model is trained matches the way it is tested. ... SFT achieves performance levels comparable to or even surpassing the RL-tuned models. ... SFT merely overfits to the proxy task patterns without internalizing the underlying geometric reasoning."

    The benchmark is defined as a 10% random subset of the same 108,000-case pool (TotalSegmentor/MIMIC-CXR) and is generated by the same perturbation algorithms used to create the RL training data. Hence Med-Scout-Bench is an in-distribution evaluation of the training distribution and task templates, not an independent definition of geometric perception. The paper's own SFT-vs-RL comparison confirms this: SFT reaches 82.2 average on Med-Scout-Bench while yielding negligible or negative gains on external benchmarks, showing the internal DGR score is not a valid proxy for curing geometric blindness.

full rationale

The central derivation chain contains a genuine circular element in the primary evaluation: Med-Scout-Bench reports the DGR score, and DGR is exactly the dense reward function that the GRPO training maximizes (Section 4.2, Eqs. 1-7; Section 5.1). The benchmark is also carved from the same 108k VQA-case pool and same generation pipeline as the training data (Section 4.3, Appendix A.1), so the headline 'over 40%' improvement largely reflects optimization of the training objective on a same-distribution subset. The paper itself supplies strong evidence that this internal score is not sufficient: SFT achieves near-RL DGR scores on Med-Scout-Bench but no external transfer (Appendix C.7, Tables 10-11). However, the paper does not rely solely on the internal benchmark: external radiological VQA, general medical VQA, and report-generation gains (Tables 1-2, Sections 5.3-5.4), plus the energy-landscape and attention analyses (Section 5.7), provide partially independent evidence that the training transfers. The external gains are modest (mostly 0.2-4.2 points), unreplicated, and reported without significance tests, so they temper but do not eliminate the circularity. No load-bearing self-citation chain, imported uniqueness theorem, or ansatz-via-citation pattern was found. Accordingly, the score is 6 rather than 8: one or more central 'predictions' (the Med-Scout-Bench scores) reduce by construction to the training reward, but independent external results keep the paper from being wholly tautological.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim rests on the assumption that the three proxy tasks and the reward function define the correct notion of geometric blindness, and that the benchmark measures generalization rather than optimization. Free parameters are all hand-set; no sensitivity analysis is provided. The benchmark's in-distribution nature is the key circularity.

free parameters (4)
  • Anomaly temperature τ (Eq. 5) = 0.1
    Controls the softness of the Euclidean distance reward for anomaly detection; chosen by hand, no sensitivity analysis.
  • Reward caps R_fmt and R_reason = 0.5 each
    Hand-set weights in the total reward R = R_acc + R_fmt + I_CoT * R_reason; affect optimization balance.
  • Scale ratios {0.20, 0.0625} and crop count N=3 = 0.20, 0.0625, N=3
    Defines task difficulty; chosen without empirical justification.
  • ROI bounds [0.2, 0.8] = 0.2, 0.8
    Restricts crop sampling to central region; hand-chosen to avoid background.
axioms (5)
  • domain assumption The three proxy tasks (scale, topology, anomaly) adequately capture the geometric constraints that matter for medical perception.
    Section 4.1 defines these as the intrinsic geometric logic; no medical expert validation of coverage.
  • domain assumption TotalSegmentator and MIMIC-CXR provide accurate anatomical ground truth.
    Used to build the dataset; errors in segmentation propagate to labels.
  • domain assumption LLM-as-a-Judge with Gemini-3-Flash reliably evaluates open-ended medical VQA responses.
    Section 4.3 and 5.1 rely on LLM judgment for semantic correctness without validation against human experts.
  • domain assumption The energy function E(x,y) = -log P(y|x) is a valid measure of geometric grounding.
    Appendix D uses this to 'prove' grounding; no formal justification that energy separation reflects geometric truth.
  • domain assumption The 10% benchmark subset is representative of the full dataset distribution.
    Required for the benchmark to be meaningful, but the subset comes from the same task templates as training, making it an in-distribution test.

pith-pipeline@v1.3.0-alltime-deepseek · 26443 in / 8096 out tokens · 90672 ms · 2026-08-03T06:08:01.677514+00:00 · methodology

0 comments
read the original abstract

Despite recent Multimodal Large Language Models (MLLMs)' linguistic prowess in medical diagnosis, we find even state-of-the-art MLLMs suffer from a critical perceptual deficit: geometric blindness. This failure to ground outputs in objective geometric constraints leads to plausible yet factually incorrect hallucinations, rooted in training paradigms that prioritize linguistic fluency over geometric fidelity. This paper introduces Med-Scout, a novel framework that "cures" this blindness via Reinforcement Learning (RL) that leverages the intrinsic geometric logic latent within unlabeled medical images. Instead of relying on costly expert annotations, Med-Scout derives verifiable supervision signals through three strategic proxy tasks inspired by the systematic reading and reasoning patterns of clinicians: Hierarchical Scale Localization, Topological Jigsaw Reconstruction, and Anomaly Consistency Detection. To rigorously quantify this deficit, we present Med-Scout-Bench, a new benchmark specifically designed to evaluate geometric perception. Extensive evaluations show that Med-Scout significantly mitigates geometric blindness, outperforming leading proprietary and open-source MLLMs by over 40% on our benchmark. Furthermore, this enhanced geometric perception generalizes to broader medical understanding, achieving superior results on radiological and comprehensive medical VQA tasks.

Figures

Figures reproduced from arXiv: 2601.23220 by Anglin Liu, Hongxia Xu, Jintai Chen, Ruichao Chen, Yi Lu.

Figure 1
Figure 1. Figure 1: Pilot Study: (Left) Scale Blindness: The model correctly describes findings in a local crop but fails in the full global view. (Middle) Topology Blindness: When the image is rotated, the model fails to update location descriptions. (Right) Anomaly Blindness: The model completely overlooks obvious artificial modifications. Med-Scout framework to actively cure these blind spots, thereby grounding semantic ge… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the Med-Scout Framework. We transform unlabeled medical images into three proxy tasks to cure geometric blindness actively. The framework is optimized using GRPO with a Dense Geometric Reward mechanism that provides stable feedback. violate anatomical coherence. Unified VQA Stream & Optimization. We unify these tasks into a standard open-set VQA format as shown in Appendix A.3 and require the m… view at source ↗
Figure 3
Figure 3. Figure 3: Performance comparison on Med-Scout-Bench. and SLAKE (Liu et al., 2021), alongside MIMIC-CXR and IU-Xray for report generation. Finally, to verify broad gen￾eralization, we extend our evaluation to PMC-VQA (Zhang et al., 2023c), OmniMedVQA (Hu et al., 2024), and MedX￾pertQA (Hu et al., 2024) for broader medical domains and diverse modalities. Evaluation Metrics. For Med-Scout-Bench, we report the DGR score… view at source ↗
Figure 4
Figure 4. Figure 4: Performance comparison on six public benchmarks. Purple colors correspond to higher accuracy. Direct Mode (+M) and Reasoning Mode (+M (R)) show close performance. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Energy Landscape of Factual Consistency. We visual￾ize energy distributions for 800 factual (purple) versus spatially inverted (blue) report pairs. Left: Qwen3-VL-4B-Instruct exhibits overlapping distributions. Right: Qwen3-VL-4B-Instruct with Med-Scout establishes a distinct energy barrier. 5.7. Extensive Analysis We conducted an extensive analysis to evaluate the impact of Med-Scout on the spatial discri… view at source ↗
Figure 5
Figure 5. Figure 5: Data Scaling and Generalization Analysis. Left: Con￾tinuous performance improvement on Med-Scout-Bench with in￾creasing training data. Right: Strong positive correlation between Med-Scout-Bench scores and average accuracy on external bench￾marks [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of attention maps on critical anatomical regions. We compare the visual attention of the baseline Qwen3- VL-4B-Instruct (Middle) versus the model aligned with Med-Scout (Right) given specific anatomical queries. While the baseline model exhibits scattered attention and frequently drifts to irrelevant background noise, Med-Scout demonstrates a highly concentrated focus on the critical target r… view at source ↗
Figure 8
Figure 8. Figure 8: Data Statistics of Med-Scout-Bench. Left: The benchmark maintains a strictly balanced distribution across CT, MRI, and X-ray modalities to ensure unbiased geometric evaluation. Right: The benchmark is under task-specific distribution. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Unified VQA Instruction Examples of Direct Mode. A.4. Med-Scout-Bench Evaluation Pipeline To ensure consistent scoring across models with different styles, we use a standardized evaluation process as shown in [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: Since MLLMs often provide detailed reasoning that makes simple text matching difficult, we use DeepSeek￾V3.2 (DeepSeek-AI et al., 2025) to extract the conclusion from the raw output. These extracted answers are then automatically compared against the correct labels using a strict scoring script [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 10
Figure 10. Figure 10: Unified VQA Instruction Examples of Reasoning Mode. B. Training Implementation Details B.1. Hyperparameters We provide a comprehensive list of hyperparameters used during the Med-Scout RL post-training phase in [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: Training Reward Dynamics. The plots illustrate the optimization trajectories for Direct Mode (left two panels) and Reasoning Mode (right three panels). We report the Dense Geometric Rewards (Racc), Format Rewards (Rfmt), and Reasoning Structure Rewards (Rreason). • Format Reward (Rfmt): As shown in the Direct - Format subplot, all models rapidly master the requisite output format constraints (e.g., coordi… view at source ↗
Figure 13
Figure 13. Figure 13: Comparison of energy score landscapes for four MLLMs. The panels illustrate the enhanced separation between factual and reversed samples achieved by Med-Scout, resulting in significantly wider energy gaps across all models. E. Future Perspectives While Med-Scout demonstrates significant improvements in curing geometric blindness, we acknowledge certain limitations in our current study that point towards p… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

4 extracted references · 1 linked inside Pith

  1. [1]

    larger is better,

    Visual Input:We use special tokens (e.g., <image>) to represent the medical scans. Note that for theHierarchical Scale Localizationtask (Figure 9a), the input specifically supports multi-image sequences (Global View + Local Crops). 2.User Prompt:A structured instruction that clearly defines the geometric objective and constrains the output format. 3.Targe...

    2025

  2. [2]

    heart” patch must be spatially adjacent to and above the “stomach

    Coordinate Regression:Explicitly predict the normalized bounding box (x1, y1, x2, y2) of the patch in the original image frame. Algorithm 1Data Generation for Hierarchical Scale Localization Require:Original ImageI∈R H×W Require: Hyperparameters: 1:Num PatchesN←3 2:Scale RatiosS ← {0.20,0.0625} {Area ratios for Level 1 and Level 2} 3:ROI Bounds[α min, αma...

  3. [2024]

    URL https: //doi.org/10.48550/arXiv.2412.08737

    doi: 10.48550/ARXIV .2412.08737. URL https: //doi.org/10.48550/arXiv.2412.08737. Zhang, J., Khayatkhoei, M., Chhikara, P., and Ilievski, F. MLLMs know where to look: Training-free perception of small visual details with multimodal LLMs. InThe Thirteenth International Conference on Learning Rep- resentations, ICLR 2025, Singapore, April 24-28, 2025. OpenRe...

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.08737 2025

  4. [2025]

    Subsequently, we describe the rigorous algorithmic protocols employed to synthesize the three geometric proxy tasks

    and MIMIC-CXR (Johnson et al., 2019). Subsequently, we describe the rigorous algorithmic protocols employed to synthesize the three geometric proxy tasks. Finally, we present the unified VQA instruction templates used to standardize these tasks for effective RL post-training. A.1. Data Composition and Distribution Statistics We primarily analyze the Med-S...

    2019