Pith. sign in

REVIEW 4 major objections 6 minor 27 references

Combining rule-based rewards with reinforcement learning sharpens AI chest X-ray impressions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:47 UTC pith:PTO3L2DE

load-bearing objection Useful application of GRPO and rule-based rewards to CXR report generation, but the missing SFT-only control means the paper cannot attribute gains to GRPO, and the evaluation lacks the statistics to back its claims. the 4 major comments →

arxiv 2607.18988 v2 pith:PTO3L2DE submitted 2026-07-21 cs.CV

DobicVLM: Aligning Chest X-Ray Report Generation with Clinically-Grounded Programmatic Rewards via Group Relative Policy Optimization

Thanni Adewuyi , Angelica Obayi , Andem Aniekan , Samuel Okoko , Angel Ezendu , Ephraim Usani , Ademide Animasaun , Philip Chibundu
show 5 more authors
Christian Maurice Mary Donald Essien Oluwaseun Odunsi Oluwasegun Oguntuase Abiodun Adereni
This is my paper
classification cs.CV
keywords chest X-ray report generationGRPOprogrammatic rewardsvision-language modelradiologyreinforcement learningclinical alignmentexpert evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that a chest X-ray report generator can be aligned to clinical standards without a learned reward model: it combines supervised fine-tuning with Group Relative Policy Optimization (GRPO) driven by four interpretable, rule-based reward components. On 69 blinded expert-reviewed cases, the resulting system, DobicVLM, achieves the highest impression accuracy (27.2%) and medical terminology score (86.5%) among Gemini 2.5 Flash, the base MedGemma-4B model, and itself, though the margin over the base model is narrow and comes with trade-offs in completeness and referral recommendations. The significance would be that transparent, reproducible reward functions can replace opaque neural reward models in resource-limited medical settings, and that the evaluation standard here is deliberately strict, with the low absolute acceptance rates indicating how far automated generation still is from radiologist-level performance.

Core claim

DobicVLM applies GRPO after supervised fine-tuning of MedGemma-4B, using a reward function that checks report structure via regex, anatomical checklist coverage, semantic overlap of the impression section against the reference, and a length penalty. In a blinded evaluation by four physicians on 69 real-world chest X-ray cases, DobicVLM achieves the best Accuracy of Impression (27.2% accepted) and Medical Terminology (86.5%) compared to Gemini 2.5 Flash (10.7% / 82.3%) and the base MedGemma-4B (26.3% / 83.2%), while reducing hallucinations relative to Gemini (65.1% vs 41.0%) but slightly regressing relative to the base model (68.8%). The authors interpret this as evidence that GRPO with progr

What carries the argument

The central mechanism is the programmatic reward function used in GRPO. GRPO is a critic-free reinforcement learning algorithm that generates a group of candidate reports per input, computes advantages from reward differences within the group, and updates the policy with a clipped objective. The reward combines structural adherence (regex-verified section headers), an anatomical checklist fraction, ROUGE-L F1 on the Impression section, and a soft length constraint, weighted 0.30/0.30/0.35/0.05. This replaces opaque neural reward models with explicit, auditable rules.

Load-bearing premise

The evaluation assumes an unstated, consistent mapping from 5-point Likert ratings to the reported acceptance percentages, and that 69 cases with variable numbers of raters are enough to distinguish a 0.9-percentage-point difference between DobicVLM and the base model.

What would settle it

A reproduction using the exact Likert-to-acceptance threshold, inter-rater agreement statistics, and a blinded evaluation on a larger multi-institution set would settle whether DobicVLM's impression-accuracy advantage over MedGemma-4B is real or within noise; a second concrete check: if the reported acceptance rates do not change monotonically with the mean Likert scores shown to raters, the dichotomization is suspect.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, GRPO with rule-based rewards is a viable alternative to neural-reward RLHF for aligning medical VLMs, lowering the barrier for resource-limited institutions.
  • The strict 27.2% acceptance rate calibrates expectations: even the best model requires mandatory human review before clinical use.
  • Deliberate reward weighting can trade completeness for accuracy, reducing verbose 'complete-looking' reports but also lowering appropriate-referral rates, suggesting reward design directly shapes clinical behavior.
  • The method may transfer to other structured reporting tasks, such as other imaging modalities, where ground-truth reports and explicit checklists exist.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported advantage over the base model (27.2% vs 26.3%) is small and may not survive a larger or more diverse evaluation; the paper does not supply inter-rater statistics or a precise Likert-to-acceptance mapping, so treating the 0.9-point gap as established is premature.
  • The hallucination regression relative to MedGemma hints at a reward-hacking risk: optimizing for impression overlap may encourage plausible but unsupported elaborations; explicit entity-level rewards could mitigate this.
  • A natural testable extension is to replace the semantic overlap metric with structured entity matching (e.g., extracting clinical entities from reports) to see whether impression accuracy and hallucination control both improve.
  • The referral drop suggests the reward function should include an explicit referral-quality component if the system is intended for triage support.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes DobicVLM, a chest X-ray report generation model built by supervised fine-tuning of MedGemma-4B followed by Group Relative Policy Optimization (GRPO) with a programmatic reward composed of structural regex verification, anatomical checklist coverage, ROUGE-L similarity to the reference impression, and a length penalty (Eqs. 3–6). Training uses approximately 1,000 private image–report pairs; evaluation uses four physicians rating 69 held-out cases on five-point Likert scales across five criteria. The paper reports DobicVLM outperforming Gemini 2.5 Flash on most criteria and narrowly exceeding base MedGemma on impression accuracy (27.2% vs 26.3%) and medical terminology (86.5% vs 83.2%), with lower completeness and referral scores.

Significance. If the empirical claims were robust, the paper would provide a useful demonstration that interpretable, rule-based rewards can replace neural reward models for clinical RL alignment in a resource-limited setting, and the explicit reward equations (3)–(6) are a concrete contribution. The evaluation by blinded physicians on real clinical cases is a strength. However, the current evidence is not sufficient to establish the central causal claim: the experimental design cannot separate the contribution of GRPO from SFT, and the statistics are not reported in a way that allows the 0.9-point headline gain to be distinguished from noise. The corrupted qualitative section further weakens the clinical plausibility argument.

major comments (4)
  1. [§4.1 and §5.1, Table 1] The design cannot identify GRPO's contribution. DobicVLM is trained with SFT followed by GRPO, while the only specialized baseline is base MedGemma-4B without SFT. The differences in Table 1 (27.2% vs 26.3%; 86.5% vs 83.2%) could be produced entirely by the 1,000-pair SFT stage. DobicVLM is also worse than base MedGemma on hallucination (65.1% vs 68.8%) and referral (30.9% vs 48.3%), which is consistent with GRPO having a mixed effect. Without a MedGemma+SFT-only arm or a GRPO-with-sham-reward control, the abstract's claim that the results 'demonstrate GRPO's value' is not supported. This is fixable within the manuscript's scope but is load-bearing.
  2. [§4.1, Eq. (7), Table 1] Acceptance percentages are never defined. The criteria are described as 5-point Likert scales, but the manuscript does not state the threshold used to dichotomize ratings into 'accepted' vs 'rejected', nor how mean Likert scores are converted. No confidence intervals, p-values, or inter-rater statistics are provided; n=276 total evaluations with a 'variable number of raters per case' is contradictory. With 69 cases, a 0.9-point difference in impression accuracy is within sampling noise. Report the raw rating distributions, the dichotomization rule, per-rater agreement (e.g., Cohen's/Fleiss' kappa or ICC), and bootstrap CIs for all entries in Table 1.
  3. [§5.3 and Figure 1] The qualitative analysis is unreadable. The section consists of escaped unicode sequences (e.g., '/uni00000033/uni00000055/...') rather than a real example, and the figure caption references content that does not appear. This removes the only qualitative support for the claim that outputs are clinically well-formed. The authors should either provide a legible example and figure or remove the qualitative claims from §1 and §5.1.
  4. [§5.2 and §6.2] The clinically meaningful trade-offs—report completeness 60.2% vs 62.7% and appropriate referral 30.9% vs 48.3% relative to base MedGemma—are attributed to reward weights (λ) and 'GRPO's exploration' without supporting analysis. The referral drop of 17.4 percentage points is particularly concerning for a system claimed to be clinically aligned, yet the explanation is speculative ('likely because R_overlap focuses on Impression'). An ablation varying λ or a component-ablation study is needed to substantiate the mechanism; otherwise these passages should be reframed as open questions in the limitations.
minor comments (6)
  1. [§3.2, Eq. (2)] The KL penalty notation is inconsistent: D_KL(πθ∥πref) is written with both πref and π_ref; define once.
  2. [§4.1] The statement 'n=276 total evaluations (69 cases×3 models×variable number of raters per case)' is internally inconsistent; 69×3×4=828, not 276. Please clarify how the 276 evaluations were obtained.
  3. [§3.3, Eq. (6)] ℓ_min and ℓ_max are given as [80,450] in the text, but the equation refers to ℓ_min, ℓ_max; define the values near the equation.
  4. [Table 1 caption] The caption says 'Percentages indicate acceptance rate' but §4.2 defines Score as mean Likert rating; reconcile.
  5. [§5.1] The phrase 'qualitative refinement' is not supported by quantitative evidence.
  6. [References] Reference [19] lacks a year/arXiv identifier; several entries (e.g., [24]) are incomplete. Typos: 'T raining Infrastructure' (heading §3.1), 'F aithfulness' (Eq. 5 heading), 'T able' (§5).

Circularity Check

0 steps flagged

No circularity: expert evaluation is independent of the programmatic rewards; the GRPO attribution gap is an experimental-design issue, not a derivation loop.

full rationale

I find no circularity under the defined patterns. The training objective (Eq. 2) is the standard GRPO loss, and each reward component (Eqs. 3–6) is an externally specified rule; the only learned-signal overlap is ROUGE-L used in Eq. 5 as a reward, but the headline results (Table 1) come from blinded expert Likert ratings, not from ROUGE-L or from the reward function itself. Thus no “prediction” reduces by construction to an input fit. The main weakness—that DobicVLM = MedGemma + SFT + GRPO is compared only with base MedGemma (no SFT-only arm)—undercuts the attribution “This demonstrates GRPO’s value” (Abstract, §1.1), but that is a confounding/experimental-design problem, not circularity: the reported numbers are independent observations. Likewise, the unstated Likert-to-acceptance-rate conversion (Table 1 vs §4.1–4.2) and missing inter-rater statistics are validity concerns, not circularity. §5.3 contains corrupted text, removing the only qualitative example, but the omission is a lack of support, not a self-referential derivation. References to GRPO, MedGemma, RadLE, etc. are external and are not used to presuppose this paper’s conclusion. No load-bearing self-citation chain is present.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The central empirical claim rests on hand-chosen reward weights, an incompletely specified anatomy checklist, and an unstated Likert-to-acceptance mapping. None of these are derived from data outside the paper; they are ad hoc choices that materially shape the reported trade-offs.

free parameters (6)
  • Reward weights λ = [0.30, 0.30, 0.35, 0.05]
    Chosen by hand for the total reward (Eq. total after Eq. 6); the paper explains the trade-offs in completeness/referral as consequences of these weights, so they directly control the central outcome.
  • Length bounds ℓ_min, ℓ_max = [80, 450] characters
    Arbitrary soft constraints in Eq. (6) that affect R_len.
  • Length penalty decay γ = 0.01
    Set in Eq. (6) for the soft length penalty.
  • Anatomical checklist A = unspecified term list (cardiac silhouette, costophrenic angles, lung fields, ...)
    Eq. (4); the content of A is not fully defined, and R_check is the fraction of these fixed terms present.
  • Likert acceptance threshold = unstated
    The evaluation converts 5-point Likert ratings into acceptance percentages (Table 1) but never defines the threshold; this is a free parameter of the evaluation that could change rankings.
  • GRPO group size G, clip ε, KL β = G=6, ε=0.2, β=0.01
    Standard GRPO hyperparameters (Eq. 2); set by the authors without sensitivity analysis.
axioms (6)
  • standard math GRPO objective (Eq. 2) is a valid policy-gradient update for this setting
    Taken from [19]; the paper applies it without re-derivation.
  • domain assumption MedGemma-4B and SigLIP encoder provide a suitable base for CXR interpretation
    The model is assumed to already possess basic image-text competence after SFT; no evidence for this specific domain is provided beyond downstream results.
  • domain assumption The 1,000 reference reports y* are clinically accurate ground truth
    Used as both the ROUGE-L reward target (Eq. 5) and as the standard for expert comparison; if the references contain errors, training and evaluation inherit them.
  • domain assumption Expert Likert ratings reflect diagnostic truth and are independent/blinded
    Section 4.1 states anonymization/randomization but gives no inter-rater reliability or formal blinding protocol; the acceptance percentages assume raters are interchangeable.
  • domain assumption The anatomy checklist A is clinically complete
    Eq. (4) scores completeness by coverage of a fixed term set; terms absent from A do not count, so completeness is defined by the authors' choice of list.
  • domain assumption ROUGE-L measures semantic faithfulness
    Eq. (5) uses ROUGE-L-F1 as the semantic reward; the paper itself acknowledges in §6.3 that this is an imperfect proxy.

pith-pipeline@v1.3.0-alltime-deepseek · 10514 in / 15084 out tokens · 129705 ms · 2026-08-01T13:47:16.301837+00:00 · methodology

0 comments
read the original abstract

Medical imaging is a cornerstone of diagnostics, yet automated chest X-ray report generation struggles with structural adherence, anatomical completeness, and semantic faithfulness. We introduce DobicVLM, a vision-language model combining supervised fine-tuning on MedGemma-4B with Group Relative Policy Optimization (GRPO) and clinically-grounded programmatic rewards. Our approach uses interpretable, rule-based reward components; structural verification, anatomical checklist, semantic similarity, and length constraints to enforce clinical standards without neural reward models. Trained on 1,000 de-identified image-report pairs from a private clinical dataset (with ethics approval and compliance to local regulations), DobicVLM is evaluated via blinded expert review on 69 held-out cases. DobicVLM outperforms Gemini 2.5 Flash across the majority of criteria, achieving the highest impression accuracy (27.2%) and medical terminology (86.5%) compared to both Gemini 2.5 Flash and MedGemma 4B baselines, with minor trade-offs in completeness and referrals. This demonstrates GRPO's value for transparent alignment in resource-limited settings. Keywords: Vision-Language Models, Radiology Report Generation, Reinforcement Learning, Medical AI, GRPO

Figures

Figures reproduced from arXiv: 2607.18988 by Abiodun Adereni, Ademide Animasaun, Andem Aniekan, Angel Ezendu, Angelica Obayi, Christian Maurice, Ephraim Usani, Mary Donald Essien, Oluwasegun Oguntuase, Oluwaseun Odunsi, Philip Chibundu, Samuel Okoko, Thanni Adewuyi.

Figure 1
Figure 1. Figure 1: Qualitative demonstration of DobicVLM. The model output demonstrates alignment with the ground truth radiologist impression, correctly identifying the nor￾mal cardiac silhouette and clear lung fields [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 8 linked inside Pith

  1. [1]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

    Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., et al.: Openflamingo: An open-source framework for training large autoregressive vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

    2023

  2. [2]

    arXiv preprint arXiv:2308.12966 (2023)

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 (2023)

    Pith/arXiv arXiv 2023

  3. [3]

    arXiv preprint arXiv:2212.08073 (2022)

    Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al.: Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 (2022)

    Pith/arXiv arXiv 2022

  4. [4]

    arXiv preprint arXiv:2512.16145 (2025)

    Bassi, P.R.A.S., et al.: Mrg-r1: Reinforcement learning for clinically aligned medical report generation. arXiv preprint arXiv:2512.16145 (2025)

    arXiv 2025

  5. [5]

    In: ACL (2021)

    Chen, Z., et al.: Cross-modal memory networks for radiology report generation. In: ACL (2021)

    2021

  6. [6]

    arXiv preprint arXiv:2509.25559 (2025)

    Datta, S., Buchireddygari, D., et al.: Radiology’s last exam (radle): Benchmarking frontier multimodal ai against human experts and a taxonomy of visual reasoning errors in radiology. arXiv preprint arXiv:2509.25559 (2025)

    arXiv 2025

  7. [7]

    In: International Conference on Learning Representations (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. In: International Conference on Learning Representations (2022)

    2022

  8. [8]

    In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) (2021),https://arxiv.org/abs/2106.14463

    Jain, S., Agrawal, A., Saporta, A., Truong, S.Q.H., Du, D.N., Bui, T., Chambon, P., Zhang, Y., Lungren, M.P., Ng, A.Y., Langlotz, C.P., Rajpurkar, P.: Radgraph: Extracting clinical entities and relations from radiology reports. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) (2021),https://arxiv...

    Pith/arXiv arXiv 2021

  9. [9]

    arXiv preprint arXiv:2306.00890 (2023)

    Li, C., et al.: Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890 (2023)

    Pith/arXiv arXiv 2023

  10. [10]

    In: Findings of ACL (2022)

    Li, H., et al.: Reinforced cross-modal alignment for radiology report generation. In: Findings of ACL (2022)

    2022

  11. [11]

    CVPR (2021)

    Liu, F., et al.: Exploring and distilling posterior and prior knowledge for radiology report generation. CVPR (2021)

    2021

  12. [12]

    arXiv preprint arXiv:2403.02469 (2024)

    Messmer, M., et al.: Vision-language models for medical report generation and visual question answering: A review. arXiv preprint arXiv:2403.02469 (2024)

    Pith/arXiv arXiv 2024

  13. [13]

    In: Pro- ceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2021)

    Miura, Y., Zhang, Y., Tsai, E., Langlotz, C.P., Jurafsky, D.: Improving factual completeness and consistency of image-to-text radiology report generation. In: Pro- ceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2021)

    2021

  14. [14]

    Nature (2023)

    Moor, M., et al.: Foundation models for generalist medical ai. Nature (2023)

    2023

  15. [15]

    NAACL (2023)

    Nooralahzadeh, F., et al.: Knowledge-driven encode, retrieve, paraphrase for med- ical image report generation. NAACL (2023)

    2023

  16. [16]

    In: Advances in Neural Information Processing Systems (NeurIPS) (2022)

    Ouyang, L., et al.: Training language models to follow instructions with human feedback. In: Advances in Neural Information Processing Systems (NeurIPS) (2022)

    2022

  17. [17]

    Advances in Neural Information Processing Systems36(2023)

    Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., Finn, C.: Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems36(2023)

    2023

  18. [18]

    arXiv preprint arXiv:2507.05201 (2025)

    Sellergren, A., et al.: Medgemma technical report. arXiv preprint arXiv:2507.05201 (2025)

    Pith/arXiv arXiv 2025

  19. [19]

    arXiv preprint (2024)

    Shao, R., et al.: Group relative policy optimization. arXiv preprint (2024)

    2024

  20. [20]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Tanida, T., M¨ uller, P., Kaissis, G., Rueckert, D.: Interactive and explainable region- guided radiology report generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7433–7442 (2023)

    2023

  21. [21]

    In: arXiv preprint arXiv:2403.08295 (2024)

    Team, G., et al.: Gemma: Open models based on gemini research and technology. In: arXiv preprint arXiv:2403.08295 (2024)

    Pith/arXiv arXiv 2024

  22. [22]

    arXiv preprint (2024)

    Tu, T., et al.: Capabilities of gemini models in medicine. arXiv preprint (2024)

    2024

  23. [23]

    arXiv preprint arXiv:2404.00287 (2024)

    Wang, Y., Chen, W., Li, Z.: Medical-dpo: Aligning medical vision-language models via direct preference optimization. arXiv preprint arXiv:2404.00287 (2024)

    arXiv 2024

  24. [24]

    In: Findings of EMNLP (2025)

    Yan, A., et al.: Medivlm: A vision language model for radiology report generation. In: Findings of EMNLP (2025)

    2025

  25. [25]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Yu, F., Endo, M., Krishnan, R., Pan, I., Tsai, A., Reis, E.P., Fonseca, E.K.U.N., Lee, H.M.H., Zaharchuk, G., Patel, B.N., et al.: Evaluating the clinical accuracy of radiology report generation models. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 14666–14674 (2023)

    2023

  26. [26]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11975–11986 (2023)

    2023

  27. [27]

    arXiv preprint arXiv:2305.17100 (2023)

    Zhang, Y., He, Y., Xu, R., Wang, J., Liu, Q., Liu, H.: Biomedgpt: A uni- fied and generalist biomedical generative pre-trained transformer. arXiv preprint arXiv:2305.17100 (2023)

    Pith/arXiv arXiv 2023