Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HSCR turns a medical VLM's own visual-token dropout into preference data, improving alignment with 2,000 training entries.

desk verdict A genuinely new low-resource preference-optimization scheme for medical VLMs with consistent but modest gains, undermined by an unvalidated self-generated dispreferred data step and some evaluation hygiene issues. read the letter →

arxiv 2506.00805 v1 pith:PRBVRTWV submitted 2025-06-01 cs.CV cs.CL

Songtao Jiang , Yan Zhang , Yeying Jin , Zhihang Tang , Yangyang Wu , Yang Feng , Jian Wu , Zuozhu Liu This is my paper
classification cs.CVcs.CL
keywords medicalvision-languagemodelspreferenceoptimizationmodalityalignmenthallucinationmitigationvisualtokendropoutself-contrastiverewardingzero-shotVQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a way to align medical vision-language models without external preference data by mining the model's own misalignment. HSCR drops 70% of visual tokens, uses the resulting logit shifts to find hallucination-prone tokens, replaces them with low-contrast tokens to build dispreferred responses, and then optimizes a two-level preference loss. With only 2,000 training entries, it reports higher zero-shot accuracy than LLaVA-Med1.5 on three medical VQA benchmarks and lower hallucination rates in a general-domain test. The value of the claim is that in data-scarce medical settings, preference data can come from the model itself rather than from expensive annotators or larger models.

What carries the argument

The load-bearing mechanism is token-level self-contrastive rewarding. HSCR drops 70% of visual tokens before the LLM, compares the next-token distributions with and without the dropout, and scores each token by an implicit reward $P_{\mathrm{diff}}=\mathrm{Softmax}[(1+\beta)\logit_\theta(y\mid i,x)-\beta\logit_\theta(y\mid i',x)]$, where $i$ and $i'$ are the original and dropped visual tokens and $\beta$ controls contrast strength. Tokens with the largest shifts are treated as modality-coupled and replaced, in ascending order of the shifted distribution, with low-contrast tokens, yielding multiple dispreferred responses; a semantic-similarity re-ranking then orders them by distance to the ground-truth response, and the multi-level preference loss adds explicit terms for $(y_w,y_{l_j})$ pairs and implicit terms for every $(y_{l_j},y_{l_m})$ pair.

What would settle it

Take a fixed prompt set and construct HSCR's dispreferred responses; then construct a control set by replacing the same number of tokens at the same positions with random low-probability tokens, keeping everything else fixed. If the control matches HSCR's gains on Rad-VQA, SLAKE, and PathVQA, the logit-shift selection is not the active ingredient. A second check: ask clinicians to mark each replaced token as a factual error, harmless paraphrase, or arbitrary corruption; the method's premise requires the first category to dominate.

Watch

Extended reading notes

Core claim

The paper's central claim is that a medical VLM's own misalignment can be converted into its own training signal: visual-token dropout exposes which output tokens are tied to the image, replacing those tokens with hallucinated ones yields dispreferred responses the model already assigns high probability to, and a hierarchical loss over preferred, less-wrong, and more-wrong responses sharpens modality alignment. Evidence is reported as zero-shot gains over LLaVA-Med1.5 on Rad-VQA (+3.61 open, +3.51 closed), SLAKE (+2.87, +6.97), and PathVQA (+2.35, +4.42), plus lower hallucination rates on AMBER when the same recipe is applied to a general VLM.

Load-bearing premise

The pipeline assumes that tokens whose logits shift most after 70% visual-token dropout are genuinely hallucination-prone, and that replacing them with low-contrast tokens produces coherent dispreferred responses; if those replacements are just arbitrary or semantically broken tokens, the preference signal does not encode the intended alignment.

Editorial extensions

If this is right

  • With 2,000 self-generated preference entries, HSCR improves zero-shot open/closed accuracy over LLaVA-Med1.5 by +3.61/+3.51 on Rad-VQA, +2.87/+6.97 on SLAKE, and +2.35/+4.42 on PathVQA.
  • Ranking among dispreferred responses (implicit preference) contributes more than binary winner-vs-loser comparisons on closed-ended medical VQA, and combining the two is better than either alone.
  • Self-generated preferences outperform GPT-4o-generated preferences in the paper's ablations, and GPT-4o data can even hurt open-ended performance.
  • The effect transfers to a general-domain VLM: on AMBER, HSCR lowers the hallucination rate from 36.4 to 25.1 and improves object coverage relative to DPO.
  • A captioning and instruction-following evaluation shows gains from 2,000 HSCR entries that exceed the gains from scaling supervised fine-tuning data from 10k to 60k entries.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because HSCR needs only the base model and 2,000 entries, a natural testable extension is to run it iteratively, using each aligned model to generate the next round's preference data; the paper does not test this self-training loop.
  • The same logit-shift criterion could be applied at inference time as a decoding correction rather than only as a training-data generator, potentially reducing hallucinations without additional training.
  • The 70% dropout ratio, top-10 token count, and 0.1 semantic-similarity threshold are tuned on the tested models, leaving open whether the recipe transfers to other architectures or image modalities without re-tuning.
  • HSCR's token-replacement strategy suggests a broader principle: a model's sensitivity to input corruption can be repurposed as a reward signal, which may generalize to other modalities such as audio or video.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Hierarchical Self-Contrastive Rewarding (HSCR), a preference-optimization method for medical vision-language models. HSCR constructs dispreferred responses by applying 70% visual-token dropout, detecting modality-sensitive tokens via logit shifts, and replacing those tokens with low-contrast alternatives during decoding. These candidates are re-ranked by semantic similarity to the ground truth, and the model is trained with two DPO-style losses: an explicit loss that separates preferred from dispreferred responses and an implicit loss that ranks dispreferred responses against each other. Experiments on Rad-VQA, SLAKE, PathVQA, captioning/instruction-following, and AMBER show consistent gains over LLaVA-Med1.5 and a general-domain baseline, using only 2,000 training entries.

Significance. If the mechanism is valid, HSCR is a practically important contribution: it removes the need for external GPT-4o or human-annotated preference data, operates on a small training budget, and reports consistent improvements across medical VQA, captioning, and hallucination benchmarks. The code release and the consistency of the ablations (both explicit and implicit components contribute) are strengths. However, the central claim—that the self-generated dispreferred responses encode genuine modal misalignment—is not directly validated, and the experimental protocol does not rule out tuning on the evaluation sets. The significance of the work therefore hinges on the unresolved validation of the preference-construction step.

major comments (4)
  1. [Sec. 3.1, Eqs. (3)-(4)] The construction of dispreferred responses y_l is the load-bearing step of the method, but the manuscript provides no evidence that the replaced tokens are actually hallucinated or that the resulting y_l are fluent, coherent, and factually wrong. The text states that logit-shift-sensitive tokens "often are error-prone" and that low-contrast replacements "often correspond to hallucinated outputs," yet no quantitative or qualitative analysis is given. Without such validation, the preference signal used in Eqs. (5)-(6) may be optimizing against arbitrary token-level noise, and the reported gains could stem from regularization or length effects rather than from correcting modality misalignment. The authors should report examples of replaced tokens and complete y_l, evaluate y_l for fluency and image-grounded correctness (e.g., via clinician or automated checks), and compare the selected positions against known hallucination patterns.
  2. [Sec. 4.1, Fig. 3, Table 6] The hyperparameters—mask ratio 0.7, contrast strength beta, top-n, similarity threshold 0.1, and j=3—appear to be selected using the evaluation benchmarks themselves. For instance, Figure 3 and Table 6 report mask-ratio ablations in terms of final test accuracy on Rad-VQA, SLAKE, and PathVQA, with 0.7 chosen because it "consistently yields optimal performance." No held-out validation or seed-averaged results with error bars are reported, and the gains on individual benchmarks are as small as 0.3 points. This makes it difficult to assess whether the improvements are robust or an artifact of tuning on these specific test sets.
  3. [Sec. 3.3, Eq. (6)] The implicit preference loss assumes that the semantic-similarity ranking of dispreferred responses reflects meaningful quality differences in a medical context. The paper does not validate this assumption: it does not show that higher-ranked y_l are actually "less wrong" than lower-ranked y_l, nor does it demonstrate that the Corley-Mihalcea semantic similarity measure correlates with clinical correctness or answer accuracy. If the ranking is noisy, the implicit objective in Eq. (6) reinforces arbitrary distinctions among dispreferred responses rather than subtle alignment cues. The authors should provide a validation of the ranking, for example by correlating similarity scores with clinician judgments or with ground-truth answer accuracy on a held-out subset.
  4. [Sec. 5, Table 3] The comparison of preference-data construction is limited to GPT-4o-generated pairs. To substantiate the claim that HSCR's self-generated data is cost-effective and superior, the authors should compare against a DPO baseline trained on the same 2,000 examples using a standard hallucination-augmented construction (e.g., POVID-style noise injection or random negative sampling from the model's own generations). Additionally, VCD and LiPO are listed as baselines in Table 1, but they are not trained under the same budget or protocol, so the reported differences do not isolate the effect of HSCR's data construction and loss. Without such controlled comparisons, the advantage over simpler self-training baselines is not established.
minor comments (6)
  1. [Sec. 3.1-3.2] The notation is inconsistent: Sec. 3.1 introduces k dispreferred responses, Sec. 3.2 selects j responses, but Eq. (5) and the surrounding text refer to "all k dispreferred outputs." Please clarify whether the losses use k or j throughout.
  2. [Sec. 4.1] The paper says "we sample 2,000 entries from the dataset used in instruction-tuning stage" but does not specify which dataset (e.g., LLaVA-Med's SFT mix, PMC-15M, or a specific subset). This should be stated precisely for reproducibility.
  3. [Appendix A.6] The subsection is titled "Implementation Details of Figure 7," but it describes the AMBER experiments shown in Figure 6. The figure/table references should be corrected.
  4. [Figure 7] The axis labels in Figure 7 appear garbled (e.g., "(SPVOE 5SVUI3FTQPOTFT"), which seems to be an encoding artifact. The labels should be rendered as readable text.
  5. [Throughout] There are several typos and formatting issues, such as "Experiements Setup" in Sec. 4.1, repeated "Config" in Table 7's caption, and inconsistent spacing in "LLaV A-Med." A careful proofreading pass is needed.
  6. [Sec. 3.1, Eq. (4)] The product notation in Eq. (4) omits the dependence of each token probability on the previously generated tokens; writing Pdiff(y_t | y_{<t}, i, i', x) explicitly would make the autoregressive decoding clearer.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the preference signal is generated on-policy and evaluated on independent benchmarks.

full rationale

The paper's derivation chain is self-contained and empirically falsifiable. The preference data are constructed from the base model's own logit shifts under visual-token dropout (Eq. 3), then re-ranked by external semantic similarity (Sec. 3.2), and the model is optimized with DPO-style losses (Eqs. 5-7). This is on-policy self-training, not a circular derivation: the training signal is not the same quantity as the reported outcome. The reported zero-shot gains on RAD-VQA, SLAKE, PathVQA, and AMBER are held-out external benchmarks, not the 2,000 training entries used to construct preference pairs. No equation reduces to a fitted target, no fitted parameter is renamed as a prediction, and no load-bearing premise is justified by a self-citation chain. Self-citations appear only as related-work context (e.g., Jiang et al. 2024a) and do not carry the argument. The skeptical concern that the token-replacement heuristic may generate noisy or incoherent negatives is a validity/robustness question, not a circularity question, because the method's success remains independently testable.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several hand-set hyperparameters and two domain-specific assumptions about how logit shifts and semantic similarity relate to hallucination. No new physical entities are introduced.

free parameters (7)
  • contrast strength beta = 0.9
    Controls the contrast between full and dropout logits in Eq. (3); set by hand, no ablation reported.
  • visual token dropout ratio = 0.7
    Fraction of visual tokens dropped; selected for best performance among {0.3, 0.5, 0.7, 0.9} on evaluation benchmarks (Figure 3, Table 6).
  • top-n sensitive tokens = n = 10
    Number of modality-coupled tokens replaced; set by hand.
  • number of dispreferred responses = j = 3
    Number of dispreferred responses kept after similarity re-ranking; set by hand.
  • similarity difference threshold = 0.1
    Minimum semantic similarity gap to the preferred response for a dispreferred candidate to be kept; set by hand.
  • DPO temperature gamma = 0.1
    Temperature in preference losses Eqs. (5)-(6); chosen by hand.
  • training entry count = 2,000
    Number of instruction-tuning entries sampled for preference data; chosen by hand, but motivated by low-resource claim.
assumptions (5)
  • standard math DPO reward-policy link: reward is a log-ratio of policy to reference policy
    Assumed in Section 2 and used to write losses in Section 3.3.
  • ad hoc to paper Visual token dropout exposes inherent misalignment, and logit shifts identify hallucination-prone tokens
    Core premise of Section 3.1; no independent evidence that top-difference tokens are actually modality-coupled or hallucinated.
  • domain assumption Semantic similarity between responses reflects true quality ordering
    Used in Section 3.2 to re-rank dispreferred responses; semantic similarity is a proxy, not a verified quality measure.
  • ad hoc to paper Replacing sensitive tokens with low-logit-contrast tokens yields varying degrees of incorrectness
    Needed for the multi-level implicit preference loss; not directly validated.
  • domain assumption LoRA training with rank 16 is sufficient for preference optimization
    Training choice in Section 4.1, not justified by analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Models." pith.science (2026). https://pith.science/paper/PRBVRTWV

@misc{pith2026250600805,
  author       = {Pith},
  title        = {Pith review of: HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PRBVRTWV}},
  note         = {Machine review of arXiv:2506.00805}
}
read the original abstract

Medical Vision-Language Models (Med-VLMs) have achieved success across various tasks, yet most existing methods overlook the modality misalignment issue that can lead to untrustworthy responses in clinical settings. In this paper, we propose Hierarchical Self-Contrastive Rewarding (HSCR), a novel approach that addresses two critical challenges in Med-VLM alignment: 1) Cost-effective generation of high-quality preference data; 2) Capturing nuanced and context-aware preferences for improved alignment. HSCR first leverages the inherent capability of Med-VLMs to generate dispreferred responses with higher sampling probability. By analyzing output logit shifts after visual token dropout, we identify modality-coupled tokens that induce misalignment and derive an implicit alignment reward function. This function guides token replacement with hallucinated ones during decoding, producing high-quality dispreferred data. Furthermore, HSCR introduces a multi-level preference optimization strategy, which extends beyond traditional adjacent-level optimization by incorporating nuanced implicit preferences, leveraging relative quality in dispreferred data to capture subtle alignment cues for more precise and context-aware optimization. Extensive experiments across multiple medical tasks, including Med-VQA, medical image captioning and instruction following, demonstrate that HSCR not only enhances zero-shot performance but also significantly improves modality alignment and trustworthiness with just 2,000 training entries.

Figures

Figures reproduced from arXiv: 2506.00805 by the authors.

Figure 1
Figure 1. Two key challenges in preference optimization [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the HSCR pipeline, comprising token-level self-contrastive rewarding for data generation [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison with different mask ratios. Details in Appendix Table 10 0 10 20 30 40 10 0 10 20 10 5 0 0 5 10 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The x-axis represents log p(Ground Truth Response) − log p(Hallucinatory) across five different medical modalities. The dashed line in the figure indicates the median value. A larger value signifies a stronger ability of the VLMs to distinguish hallucinatory responses.…
Figure 5
Figure 5. Figure 5: Zero-shot comparison of LLaVA-Med1.5 (Li et al., 2024a) and HSCR. HSCR generates more accurate, detailed, and image-relevant responses, with attention maps (highlighted in black) showing stronger focus on the image modality, achieving superior alignment and trustworthi…
Figure 6
Figure 6. Figure 6: Performance on general multimodal tasks. Details in Appendix Table [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Representation distribution comparison. allows HSCR to achieve superior modality align￾ment, resulting in more reliable and high-quality answers. By effectively bridging the gap between visual and textual modalities, HSCR not only en￾hances the accuracy but also boosts…
Figure 8
Figure 8. Figure 8: More cases on zero-shot comparison of LLaVA-Med1.5 ( [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. V2T-CoT: From Vision to Text Chain-of-Thought for Medical Reasoning and Diagnosis

    cs.CE 2025-06 conditional novelty 6.0 of 10

    V2T-CoT combines visual region grounding with LLM-generated text rationale training to improve medical visual question answering accuracy and interpretability on four benchmarks.

  2. Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning

    cs.CL 2025-08 conditional novelty 5.0 of 10

    RoMed and CCL: a 144k-question perturbation benchmark for medical VQA and a consistency-plus-contrastive training method that improves LLaVA-Med's accuracy and reduces answer variation.

  3. A Comprehensive Survey of Electronic Health Record Modeling: From Deep Learning Approaches to Large Language Models

    cs.LG 2025-07 reject novelty 4.0 of 10

    A survey that taxonomizes EHR modeling research into data-centric, architectural, learning-focused, multimodal, and LLM-based categories, with datasets and metrics.

Reference graph

Works this paper leans on

68 extracted references · 22 canonical work pages · cited by 3 Pith papers

  1. [1]

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447--4455. PMLR

    2024

  2. [2]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  3. [3]

    Yakoub Bazi, Mohamad Mahmoud Al Rahhal, Laila Bashmal, and Mansour Zuair. 2023. Vision--language model for visual question answering in medical imagery. Bioengineering

    work page 2023

  4. [4]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30

    2017

  5. [5]

    Courtney D Corley and Rada Mihalcea. 2005. Measuring the semantic similarity of texts. In Proceedings of the ACL workshop on empirical modeling of semantic equivalence and entailment, pages 13--18

    work page 2005

  6. [6]

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863

    arXiv 2024

  7. [7]

    Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    arXiv 2020

  8. [8]

    Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. 2023. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023, pages 1151--1163

    work page 2023

Show all 68 references
  1. [9]

    Xiaotang Gai, Chenyi Zhou, Jiaxiang Liu, Yang Feng, Jian Wu, and Zuozhu Liu. 2025. https://aclanthology.org/2025.findings-naacl.415/ M ed T hink: A rationale-guided framework for explaining medical visual question answering . In Findings of the Association for Computational Li...

    2025

  2. [10]

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll \'a r, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000--16009

    2022

  3. [11]

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286

    2020 arXiv

  4. [12]

    Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415

    2016 arXiv

  5. [13]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  6. [14]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  7. [15]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  8. [16]

    Songtao Jiang, Yuan Wang, Sibo Song, Yan Zhang, Zijie Meng, Bohan Lei, Jian Wu, Jimeng Sun, and Zuozhu Liu. 2025. Omniv-med: Scaling medical vision-language model for universal visual understanding. arXiv preprint arXiv:2504.14692

    2025 arXiv

  9. [17]

    Songtao Jiang, Yan Zhang, Ruizhe Chen, Yeying Jin, and Zuozhu Liu. 2024 a . Modality-fair preference optimization for trustworthy mllm alignment. arXiv preprint arXiv:2410.15334

    2024 arXiv

  10. [18]

    Songtao Jiang, Yan Zhang, Chenyi Zhou, Yeying Jin, Yang Feng, Jian Wu, and Zuozhu Liu. 2024 b . Joint visual and text prompting for improved object-centric perception with multimodal large language models. arXiv preprint arXiv:2404.04514

    2024 arXiv

  11. [19]

    Songtao Jiang, Tuo Zheng, Yan Zhang, Yeying Jin, and Zuozhu Liu. 2024 c . Moe-tinymed: Mixture of experts for tiny medical large vision-language models. arXiv e-prints, pages arXiv--2404

    2024

  12. [20]

    Songtao Jiang, Tuo Zheng, Yan Zhang, Yeying Jin, Li Yuan, and Zuozhu Liu. 2024 d . Med-moe: Mixture of domain-specific experts for lightweight medical vision-language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3843--3860

    2024

  13. [21]

    W Bradley Knox and Peter Stone. 2011. Augmenting reinforcement learning with human feedback. In ICML 2011 Workshop on New Developments in Imitation Learning (July 2011), volume 855

    2011

  14. [22]

    Nancy Krieger. 1992. Overcoming the absence of socioeconomic data in medical records: validation and application of a census-based methodology. American journal of public health, 82(5):703--710

    1992

  15. [23]

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1--10

    2018

  16. [24]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  17. [25]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024 a . Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36

    2024

  18. [26]

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024 b . Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895

    2024 arXiv

  19. [27]

    Pengfei Li, Gang Liu, Lin Tan, Jinying Liao, and Shenjun Zhong. 2022. Self-supervised vision-language pretraining for medical visual question answering. arXiv preprint arXiv:2211.13594

    2022 arXiv

  20. [28]

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650--1654. IEEE

    2021

  21. [29]

    Fenglin Liu, Tingting Zhu, Xian Wu, Bang Yang, Chenyu You, Chenyang Wang, Lei Lu, Zhangdaihong Liu, Yefeng Zheng, Xu Sun, et al. 2023 a . A medical multimodal large language model for future pandemics. NPJ Digital Medicine, 6(1):226

    2023

  22. [30]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296--26306

    2024

  23. [31]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 b . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296--26306

    2024

  24. [32]

    Jiaxiang Liu, Jin Hao, Hangzheng Lin, Wei Pan, Jianfei Yang, Yang Feng, Gaoang Wang, Jin Li, Zuolin Jin, Zhihe Zhao, et al. 2023 b . Deep learning-enabled 3d multimodal fusion of cone-beam ct and intraoral mesh scans for clinically applicable tooth-bone reconstruction. Patterns, 4(9)

    2023

  25. [33]

    Jiaxiang Liu, Tianxiang Hu, Jiawei Du, Ruiyuan Zhang, Joey Tianyi Zhou, and Zuozhu Liu. 2025. Kpl: Training-free medical knowledge mining of vision-language models. arXiv preprint arXiv:2501.11231

    2025 arXiv

  26. [34]

    Jiaxiang Liu, Yuan Wang, Jiawei Du, Joey Zhou, and Zuozhu Liu. 2024 c . Medcot: Medical chain of thought via hierarchical expert. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17371--17389

    2024

  27. [35]

    Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, et al. 2024 d . Lipo: Listwise preference optimization through learning-to-rank. arXiv preprint arXiv:2402.01878

    2024 arXiv

  28. [36]

    Yunyi Liu, Zhanyu Wang, Dong Xu, and Luping Zhou. 2023 c . Q2atransformer: Improving medical vqa via an answer querying decoder. arXiv preprint arXiv:2304.01611

    2023 arXiv

  29. [37]

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pages 353--367. PMLR

    2023

  30. [38]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 GPT -4 technical report . https://arxiv.org/abs/2303.08774. Preprint, arXiv:2303.08774

    2023 arXiv

  31. [39]

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. 2024. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228

    2024 arXiv

  32. [40]

    Baolin Peng, Linfeng Song, Ye Tian, Lifeng Jin, Haitao Mi, and Dong Yu. 2023. Stabilizing rlhf through advantage model and selective rehearsal. arXiv preprint arXiv:2309.10202

    2023 arXiv

  33. [41]

    Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. 2024. Strengthening multimodal large language model with bootstrapped preference optimization. arXiv preprint arXiv:2403.08730

    2024 arXiv

  34. [42]

    Roger Kevin Pringle and Lawrence H Wyatt. 2006. The appropriate use of radiography in clinical practice: a report of two cases of biomechanical versus malignant spine pain. Chiropractic & Osteopathy, 14(1):1--8

    2006

  35. [43]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  36. [44]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  37. [45]

    Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. 2005. Semi-supervised self-training of object detection models

    2005

  38. [46]

    Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. 2024. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416

    2024 arXiv

  39. [47]

    Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2024. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18990--18998

    2024

  40. [48]

    Guohao Sun, Can Qin, Huazhu Fu, Linwei Wang, and Zhiqiang Tao. 2024. Stllava-med: Self-training large language and vision assistant for medical question-answering. arXiv preprint arXiv:2406.19973

    2024 arXiv

  41. [49]

    Guohao Sun, Can Qin, Jiamian Wang, Zeyuan Chen, Ran Xu, and Zhiqiang Tao. 2025. Sq-llava: Self-questioning for large vision-language assistant. In European Conference on Computer Vision, pages 156--172. Springer

    2025

  42. [50]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525

    2023 arXiv

  43. [51]

    Tim Van Erven and Peter Harremos. 2014. R \'e nyi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60(7):3797--3820

    2014

  44. [52]

    Tom van Sonsbeek, Mohammad Mahdi Derakhshani, Ivona Najdenkoska, Cees GM Snoek, and Marcel Worring. 2023. Open-ended medical visual question answering through prefix tuning of language models. arXiv preprint arXiv:2303.05977

    2023 arXiv

  45. [53]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  46. [54]

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. 2023. An llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397

    2023 arXiv

  47. [55]

    Qinwei Xu, Xingkun Xu, Chenyi Zhou, Zuozhu Liu, Feiyue Huang, Shaoxin Li, Lifeng Zhu, Zhian Bai, Yuchen Xu, and Weiguo Hu. 2025. https://doi.org/10.1016/j.eswa.2025.126585 Towards normalized clinical information extraction in chinese radiology report with large language models...

    2025

  48. [56]

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. 2024 a . Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Confere...

    2024

  49. [57]

    Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. 2024 b . Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220

    2024

  50. [58]

    Kai Zhang, Jun Yu, Zhiling Yan, Yixin Liu, Eashan Adhikarla, Sunyang Fu, Xun Chen, Chen Chen, Yuyin Zhou, Xiang Li, et al. 2023 a . Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks. arXiv preprint arXi...

    2023 arXiv

  51. [59]

    Mengxi Zhang, Wenhao Wu, Yu Lu, Yuxin Song, Kang Rong, Huanjin Yao, Jianbo Zhao, Fanglong Liu, Yifan Sun, Haocheng Feng, et al. 2024. Automated multi-level preference for mllms. arXiv preprint arXiv:2405.11165

    2024 arXiv

  52. [60]

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, et al. 2023 b . Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915

    2023 arXiv

  53. [61]

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023 c . Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415

    2023 arXiv

  54. [62]

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. 2024 a . Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411

    2024 arXiv

  55. [63]

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. 2024 b . Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622

    2024 arXiv

  56. [64]

    Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. 2024 c . Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024, pages 10586--10613

    2024

  57. [65]

    Banghua Zhu, Michael I Jordan, and Jiantao Jiao. 2024 a . Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arXiv:2401.16335

    2024 arXiv

  58. [66]

    Kangyu Zhu, Peng Xia, Yun Li, Hongtu Zhu, Sheng Wang, and Huaxiu Yao. 2024 b . Mmedpo: Aligning medical vision-language models with clinical-aware multimodal preference optimization. arXiv preprint arXiv:2412.06141

    2024 arXiv

  59. [67]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  60. [68]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.