Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models

Cheng-Yi Li; Haiqin Zhong; Hsin-Yu Wu; Jipeng Zhang; Linlin Shen; Meidan Ding; Wei-Chieh Fang; Wenting Chen; Wenxuan Wang

REVIEW 4 major objections 5 minor 20 references

Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Med-RewardBench: a new benchmark for medical reward models and judges reports that the best tested model, O1, agrees with clinical experts only 68.86% of the time, while several open-source and medical-specialist models hover near chance.

desk verdict A useful first benchmark for medical reward models, but the gold labels are weaker than the paper admits and the exact numbers need a cleanup pass. read the letter →

arxiv 2508.21430 v1 pith:FUAAFOMR submitted 2025-08-29 cs.CL cs.AIcs.CV

Meidan Ding , Jipeng Zhang , Wenxuan Wang , Cheng-Yi Li , Wei-Chieh Fang , Hsin-Yu Wu , Haiqin Zhong , Wenting Chen

show 1 more author

Linlin Shen

This is my paper

classification cs.CLcs.AIcs.CV

keywords rewardmodelmultimodallargelanguagemedicalbenchmarkjudgeevaluationclinicalalignmentpreferenceannotation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Med-RewardBench is a benchmark designed to test whether multimodal large language models can act as reliable judges or reward models in medicine, meaning they choose which of two AI answers to a clinical question is better. It contains 1,026 image-question-response triples spanning 13 organ systems and 8 departments, with pairwise preferences annotated by three practicing general practitioners on six dimensions: accuracy, relevance, comprehensiveness, creativity, responsiveness, and overall. The paper reports that the strongest proprietary model, O1, agrees with the clinicians 68.86% of the time overall, while medical-specific models such as MedDr and HuatuoGPT-Vision cluster near or barely above the 50% chance level. The central claim is that current models do not align well enough with expert medical judgment to be trusted as reward models or judges, and that medical-specialist training does not automatically produce good judging. If true, this gives the field a missing measurement tool and a concrete target for improving clinically aligned AI.

What carries the argument

The core object is a pairwise preference dataset: for each of 1,026 cases, two responses sampled from a pool of 12 MLLMs are paired with a medical image and question, and clinicians choose the better response on six dimensions. Construction used five small MLLMs as weak filters to keep only questions that fewer than three could answer, giving deliberately hard cases; expert annotation with majority voting supplies gold labels; and model evaluation measures agreement with those labels. The six dimensions—accuracy, relevance, comprehensiveness, creativity, responsiveness, and overall—are the scoring rubric that makes judging measurable.

What would settle it

Re-annotate a stratified sample of the 1,026 cases with specialty-matched expert panels, for example ophthalmologists for eye cases and radiologists for radiology cases, then compare their majority labels to the original general practitioners' labels. If agreement is low, or if the model leaderboard changes materially under the new gold labels, the central claim of moderate agreement would not be robust.

Watch

Extended reading notes

Core claim

The paper's central discovery is a measurement: when multimodal LLMs are asked to pick the better of two candidate medical answers, their choices match expert clinician choices only moderately, and no model reaches what the authors would consider reliable agreement. The benchmark is organized as preference pairs of an image, an instruction, and two responses, and a model's score is the rate at which its chosen response matches the majority vote of three general practitioners across six evaluation dimensions. The top result, O1, reaches 68.86% overall agreement; the best open-source model, Qwen2-VL-72B, reaches 65.27%; and medical-specific models such as MedDr, LLaVA-Med, STLLaVA-Med, HuatuoG

Load-bearing premise

The benchmark's gold labels are the majority vote of three general practitioners with 4–5 years of clinical experience, applied to every dimension and every specialty; if those labels are noisy or systematically biased in subspecialty cases, the reported model rankings inherit that noise.

Editorial extensions

If this is right

Models used as reward models or judges in medical reinforcement learning and test-time best-of-n selection will carry their own judgment biases into clinical outputs, so a model's score here maps directly to downstream trust.
Medical-specific training on diagnosis and visual question answering does not guarantee judging ability; diagnostic competence and preference ranking appear to be separate skills.
Fine-tuning Qwen2-VL-7B on 10,000 curated preference pairs with SFT and DPO improves its agreement with clinicians, suggesting the judging gap is at least partly trainable.
Highly specialized domains such as ophthalmology remain hard for every model, indicating where more data and scrutiny are needed most.
The benchmark provides a standardized, dimension-by-dimension report so developers can see which aspect of judgment, such as creativity versus accuracy, a model fails on.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct extension would test whether model rankings hold up when labels come from specialist panels per organ system rather than general practitioners; the paper's own consistency check covers only 84 uncertain samples.
Since the fine-tuning baselines use Qwen2-VL-72B outputs as ground truth, the reported improvement measures alignment with a strong model, not necessarily with clinicians; human-labeled preference data would be a stricter test.
The A/B balanced design likely mitigates position bias, but it does not fully replace control experiments that swap response order; those controls are an implicit next step.
The benchmark could be extended from pairwise choice to continuous reward scores and multi-response ranking, which would test whether coarse A/B agreement translates to fine-grained reward modeling.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Med-RewardBench is a genuinely useful first step: it is the first benchmark I have seen that evaluates medical MLLMs as reward models or judges rather than as solvers. The coverage is reasonable (13 organs, 8 departments, 1,026 cases) and the six dimensions (accuracy, relevance, comprehensiveness, creativity, responsiveness, overall) are clinically plausible. The A/B position balance is handled, and the three-step filtering shows care. The headline result—O1 at 68.86% agreement, medical-specific models near chance—is directionally credible, and the fine-tuning baselines show real gains.

The soft spots are real but addressable. The gold labels are a majority vote of three GPs with 4-5 years' experience. The only agreement check is on 84 samples drawn from 'uncertain' cases, and there the three-way consensus is often below half (e.g., 25/84 for Comprehensiveness). No kappa or alpha is reported for the full set, and the Limitations section is silent on label reliability. For specialties like ophthalmology, a GP majority is a weak gold standard. This does not kill the benchmark, but the exact rankings in the dense 55-65% band could shift with better labels.

Second, the paper gives no confidence intervals or significance tests, and the tables have visible errors: MedDr OVE = 5600.00 in Table 11, garbled entries in Table 2, 'Claude-1.5' in Table 17. These look like transcription problems, but they erode trust in the precise numbers. Third, the SFT/DPO baselines use Qwen2-VL-72B outputs as ground truth and 'chosen' responses; the improvement is partly alignment to that model's style. Since evaluation is against human labels, this is not circular, but it is not a pure measure of clinician alignment. And the code and data are not yet public.

Who should read it? Anyone building or evaluating medical reward models or doing RLHF for medical MLLMs. The gap is real, and the empirical finding is useful even if the numbers need a cleanup pass. It deserves a serious referee, not a desk reject. The review should ask for full annotation statistics, confidence intervals, corrected tables, and data release.

Referee Report

4 major / 5 minor

Summary. The paper introduces Med-RewardBench, a medical multimodal reward-model/judge benchmark consisting of 1,026 expert-annotated image-question-response preference pairs spanning 13 organ systems and 8 departments. The benchmark scores pairwise judgments on six clinical dimensions (Accuracy, Relevance, Comprehensiveness, Creativity, Responsiveness, Overall) against labels from three general practitioners. The authors evaluate 32 MLLMs and report that the best proprietary model (O1) reaches 68.86% overall agreement with human labels, while many open-source and medical-specific models score between 51% and 58%, leading to the claim that current medical MLLMs are not well aligned with expert judgment. They also fine-tune Qwen2-VL-7B with SFT and DPO and report substantial improvements in judging performance.

Significance. If the human labels are reliable, Med-RewardBench fills a genuine gap: existing medical benchmarks evaluate models as solvers rather than judges, and general reward benchmarks omit clinically critical dimensions. The 32-model comparison is a useful reference, the six-dimension annotation protocol is sensible, and the construction is not circular because the gold labels come from human experts. The main risks are label reliability and statistical precision, not the benchmark concept itself. The paper also provides explicit annotation guidelines and plans public release, which are strengths.

major comments (4)

[Section 3.3 and Figure 3] The inter-annotator reliability evidence is insufficient for the central claim. The consistency check covers only 84 uncertain samples, and Figure 3 reports raw counts of two-vs-three doctor agreement without any kappa or Fleiss' alpha for the full 1,026 cases. The counts show substantial non-unanimity (e.g., for Comprehensiveness, 59/84 samples have only two doctors agreeing). Since the gold labels are the majority vote of three general practitioners across specialties such as ophthalmology and radiology, unmeasured label noise propagates directly into every model agreement score in Tables 2–3. Please report full inter-annotator statistics, add specialty-level adjudication or a label-noise sensitivity analysis, and quantify how the reported rankings would shift under plausible label-flip rates.
[Tables 2–3 and Section 4.3] The empirical comparisons are presented as point estimates without confidence intervals or significance tests. For per-organ samples (n≈80), a 10-point difference is needed for rough significance at the 5% level; for the overall n=1026, differences below about 3 points are within sampling error. Several conclusions in Section 4.3.1 and 4.3.2 rely on such differences (e.g., GPT-4o at 52.06% vs. MedDr at 57.84%; per-organ cardiac scores of 76% vs. 72%). Conversely, the Introduction's statement that medical-specific models 'struggle to perform better than random chance' is ambiguous: at n=1026, HuatuoGPT-Vision's 55.58% in Table 2 is statistically distinguishable from 50% under a simple binomial test. Reporting confidence intervals and, where appropriate, pairwise significance tests would substantially strengthen the tiered-ranking and near-chance claims.
[Table 11 and Appendix Tables 4–16] There are data-integrity errors in the numerical results. Table 11 (Heart) lists MedDr's Overall score as 5600.00, which is an obvious typo for 56.00. Several appendix tables also contain suspiciously identical rows for distinct models (e.g., Table 8 Eye and Table 9 Foot list the same six values for LLaVA-Med and STLLaVA-Med; see also Table 10 GI). Because the paper's empirical conclusions rest entirely on these numbers, all tables should be regenerated and audited before the results can be considered reliable.
[Section 4.3.3] The baseline training procedure should be interpreted more cautiously. The SFT baseline uses Qwen2-VL-72B responses as ground truth, and the DPO baseline uses Qwen2-VL-72B as 'chosen' and Qwen2-VL-2B as 'rejected'. The reported improvement (Qwen2-VL-7B 52.16% to Qwen2-VL-Judge 57.46% in Table 2) therefore partly reflects imitation of a stronger model's judging behavior, not necessarily improved alignment with clinical expertise. The paper should report agreement with Qwen2-VL-72B before/after fine-tuning, and temper the language in the Abstract and Conclusion that presents the baselines as demonstrating improved judgment per se.

minor comments (5)

[Section 3.3] The phrase 'Uncertain cases were resolved through majority voting' is ambiguous. If all 1,026 labels are majority-voted, state so explicitly and report the initial agreement distribution over the full dataset, not just 84 uncertain samples.
[Figure 5 caption] The caption says 'rain (b)' but should read 'brain (b)'.
[Table 3 and Section 4.1] Table 3 labels a model 'Claude-1.5', but Section 4.1 lists 'Claude-3.5-Sonnet'. The table should match the model list.
[Appendix Tables 4–16] The fine-tuned baseline is called 'Qwen2-VL-Judge' in Table 2 and Section 4.3.3, but the appendix tables use 'Qwen2-VL-SFT'. Please use consistent naming.
[References] The LLaVA-Med reference appears twice with different years (Li et al., 2024a and 2024b) for what appears to be the same paper, and the entry 'et al. Le Xue' is malformed. These should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Med-RewardBench's gold labels are independent human expert judgments, and the model evaluation is measured against those labels; training baselines use a model teacher but are evaluated on human labels, so no claim reduces to its inputs.

full rationale

The paper's central evaluation is self-contained: Med-RewardBench is constructed by collecting image-question pairs from external datasets, generating candidate responses from a pool of MLLMs, and obtaining gold preference labels from three human general practitioners via majority voting (Section 3.3). The reported agreement scores (e.g., O1 at 68.86%) are computed as consistency between model-selected preferences and these human-selected preferences, so the benchmark ground truth is independent of any model output. The selection of 'difficult' pairs using five small MLLMs (Section 3.1) may affect difficulty distribution, but it does not define the labels or the evaluation metric. The baseline training (Section 4.3.3) uses Qwen2-VL-72B responses as SFT ground truth and as 'chosen' pairs in DPO, but the baselines are then evaluated on Med-RewardBench's human labels, not on the teacher model's preferences. Therefore the observed improvement is an empirical result, not an equivalence by construction. No self-citation chain, uniqueness argument, or ansatz-smuggling is used to justify the main claims. The paper's limitations concern baseline generality rather than hiding a circular step. Consequently, no circular step meets the evidentiary bar required by the review rules.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on expert-judgment assumptions rather than mathematical axioms. The main entries are that three general practitioners' majority vote is a reliable gold standard, that the chosen six dimensions capture clinical quality, and that the difficulty filter selects a useful test set. No continuous free parameters are fitted in the evaluation, though dataset-construction thresholds (e.g., 'fewer than three of five weak models correct', 80 items per organ) are hand-selected design choices.

assumptions (4)

domain assumption Majority vote of three general practitioners with 4-5 years of experience is a reliable gold standard for all six evaluation dimensions across all 13 organ systems.
Section 3.3 describes labels produced by 3 GPs with majority voting, with consistency checked on only 84 samples. If the experts are not authoritative in specialties like ophthalmology, the benchmark labels are noisy.
domain assumption The six dimensions (Accuracy, Relevance, Comprehensiveness, Creativity, Responsiveness, Overall) capture the clinically critical aspects of response quality.
Section 3.3 introduces these dimensions as the evaluation protocol. There is no independent validation that these six dimensions are the correct or sufficient axes for medical reward models.
domain assumption Selecting image-question pairs that fewer than three of five small MLLMs answer correctly yields a useful and appropriately challenging benchmark.
Section 3.1 describes the difficulty filter. This hand-selected threshold ('fewer than three') may bias the benchmark toward atypical or adversarial items and affects all downstream model rankings.
domain assumption The pairwise comparison prompt (Appendix A) is neutral and does not introduce ordering or position bias.
Section 4.2 uses a fixed prompt for all evaluated models. The paper does not report evaluating both A/B orders, so position bias is not controlled for in the 32-model evaluation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/FUAAFOMR

@misc{pith2026250821430,
  author       = {Pith},
  title        = {Pith review of: Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FUAAFOMR}},
  note         = {Machine review of arXiv:2508.21430}
}

read the original abstract

Multimodal large language models (MLLMs) hold significant potential in medical applications, including disease diagnosis and clinical decision-making. However, these tasks require highly accurate, context-sensitive, and professionally aligned responses, making reliable reward models and judges critical. Despite their importance, medical reward models (MRMs) and judges remain underexplored, with no dedicated benchmarks addressing clinical requirements. Existing benchmarks focus on general MLLM capabilities or evaluate models as solvers, neglecting essential evaluation dimensions like diagnostic accuracy and clinical relevance. To address this, we introduce Med-RewardBench, the first benchmark specifically designed to evaluate MRMs and judges in medical scenarios. Med-RewardBench features a multimodal dataset spanning 13 organ systems and 8 clinical departments, with 1,026 expert-annotated cases. A rigorous three-step process ensures high-quality evaluation data across six clinically critical dimensions. We evaluate 32 state-of-the-art MLLMs, including open-source, proprietary, and medical-specific models, revealing substantial challenges in aligning outputs with expert judgment. Additionally, we develop baseline models that demonstrate substantial performance improvements through fine-tuning.

Figures

Figures reproduced from arXiv: 2508.21430 by the authors.

**Figure 2.** Overview of Med-RewardBench with three-step construction process, i.e., 1) image-question pair collection, 2) MLLM response collection, and 3) comparison with human annotations. 3 Med-RewardBench In this section, we introduce the construction process of Med-RewardBench in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Consistency statistics of experts’ annotations. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Box plots illustrate the distribution of perfor [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Performance of different MLLMs in abdomen [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: The default prompt for pairwise comparison. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

**Figure 7.** Figure 7: Annotation guidelines for medical experts. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

**Figure 8.** Figure 8: An example of Med-RewardBench in six dimensions. Model ACC REL COM CRE RES OVE Open-Source MLLMs VILA1.5-3b 43.66 43.66 49.30 52.11 43.66 43.66 xGen-MM-instruct 43.66 49.30 43.66 43.66 52.11 43.66 Deepseek-vl2 64.79 54.93 52.11 46.48 54.93 59.15 Phi-3-5-Vision 46.48 45…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 13 canonical work pages

[1]

**Accuracy**: How correct and precise is the information provided?

work page
[2]

**Relevance**: How well does the response address the question or topic at hand?

work page
[3]

**Comprehensiveness**: Does the response cover all relevant aspects of the topic?

work page
[4]

**Creativity**: How original and innovative is the response in its approach or explanation?

work page
[5]

**Responsiveness**: How well does the response engage with the question, considering context and clarity?

work page
[6]

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 13183–13192

Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 13183–13192. Guijin Son, Dongkeun Yoon, Juyoung Suk, Javier Aula- Blasco, Mano Aslan, Vu Trong Kim, Shayekh Bin Islam, Jaume Prats-Cristià, Lucía Tor...

arXiv 2024
[7]

arXiv preprint arXiv:2403.05530

Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhanc- ing vision-language model’s perception of the world at any resolution. arXiv preprint ...

arXiv 2024
[8]

In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 3: System Demonstra- tions), Bangkok, Thailand

Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 3: System Demonstra- tions), Bangkok, Thailand. Association for Computa- tional Linguistics. Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, and Shafiq Joty. 2025. Evaluating judges a...

arXiv 2025

Show all 20 references

[14]

Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision

**Overall**: Considering all the above factors, which response is better overall? Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence ...
[15]

immediate PET-CT for a suspected pulmonary nodule,

Accuracy: - How medically correct and evidence-based is the information provided? - Example: If the model suggests "immediate PET-CT for a suspected pulmonary nodule," determine whether this recommendation aligns with clinical guidelines
[16]

Relevance: - How well does the response stay focused on the core question and the image information? - Example: If the question is about potential lung cancer but discusses COPD, the response is less relevant
[17]

Comprehensiveness: - Does the response cover all clinically important aspects, like differential diagnosis or risks? - Example: Mentioning only one possible diagnosis shows less comprehensiveness
[18]

Creativity: - Does the response offer insightful or innovative reasoning? - Example: Detecting subtle signs and proposing early diagnosis indicates creativity
[19]

The most likely cause is... I recommend

Responsiveness: - Does the response clearly address the inquiry with structured, clinical clarity? - Example: A response that says, "The most likely cause is... I recommend..." is preferable
[20]

Additional Notes - All medical images are de-identified from real clinical cases

Overall: - Considering all criteria, which response is better overall? Choose one and justify briefly. Additional Notes - All medical images are de-identified from real clinical cases. - Evaluate based on clinical standards, not writing style or length. - Your input supports i...
[1654]

Fenglin Liu, Tingting Zhu, Xian Wu, Bang Yang, Chenyu You, Chenyang Wang, Lei Lu, Zhangdai- hong Liu, Yefeng Zheng, Xu Sun, et al

IEEE. Fenglin Liu, Tingting Zhu, Xian Wu, Bang Yang, Chenyu You, Chenyang Wang, Lei Lu, Zhangdai- hong Liu, Yefeng Zheng, Xu Sun, et al. 2023. A medical multimodal large language model for future pandemics. NPJ Digital Medicine, 6(1):226. Haotian Liu, Chunyuan Li, Qingyang Wu,...

2023
[2022]

Advances in neural in- formation processing systems, 35:27730–27744

Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730–27744. Shu Pu, Yaochen Wang, Dongping Chen, Yuhang Chen, Guohao Wang, Qi Qin, Zhongyi Zhang, Zhiyuan Zhang, Zetong Zhou, Shuang Gong, et al. 202...

2025 arXiv
[2023]

Advances in Neural Information Processing Systems, 36:53728– 53741

Direct preference optimization: Your lan- guage model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728– 53741. Jiacheng Ruan, Wenzhen Yuan, Xian Gao, Ye Guo, Daoxin Zhang, Zhe Xu, Yao Hu, Ting Liu, and Yuzhuo Fu. 2025. Vlrmbench: A compre...

2025 arXiv
[2024]

arXiv preprint arXiv:2404.15127

Meddr: Diagnosis-guided bootstrapping for large-scale medical vision-language learning. arXiv preprint arXiv:2404.15127. Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXi...

2020 arXiv
[2025]

arXiv preprint arXiv:2502.13923

Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923. Asma Ben Abacha, Sadid A Hasan, Vivek V Datla, Dina Demner-Fushman, and Henning Müller. 2019. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. In Proceedings of CLEF (Conference and...

2019 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

**Accuracy**: How correct and precise is the information provided?

work page

[2] [2]

**Relevance**: How well does the response address the question or topic at hand?

work page

[3] [3]

**Comprehensiveness**: Does the response cover all relevant aspects of the topic?

work page

[4] [4]

**Creativity**: How original and innovative is the response in its approach or explanation?

work page

[5] [5]

**Responsiveness**: How well does the response engage with the question, considering context and clarity?

work page

[6] [6]

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 13183–13192

Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 13183–13192. Guijin Son, Dongkeun Yoon, Juyoung Suk, Javier Aula- Blasco, Mano Aslan, Vu Trong Kim, Shayekh Bin Islam, Jaume Prats-Cristià, Lucía Tor...

arXiv 2024

[7] [7]

arXiv preprint arXiv:2403.05530

Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhanc- ing vision-language model’s perception of the world at any resolution. arXiv preprint ...

arXiv 2024

[8] [8]

In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 3: System Demonstra- tions), Bangkok, Thailand

Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 3: System Demonstra- tions), Bangkok, Thailand. Association for Computa- tional Linguistics. Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, and Shafiq Joty. 2025. Evaluating judges a...

arXiv 2025