Pith. sign in

REVIEW 4 major objections 5 minor 42 references

For multi-image medical question answering, the choice of inference-time decision rule—not the amount of search—determines final accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 05:34 UTC pith:4DL346ZT

load-bearing objection A well-scoped systems comparison with a solid main result and a budget-sensitivity headline that currently leans on a single run. the 4 major comments →

arxiv 2607.27564 v1 pith:4DL346ZT submitted 2026-07-30 cs.CV

Inference-Time Agentic Decision Rules Beat Longer Evolving Search for Multi-Image Medical Reasoning

Site Li , Jianyi Hao , Xiaofeng Liu This is my paper
classification cs.CV
keywords multi-image medical VQAinference-time decision rulesorder-vote policyanswer-order robustnessevolutionary prompt searchbudget sensitivityagentic reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

For multi-image medical visual question answering, the paper claims that the design of the inference-time decision rule matters more than the amount of evolutionary search used to optimize the prompt. Under a controlled protocol with a frozen split, a simple order-vote policy—reordering answer options, mapping local predictions back to the original labels, and aggregating by majority—reaches 57.89% accuracy on an independent test set, beating a fixed baseline (52.73%) and a more complex reranking variant (55.79%). The paper also finds that doubling the search budget from 50 to 100 generations does not improve final test accuracy, and slightly lowers it. The point is that robust evidence aggregation, rather than more deliberation or more search, sets the ceiling for this benchmark class.

Core claim

The central claim, stated on the paper's own terms, is that 'defining the correct agentic decision rule is substantially more impactful than expanding the optimization search budget.' Concretely, the paper shows that on a multi-image medical VQA benchmark, an order-vote policy—which perturbs answer order across multiple local views, maps each prediction back to the original label space, and aggregates by majority—outperforms both a one-shot fixed prompt baseline and an order-rerank variant that adds a conditional second-stage comparison. The margin persists across repeated runs (57.89% vs 52.73% and 55.79%), and the reranking variant costs more compute. Extending evolutionary search to 100 g

What carries the argument

The load-bearing object is the order-vote decision rule: for each question, the model receives the ordered image set with answer options in several shuffled orders, produces a local answer for each order, maps those answers back to the original option labels, and returns the majority vote. Its work is to make the inference policy robust to answer-order perturbation—the paper's identified dominant nuisance in multiple-choice multi-image reasoning—while staying cheaper than second-stage reranking. The evaluation protocol (separated evolution, holdout selection, and frozen independent test) is the other key machinery, because it lets the authors attribute differences to the decision rule rather

Load-bearing premise

The internal frozen final-test split (855 examples) was built by the authors from the visible public benchmark pool using a search that explicitly balanced answer-label and modality distributions; the whole generalization claim assumes this label-peeking split behaves like a true held-out test and does not distort which strategy wins.

What would settle it

Run the same five strategies on a test set that was never used in split construction (e.g., an official hidden server or a freshly collected set of multi-image medical questions) and compare order-vote against the fixed baseline; if the margin disappears, or if a 100-generation search then beats 50 generations on that external split, the paper's central claim is refuted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • For this benchmark class, further search budget on the same decision rule will not reliably improve final accuracy; effort should shift to the structure of the inference policy.
  • An order-vote policy gives a better accuracy–cost tradeoff than reranking, making it a practical default for agents that need fast responses on sequential medical evidence.
  • Longer evolutionary search can overfit the holdout signal; evaluations of prompt evolution should always include an independent final-test split.
  • Answer-order sensitivity is a measurable and dominant failure mode in medical VQA; reporting robustness to option-order perturbation should become standard.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the decision rule is the bottleneck, similar gains might appear in other sequential multimodal tasks (e.g., longitudinal scans, multi-view radiographs) where evidence aggregation across frames is central and answer-order perturbations may be replaced by other nuisance variations.
  • A testable extension: let the evolutionary search vary the aggregation operation itself (e.g., weighted votes, tie-break rules, or a learned fusion) rather than only prompt wording; the current comparison keeps the vote rule fixed while evolving text.
  • The internal split was constructed by balancing answer labels, which may change the difficulty ordering; an external or freshly sampled test set would confirm whether the 57.89% figure generalizes.
  • The budget-sensitivity result is one run; a multi-seed 100-generation comparison would sharpen the claim that longer search 'amplifies selection noise.'

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a controlled comparison of five inference-time agentic strategies for multi-image medical VQA on MedFrameQA, all optimized with the same high-budget ShinkaEvolve search engine. The authors construct an internal frozen split (1,331 evolution / 665 holdout / 855 final test), group-preserving by video id and balanced for modality and answer-label distributions. The main result, based on five independent repeated runs, is that the Order-Vote strategy achieves 57.89 ± 0.65% final-test accuracy, outperforming the Fixed baseline (52.73 ± 0.42%) and the Order-Rerank variant (55.79 ± 0.43%). A single 100-generation ablation shows a 1.87-point drop in final-test accuracy relative to the 50-generation mean, which the authors interpret as evidence that decision-rule choice matters more than additional search budget.

Significance. If the main finding holds, the paper provides a practical design principle: for ordered multi-image medical reasoning, a simple order-robust vote aggregation can outperform more complex reranking while also being computationally cheaper. The paper has genuine strengths: the protocol cleanly separates search, holdout selection, and final testing; the three main-table methods are each run five times with reported means and SDs on a frozen split; and the authors are transparent about the internal split, the single VLM setup, and the lack of a stable seed. These are commendable for a systems paper. However, the headline budget-sensitivity claim rests on a single 100-generation run, and the internal split is constructed with answer-label information, so the generalization claim is not yet fully supported. The paucity of repeated evidence for the budget conclusion and the label-aware split construction are the main barriers to accepting the paper's strongest claims as stated.

major comments (4)
  1. [§A.1, Table 10, Figure 6] The abstract and conclusion claim that extending the search budget from 50 to 100 generations 'yields no generalization benefit' and 'can even degrade it.' This claim is supported by a single 100-generation run (56.02% final) compared against a 50-generation mean (57.89 ± 0.65%, n=5). The 1.87-point gap is within roughly three SDs of the 50-generation mean, and the 100-generation condition has no error bar. Because the title and central narrative hinge on 'beat longer evolving search,' this comparison needs at least five independent 100-generation repeats (or a confidence interval) before the negative budget-sensitivity result can be considered established.
  2. [§3, Task Setup and Internal Frozen Split] The 'independent final test' split is constructed by the authors from the visible public MedFrameQA pool using a multi-start bundle-exchange search that explicitly balances modality and answer-label distributions. Using answer labels to construct the test set means the split is not a random sample from the target distribution, and it can distort the difficulty ordering across strategies in ways that may favor vote aggregation. The paper correctly labels it an 'internal frozen split,' but the abstract and §7 describe final-test accuracy as 'independent' and 'generalization.' Please demonstrate robustness of the main ranking to a random split (or the official hidden test), or at minimum report the split optimization objective and show that the main conclusions are insensitive to the balancing weights.
  3. [§5.1, Table 5] Table 5 reports paired bootstrap p-values (0.0000 and 0.0034) computed on 'representative selected runs' rather than on the full set of five repeated runs. This analysis does not propagate between-run variability, so the abstract's statement that 'paired bootstrap analysis confirms these significant gains' is not directly supported by the reported evidence. Please compute the bootstrap over pooled repeated-run data, or report the distribution of p-values across all pairs of runs. Given the effect sizes, the significance conclusion may survive, but the current presentation is not a valid significance test for the repeated-run means.
  4. [§5, Budget and selection] Order-Rerank uses a smaller search mini (128) than all other methods (256) because of its higher inference cost. The abstract says the strategies are 'optimized using the same high-budget ShinkaEvolve configuration,' but the per-generation search signal is twice as noisy for Order-Rerank. This is a confound when comparing final accuracy across strategies: Order-Rerank may be disadvantaged by noisier search feedback, not only by its decision rule. Please equalize the search mini for Order-Rerank (accepting the higher cost), or provide evidence that the main ranking is insensitive to this asymmetry.
minor comments (5)
  1. [§5.2, Modality breakdown] The sentence 'order-vote is better on CT, MRI, X-ray, and other categories, and on every image-count bucket in the representative selected runs, with a small ultrasound deficit as the clearest local exception' appears twice in the same paragraph. Please remove the duplicate.
  2. [Table 7] The column header 'Selected Gen Gen-0 Final (%) Selected Final (%)' is ambiguous. Use separate headers such as 'Gen-0 Final (%)' and 'Selected Final (%)', and clarify that these are representative runs, not averages.
  3. [References] Several reference entries are malformed, e.g., 'and 1 oth ers.' in [2], [11], [22], [25], [28], [29], [33], and [35]. Some also lack complete author lists. Please fix these.
  4. [Figure 6] The 100-generation point is shown without an error bar, while the 50-generation point is a mean with SD. Since the paper treats this as a quantitative comparison, the figure should make the n=1 nature of the 100-generation condition explicit, e.g., by showing individual run values if repeats are added.
  5. [§A.1] The table caption calls the 100-generation run 'single'; however, the abstract and main text do not carry the same qualification. At minimum, add a sentence in the abstract or §1 that the budget-sensitivity conclusion comes from a single run and should be interpreted as preliminary.

Circularity Check

0 steps flagged

No circularity: the paper's claims are empirical comparisons under a shared protocol, not reductions to fitted inputs or self-citation chains.

full rationale

This is an empirical systems-and-evaluation paper, not a derivation. The central result—that order-vote outperforms fixed and order-rerank—is a measured accuracy comparison on a shared internal frozen split; no equation or construction makes that outcome equal to any input. The internal split is constructed by balancing modality and answer-label distributions, but the reported accuracies are not parameters of that construction, and all five strategies are evaluated on the exact same split, so the relative comparison is not forced by the split design. The budget-sensitivity claim (50 vs. 100 generations) is based on a single 100-generation run compared to a five-run mean, which is a statistical-validity weakness rather than a circular reduction; the paper itself flags the run as a single ablation (Appendix A.1, Table 10). ShinkaEvolve [17] and MedFrameQA [32] are external works, and no load-bearing argument depends on a self-citation. The paper's stated limitations—single benchmark, internal split rather than hidden test server, non-deterministic repeated runs instead of seed sweeps, and one VLM setup—are acknowledged confounds, not circular steps. Nothing in the manuscript reduces, by construction or by definition, to a fitted quantity or a self-referential uniqueness claim. Score 0.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

As a systems-and-evaluation paper there are no fitted physical constants; the ledger instead catalogs the hand-chosen protocol hyper-parameters (search-mini asymmetry, evolution profile, unreported vote view count, split-balancing targets) and the domain assumptions on which the comparative claim rests (benchmark validity, internal-split independence, single VLM stability, search fairness, single-run bootstrap validity). The label-peeking split construction (balancing by answer-label distribution) is the most consequential assumption: it limits the claim that 57.89% measures generalization rather than fit to the constructed split.

free parameters (4)
  • search mini size (per-generation search subset) = 256 (Fixed/Reasoning/Order-Vote/Order-Vote+); 128 (Order-Rerank)
    Hand-chosen protocol asymmetry; order-rerank received half the per-generation search examples, confounding the strategy comparison (§5 'Budget and selection').
  • evolution profile hyperparameters = 5 islands, archive 40, migration every 10 gens, 3 patch attempts, diff/full patches
    Chosen by hand; cross-style edits 'empirically destabilizing' (Appendix A.7).
  • number of reordered views in Order-Vote / Order-Rerank = not reported
    The vote size is a strategy parameter that is never given numerically; required for replication (Table 11 lists it as evolvable but no value is stated).
  • split-balancing optimizer targets (modality and answer-label distribution weights) = unspecified
    The internal split is optimized to balance modality and answer-label distributions; the target weights are not given, and label-peeking during split construction weakens the independence of the final test (Section 3).
axioms (5)
  • domain assumption MedFrameQA's two-to-five-image multiple-choice items are a valid measure of multi-image clinical reasoning.
    The benchmark is used as the sole testbed; conclusions are generalized to 'multi-image medical reasoning' from this single benchmark (Sections 3 and 6).
  • domain assumption The internal frozen split (1331/665/855) preserves the public pool's difficulty ordering across strategies.
    The split is constructed from the visible public pool with a bundle-exchange optimizer that balances answer-label distributions; the paper assumes this internal final test behaves like an independent test set (Section 3).
  • domain assumption The local Qwen3.5-35B-A3B VLM endpoint provides a stable enough answerer for accuracy comparisons.
    Single evaluation backbone; cross-model robustness is not tested and the authors list this as a limitation (Section 6).
  • domain assumption ShinkaEvolve's search is a fair shared optimizer across strategies despite the asymmetric search-mini sizes.
    The paper's comparative logic requires that the shared engine and unequal search budgets do not systematically disadvantage order-rerank (§5 'Budget and selection'; Appendix A.7).
  • ad hoc to paper Paired bootstrap on a single selected 'representative' run measures cross-method significance.
    Bootstrap is standard mathematics, but its application to a post-hoc chosen 'representative' run violates the i.i.d. assumption across runs (Table 5 caption).
invented entities (1)
  • medframeqa split manifest v2 (internal frozen split) no independent evidence
    purpose: Provides evolution/holdout/final-test separation on a visible public pool without an official hidden test server.
    Constructed by the authors via label-balancing bundle-exchange search over the public MedFrameQA pool; no external validation that this split's difficulty ordering matches the official hidden test, and no public artifact is provided to inspect it.

pith-pipeline@v1.3.0-daily-deepseek · 13314 in / 21871 out tokens · 204023 ms · 2026-08-01T05:34:46.348758+00:00 · methodology

0 comments
read the original abstract

Multi-image medical VQA is not merely a prompt-length problem; it is a fundamental challenge of agentic decision-making. Medical vision-language agents must aggregate evidence across ordered images, remain robust to answer-order perturbations, and avoid overfitting to noisy search-time feedback. We study MedFrameQA through a controlled comparison of five inference-time agentic strategies, optimized using the same high-budget ShinkaEvolve configuration and evaluated on a reproducible internal frozen split (1,331 evolution, 665 holdout, 855 final test). Across five independent repeated runs, the strongest method emerges as the simplest robust aggregator: the \textbf{order-vote} policy achieves $57.89 \pm 0.65\%$ final-test accuracy, significantly outperforming the fixed baseline ($52.73 \pm 0.42\%$) and the more complex, albeit brittle, order-rerank variant ($55.79 \pm 0.43\%$). Paired bootstrap analysis confirms these significant gains. Extending the evolutionary search budget from 50 to 100 generations yields no generalization benefit: while holdout performance marginally increases, final-test accuracy drops from $57.89\%$ to $56.02\%$. Our findings suggest that for multi-image medical reasoning, defining the correct agentic decision rule is substantially more impactful than expanding the optimization search budget.

Figures

Figures reproduced from arXiv: 2607.27564 by Jianyi Hao, Site Li, Xiaofeng Liu.

Figure 1
Figure 1. Figure 1: A real MedFrameQA example optimized for single [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Inference-time control structures for the three most important agentic strategies. The crucial difference is not prompt length but [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: End-to-end protocol. Search uses a fixed [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Zoom-in of the intra-run search loop. Evolution edits only the strategy-specific config block, scores candidates on [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Representative 50-generation search progress. Each panel overlays the noisy raw search trace with a bold best-so-far curve, [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Final-test comparison between the 50-generation repeat [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Representative final-test change from generation 0 to the selected generation. The strongest final method is not the one with the [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Raw representative search trajectories for the three main-table methods. The appendix keeps the noisy view, including invalid [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Selection-holdout versus final-test means for the three [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Three case cards drawn from representative selected-run final-test predictions. Case A is a positive example where Order-Vote [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 22 linked inside Pith

  1. [1]

    Chen, Freya Gulamali, and Shalmali Joshi

    Monica Agrawal, Irene Y . Chen, Freya Gulamali, and Shalmali Joshi. 2025. The evaluation illusion of large language models in medicine.npj Digital Medicine. 2, 8

    2025

  2. [2]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Matthew Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tianhe Han, Zhitao Gong, Sina Samangooei, Maria Mon- teiro, Jacob Menick, Sebastian Borgeaud, and 1 oth- ers. 2022. Flamingo: a visual language model for few- shot ...

    Pith/arXiv arXiv 2022

  3. [3]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localiza- tion, text reading, and beyond.arXiv preprint arXiv:2308.12966. 2

    Pith/arXiv arXiv 2023

  4. [4]

    Zhihong Chen, Maya Varma, Justin Xu, Mag- dalini Paschali, Dave Van Veen, Andrew Johnston, Alaa Youssef, Louis Blankemeier, Christian Blueth- gen, Stephan Altmayer, Jeya Maria Jose Valanarasu, and Mohamed Siddig Eltayeb Muneer. 2024. A vision-language foundation model to enhance effi- ciency of chest x-ray interpretation.arXiv preprint arXiv:2401.12208. 2

    Pith/arXiv arXiv 2024

  5. [5]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. Instructblip: To- wards general-purpose vision-language models with instruction tuning. InAdvances in Neural Information Processing Systems. 2

    2023

  6. [6]

    Xing, and Zhiting Hu

    Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yi- han Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. 2022. Rlprompt: Opti- mizing discrete text prompts with reinforcement learn- ing. InProceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing. 2, 3

    2022

  7. [7]

    Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt ¨aschel

  8. [8]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji

  9. [9]

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2023. Connecting large language models with evolutionary algorithms yields powerful prompt opti- mizers.arXiv preprint arXiv:2309.08532. 2, 3

    Pith/arXiv arXiv 2023

  10. [10]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394. 2

    Pith/arXiv arXiv

  11. [11]

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. 2024. Omn- imedvqa: A new large-scale comprehensive evalu- ation benchmark for medical lvlm.arXiv preprint arXiv:2402.09181. 2

    Pith/arXiv arXiv 2024

  12. [12]

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286. 2, 3

    Pith/arXiv arXiv 2020

  13. [13]

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. 2024. Mantis: Inter- leaved multi-image instruction tuning.arXiv preprint arXiv:2405.01483. 1, 2

    Pith/arXiv arXiv 2024

  14. [14]

    Jeong, Saurabh Garg, Zachary C

    Daniel P. Jeong, Saurabh Garg, Zachary C. Lipton, and Michael Oberst. 2024. Medical adaptation of large language and vision-language models: Are we making progress? InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2, 8

    2024

  15. [15]

    Omar Khattab, Arnav Singhvi, Parth Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Heather Moazam, and 1 others. 2024. Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714. 2, 3

    Pith/arXiv arXiv 2024

  16. [16]

    Chen, and An- drew Y

    Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muham- mad Ahmed Chaudhry, Jonathan H. Chen, and An- drew Y . Ng. 2024. Many-shot in-context learning in multimodal foundation models.arXiv preprint arXiv:2405.09798. 1

    Pith/arXiv arXiv 2024

  17. [17]

    Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin. 2025. Shinkaevolve: Towards open-ended and sample-efficient program evolution.arXiv preprint arXiv:2509.19349. 2, 3, 5

    Pith/arXiv arXiv 2025

  18. [18]

    Minchan Kwon, Gaeun Kim, Jongsuk Kim, Haeil Lee, and Junmo Kim. 2024. Stableprompt: Automatic prompt tuning using reinforcement learning for large language models. InProceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing. 2, 3

    2024

  19. [19]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Bench- marking multimodal llms with generative comprehen- sion.arXiv preprint arXiv:2307.16125. 2

    Pith/arXiv arXiv 2023

  20. [20]

    Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman

    Jason J. Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiol- ogy images.Scientific Data. 2, 3

    2018

  21. [21]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. InProceedings of the 40th In- ternational Conference on Machine Learning. 2

    2023

  22. [22]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. 2023. Llava- med: Training a large language-and-vision assis- tant for biomedicine in one day.arXiv preprint arXiv:2306.00890. 2

    Pith/arXiv arXiv 2023

  23. [23]

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual ques- tion answering.arXiv preprint arXiv:2102.09542. 2, 3

    Pith/arXiv arXiv 2021

  24. [24]

    Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li, Xiaohui Song, and 1 others. 2025. Healthgpt: A medical large vision-language model for unifying comprehension and generation via heteroge- neous knowledge adaptation. InProceedings of the 42nd International Conference on Machine Learning. 2

    2025

  25. [25]

    Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Tianshuo Yang, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, and 1 others. 2025. Mmiu: Multi- modal multi-image understanding for evaluating large vision-language models. InThe Thirteenth Interna- tional Conference on Learning Representations. 1, 2

    2025

  26. [26]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning.arXiv preprint arXiv:2304.08485. 2

    Pith/arXiv arXiv 2023

  27. [27]

    gradient descent

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chen- guang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with “gradient descent” and beam search. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing. 2, 3

    2023

  28. [28]

    Michael Moor, Qian Huang, Shirley Wu, Michi- hiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, and Jure Leskovec

  29. [29]

    Med-flamingo: a multimodal medical few-shot learner.arXiv preprint arXiv:2307.15189. 2

    Pith/arXiv arXiv

  30. [30]

    Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2024. Mlevlm: Improve multi-level progressive capabilities based on multi- modal large language model for medical visual ques- tion answering. InFindings of the Association for Computational Linguistics: ACL 2024. 2

    2024

  31. [31]

    Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, and 1 others

    Fei Wang, Xingyu Fu, James Y . Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, and 1 others. 2025. Muir- bench: A comprehensive benchmark for robust multi- image understanding. InThe Thirteenth International Conference on Learning Representations. 1, 2

    2025

  32. [32]

    Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang, Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li, and 1 oth- ers. 2025. Mmie: Massive multimodal interleaved comprehension benchmark for large vision-language models. InThe Thirteenth International Conference on Learning Representations. 1, 2

    2025

  33. [33]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2023. Mmmu: A massive multi-discipline multimodal un- derstanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502. 2

    Pith/arXiv arXiv 2023

  34. [34]

    Le, Denny Zhou, and Xinyun Chen

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V . Le, Denny Zhou, and Xinyun Chen

  35. [35]

    Davison, Hui Ren, and 1 others

    Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D. Davison, Hui Ren, and 1 others. 2023. Biomedgpt: A generalist vision-language foundation model for diverse biomedical tasks.arXiv preprint arXiv:2305.17100. 2

    Pith/arXiv arXiv 2023

  36. [36]

    Suhao Yu, Haojin Wang, Juncheng Wu, Luyang Luo, Jingshen Wang, Cihang Xie, Pranav Rajpurkar, Carl Yang, Yang Yang, Kang Wang, Yannan Yu, and Yuyin Zhou. 2025. Medframeqa: A multi-image medical vqa benchmark for clinical reasoning.arXiv preprint arXiv:2505.16964. 1, 2, 3

    arXiv 2025

  37. [37]

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multimodal chain-of-thought reasoning in language models.arXiv preprint arXiv:2302.00923. 1

    Pith/arXiv arXiv 2023

  38. [38]

    differentiation

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. Textgrad: Automatic “differentiation” via text.arXiv preprint arXiv:2406.07496. 2, 3

    Pith/arXiv arXiv 2024

  39. [40]

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weix- iong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. Pmc-vqa: Visual instruction tuning for medical visual question answering.arXiv preprint arXiv:2305.10415. 2

    Pith/arXiv arXiv 2023

  40. [42]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. InThe Eleventh International Con- ference on Learning Representations. 2, 3 A. Appendix A.1. Budget Sensitivity: 50 vs. 100 Generations Budget setting Holdout (%) Final (%) Invalid (%) Order-...

    2023

  41. [2023]

    Promptbreeder: Self-referential self- improvement via prompt evolution.arXiv preprint arXiv:2309.16797. 2, 3

    Pith/arXiv arXiv

  42. [2024]

    InThe Twelfth International Conference on Learning Repre- sentations

    Large language models as optimizers. InThe Twelfth International Conference on Learning Repre- sentations. 2, 3