Pith. sign in

REVIEW 3 major objections 5 minor 77 references

Text-only prompt optimizers never see the images that caused failures; giving them those images only at train time produces better ordinary text prompts at no extra inference cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 17:17 UTC pith:ZBIFWSXZ

load-bearing objection Solid systems fix for a real APO blind spot; headline “ranks first everywhere” is a bit stronger than the noise bars support, but the mechanism and ablations hold. the 3 major comments →

arxiv 2607.24354 v1 pith:ZBIFWSXZ submitted 2026-07-27 cs.AI

Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization

Haoyue Liu , Xiaoyu Ma , Ye Chen , Yuexian Zou , Xiaoying Tang This is my paper
classification cs.AI
keywords automatic prompt optimizationvision-language modelsvisual question answeringcross-modal feedbackvisual blind spotstext-only deploymentprompt transfer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Automatic prompt optimization is widely used to adapt vision-language models without changing their weights, but on multimodal tasks the optimizer only ever sees the question, the wrong answer, and the gold label. It never sees the image, so it cannot tell whether the model missed a small axis label, confused an anatomical slice, or leaned on a global scene prior. This paper argues that the bottleneck is missing visual evidence, not the search algorithm. It proposes Cross-Modal Visual Feedback (CMVF): a stronger model inspects each failed image without labels, then aggregates those observations into reusable visual blind-spot patterns that rewrite a plain text prompt. The image is used only during optimization; deployment stays a normal text prompt with unchanged cost. Across twelve VQA benchmarks and four target models the method ranks first, averaging a 2.4-point gain over the strongest baseline and producing expert-style visual checklists that transfer across models.

Core claim

The effectiveness of automatic prompt optimization on multimodal tasks is limited by a blind feedback channel: the optimizer never sees the input image on which the target failed, so it cannot diagnose visually grounded errors. Closing that channel with failure-conditioned, label-free visual diagnosis plus error-aware aggregation into task-level blind-spot patterns produces ordinary text prompts that consistently outperform strong text-only APO baselines across twelve VQA datasets and four target VLMs, with average gains of 2.4 points (up to 6.5) and no added inference cost.

What carries the argument

Cross-Modal Visual Feedback (CMVF): a two-stage visual-gradient channel in which a stronger optimizer VLM first produces question-aware descriptions of failed images with predictions and labels withheld, then aggregates those descriptions with error triples into compact reusable visual blind-spot instructions that drive a standard text-prompt rewrite.

Load-bearing premise

A stronger vision-language model that looks only at failed images and questions, without seeing answers, can extract recurring task-level visual blind spots rather than noise or model-specific excuses.

What would settle it

Run the same optimization pipeline with the image channel ablated or replaced by generic captions under a matched token budget: if mean accuracy and the expert-style checklist content no longer improve over strong text-only baselines on perception-heavy sets such as RealWorldQA, TextVQA, and medical VQA, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Multimodal APO should treat visual evidence as first-class feedback rather than relying only on textual error triples.
  • Deployed artifacts can remain pure text prompts, so existing serving stacks gain accuracy without extra VLM calls at test time.
  • Optimizers can self-organize into transferable expert-style visual checklists (slice level, axes/legends, small text, local vs global priors) without hand-written rules.
  • Gains concentrate where failures are visually grounded; tasks with little visual error should see little benefit.
  • Prompts optimized on one model family can transfer to another without re-optimization when they encode task-level visual strategies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same train-time-only visual channel could be tried on other multimodal settings where prompts are discrete and inference cost is constrained, such as document or chart agents.
  • If aggregation is what turns noisy per-image notes into reusable checklists, similar compress-then-rewrite steps may help text-only APO that currently overfits isolated error triples.
  • Weaker target models appear to leave more headroom for the visual channel; the method may be most useful precisely where base accuracy is limited by perception rather than reasoning.
  • Label-free Stage-1 description is doing double duty as both anti-leakage and anti-rationalization; removing either property would be a sharp follow-up ablation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript identifies a structural weakness of text-only automatic prompt optimization (APO) applied to multimodal tasks: the optimizer's feedback channel contains the question, prediction, and gold answer, but never the image on which the target VLM failed. The proposed remedy, Cross-Modal Visual Feedback (CMVF), adds a two-stage visual channel used only during optimization: (1) a label-free, question-aware description of each failed image by a stronger optimizer VLM (Eq. 2), and (2) error-aware aggregation of up to 30 failed cases into reusable task-level "visual blind-spot" patterns that drive a TextGrad-style (or REVOLVE-style, CMVF*) prompt rewrite (Eqs. 3–4). The deployed artifact is an ordinary text prompt with no added inference cost. Across 12 VQA datasets and 4 target VLMs, CMVF reports the best mean accuracy on every target (+2.4 pp over the strongest baseline on average, up to +6.5 pp on RealWorldQA), token-budget-matched ablations (CM-Blind / CM-Caption / CM-Vis / full) attribute the gain to image-aware perception plus aggregation rather than feedback length, and optimized prompts transfer to a new target model without re-optimization (Table 4).

Significance. If the results hold, the contribution is a practical and cleanly motivated one: it identifies a structural limitation of text-only APO on multimodal tasks, fixes it with a train-time-only visual channel, and ships an artifact (a text prompt) with zero inference-cost overhead — a deployability property most multimodal prompting methods (e.g., MPO) do not have. The paper ships genuinely falsifiable evidence: a monotone, token-budget-matched ablation ladder (Blind < Caption < Vis < full) replicated on three targets, per-dataset standard deviations, exact McNemar tests on key pairs, cross-model transfer measurements, and a promised release of code and optimized prompts. The qualitative finding that the optimizer converges on domain-plausible visual checklists from a deliberately broad Stage-1 prompt is interesting independently of the accuracy numbers. The work extends IPO's recipe to generative VQA with a clearer failure-conditioned design. Significance is solid but incremental: the gain magnitudes (+1.5 to +3.1 pp on target means) are useful rather than dramatic, and the core claim's strength currently exceeds what the reported statistics establish.

major comments (3)
  1. [§3.2, Table 1] Table 1 / §3.2: the headline 'ranks first on every target' is statistically under-supported. On Phi-3.5-vision, CMVF (55.9) ties CMVF* (55.8) within 0.1 and loses per-dataset on CT and X-Ray; on Qwen2.5-VL the margin over the best baseline is +1.5 (the text says 1.6), while per-dataset stds over 3 seeds are ±1–3 (PathVQA ±2.2, X-Ray ±3.0) and no std is reported for the 8-benchmark mean. McNemar tests are shown for only 3 of the 32 (target, dataset) pairs, and those three appear to be the largest-gain cells, with no multiple-comparison correction. Combined with validation-best selection over 20 steps on 150 examples, the 'every target' wording is not established. Please report significance for all pairs (or correct across them), give the std/CI of the mean, and temper the headline accordingly.
  2. [§3.4, Table 4] Table 4 / §3.4: the transfer claim compares transferred CMVF prompts only against TextGrad optimized directly on Qwen3.5-4B (70.3), never against *transferred* TextGrad or REVOLVE prompts. If a text-only optimized prompt also transfers +1–2 pp across models (plausible, since generic 'expert-style' instructions are known to be somewhat model-agnostic), then transfer is a property of optimized prompts generally, not evidence that Stage-2 aggregation surfaces reusable *visual* blind-spot patterns. This comparison is load-bearing for contribution 3 and is cheap to run: the optimized baseline prompts already exist from Table 1, so transferring them requires inference only. Please add transferred-baseline rows to Table 4.
  3. [§3.1 / §2.3 (Eq. 2)] The optimizer VLM (Qwen3.5-9B) is a stronger member of the same family as two of the four targets (Qwen3.5-4B, Qwen2.5-VL-7B). Stage-1 diagnosis quality (Eq. 2, I_V in Eq. 5) plausibly depends on the optimizer's own perception ability on the failed images, and same-family pairing could inflate the diagnosis-target alignment. The cross-family LLaVA/Phi results mitigate this, but the paper never analyzes sensitivity to the choice of M_O. At minimum, the text should state this dependence explicitly and discuss how gains scale with optimizer strength; ideally one swap of M_O (e.g., a non-Qwen VLM) would be reported.
minor comments (5)
  1. [§3.2 vs Table 1] Text in §3.2 reports the Qwen2.5-VL-7B gain as +1.6, but Table 1 shows 60.1 vs 58.6 = +1.5. Also the abstract's '+2.4 points on average' appears to be (3.0+1.5+3.1+2.1)/4 = 2.4 only if 1.5 is used inconsistently — please reconcile.
  2. [Table 1] OPRO on Qwen3.5-4B ChartQA is 25.1 ± 28.4, an enormous variance that suggests protocol instability for that (method, dataset) cell; since OPRO is not the comparator for the headline claims this is not load-bearing, but a one-line remark would help readers trust the table.
  3. [Table 5] Table 5's columns for Qwen2.5-VL-7B and LLaVA-1.6-7B are labeled only 'RWQA Text'; clarify in the caption that these are RealWorldQA and TextVQA accuracies under the official protocol, and state whether they are single-run or 3-seed means.
  4. [§2.3 / §3.3] The Stage-1 meta-prompt π_vis is said to be in supplementary material; given that the 'self-organization' claim in §3.3 rests on π_vis not mentioning axes, small text, or slice levels, the full prompt text should be in the main appendix and quoted where the claim is made.
  5. [§3.2] The phrase 'Table 1 restores the full comparison' (§3.2) reads like a rebuttal artifact; rephrase for the standalone manuscript.

Circularity Check

0 steps flagged

No significant circularity: empirical APO method evaluated on held-out tests; gains are not forced by definition or self-citation.

full rationale

CMVF is an empirical systems paper, not a first-principles derivation. The optimization objective (Eq. 1) is standard held-out accuracy; Stage-1 visual descriptions (Eq. 2) deliberately withhold labels/predictions; Stage-2 aggregation (Eq. 3) and the fused feedback (Eq. 4) are procedural design choices, not quantities defined in terms of the reported test metrics. Equation (5) is explicitly labeled modeling intuition, not a proved identity. Accuracy claims rest on external test sets, independent baselines (ProTeGi, OPRO, DSPy, TextGrad, REVOLVE), token-budget-matched ablations, and cross-model transfer—none of which reduce by construction to fitted inputs. The sole author self-citation (Liu et al. 2026 on prompt factorization) appears only in Related Work and is not load-bearing for the visual-feedback claim or the ranking results. No uniqueness theorem, smuggled ansatz, or renamed known law underpins the central result. Circularity score is therefore 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central claim rests on standard empirical ML assumptions plus a small number of design choices (optimizer strength, failure-only inspection, aggregation budget, TextGrad-style rewrite). No new physical entities or free parameters fitted to produce a theoretical constant are introduced; the free parameters are ordinary experimental knobs.

free parameters (3)
  • max wrong cases per aggregation step = 30
    Aggregator receives at most 30 wrong cases per step with field-level truncation; chosen by authors, affects the visual signal density.
  • optimization steps T = 20
    Fixed to 20 steps for all methods; controls how long the visual channel can act.
  • train/val split size = 150/150
    150 training and 150 validation samples per benchmark; small relative to full test sets and may affect stability of discovered checklists.
axioms (4)
  • domain assumption A stronger optimizer VLM can extract question-relevant visual evidence from an image without access to the label or prediction.
    Stage 1 (Eq. 2) and the label-free design rest on this; if the optimizer cannot perceive the relevant features, g_vis is noise.
  • domain assumption Recurring visual failure modes of a target VLM can be compressed into short natural-language instructions that improve a weaker target when inserted into its prompt.
    Load-bearing for both accuracy gains and the transfer claim (Section 3.4).
  • domain assumption Textual-gradient / REVOLVE-style prompt rewrites are a valid Update operator that can incorporate the fused visual feedback string.
    Section 2.4; CMVF* shows the channel is not tied to one rewriter.
  • standard math Standard exact-match / synonym scoring protocols of the 12 VQA benchmarks are adequate external measures of success.
    Evaluation protocol in Section 3.1; conventional in the VQA literature.
invented entities (2)
  • Cross-Modal Visual Feedback (CMVF) two-stage visual-gradient channel no independent evidence
    purpose: Name for the failure-conditioned diagnosis + error-aware aggregation pipeline that produces g_vis and g_CM.
    Procedural construct, not a physical entity; defined by Eqs. 2–4 and Algorithm 1. No independent existence claimed outside the method.
  • Reusable task-level visual blind-spot patterns / expert-style visual checklists no independent evidence
    purpose: The compressed linguistic output of Stage-2 aggregation that is written into the deployed prompt.
    Emergent from the optimizer rather than hand-specified; evidence is qualitative prompt inspection (Table 3) and transfer (Table 4).

pith-pipeline@v1.2.0-grok45-kimik3 · 20095 in / 3150 out tokens · 50864 ms · 2026-07-31T17:17:53.736993+00:00 · methodology

0 comments
read the original abstract

Automatic prompt optimization (APO) has been widely adopted to adapt vision-language models (VLMs) to downstream tasks without weight updates, yielding promising results. However, on multimodal tasks, the effectiveness of APO is fundamentally bottlenecked by a blind feedback channel: the optimizer reads the question, the prediction, and the gold answer, but never the input image on which the model failed, and therefore cannot diagnose visually grounded errors. As a remedy, we introduce Cross-Modal Visual Feedback (CMVF). CMVF incorporates (1) a failure-conditioned visual diagnosis stage, in which a stronger optimizer VLM inspects each failed image without access to predictions or labels, and (2) an error-aware aggregation stage that compresses these observations into reusable, task-level visual blind-spot patterns that drive the prompt rewrite. Crucially, the image is consumed only during optimization; the deployed artifact is an ordinary text prompt that runs at the same inference cost as any text-only baseline. Extensive results across 12 VQA datasets and 4 target VLMs demonstrate that CMVF consistently ranks first, improving over the strongest baseline on every target by 2.4 points on average, with gains of up to 6.5 points on individual benchmarks. Moreover, the optimizer self-organizes into expert-style visual checklists that transfer across models without re-optimization.

Figures

Figures reproduced from arXiv: 2607.24354 by Haoyue Liu, Xiaoying Tang, Xiaoyu Ma, Ye Chen, Yuexian Zou.

Figure 1
Figure 1. Figure 1: Motivation for CMVF. Existing text-only APO [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: CMVF optimization pipeline. The target VLM is evaluated on training examples; the optimizer VLM inspects failed [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Validation accuracy over 20 optimization steps on eight Phi-3.5-vision benchmarks (top: MRI, X-Ray, VQA-RAD, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Generalization gap (validation-best accuracy minus [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 4 linked inside Pith

  1. [2]

    Chen, L.; Chen, J.; Goldstein, T.; Huang, H.; and Zhou, T. 2024. InstructZero : Efficient Instruction Optimization for Black-Box Large Language Models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 6503--6518. PMLR

    2024

  2. [3]

    Choi, Y.; Kim, D.; Baek, J.; and Hwang, S. J. 2026. Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs. In International Conference on Learning Representations

    2026

  3. [4]

    P.; and Hu, Z

    Deng, M.; Wang, J.; Hsieh, C.-P.; Wang, Y.; Guo, H.; Shu, T.; Song, M.; Xing, E. P.; and Hu, Z. 2022. RLPrompt : Optimizing Discrete Text Prompts with Reinforcement Learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3369--3391

    2022

  4. [5]

    Du, Y.; Sun, W.; and Snoek, C. G. 2024. IPO : Interpretable Prompt Optimization for Vision-Language Models. Advances in Neural Information Processing Systems, 37: 126725--126766

    2024

  5. [6]

    S.; Michalewski, H.; Osindero, S.; and Rockt \"a schel, T

    Fernando, C.; Banarse, D. S.; Michalewski, H.; Osindero, S.; and Rockt \"a schel, T. 2024. Promptbreeder : Self-Referential Self-Improvement via Prompt Evolution. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 13481--13544. PMLR

    2024

  6. [7]

    Guo, Q.; Wang, R.; Guo, J.; Li, B.; Song, K.; Tan, X.; Liu, G.; Bian, J.; and Yang, Y. 2024. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers. In International Conference on Learning Representations

    2024

  7. [8]

    He, X.; Zhang, Y.; Mou, L.; Xing, E.; and Xie, P. 2020. PathVQA : 30000+ Questions for Medical Visual Question Answering. arXiv preprint arXiv:2003.10286

    Pith/arXiv arXiv 2020

  8. [9]

    Jia, M.; Tang, L.; Chen, B.-C.; Cardie, C.; Belongie, S.; Hariharan, B.; and Lim, S.-N. 2022. Visual prompt tuning. In European Conference on Computer Vision, 709--727. Springer

    2022

  9. [10]

    Kembhavi, A.; Salvato, M.; Kolve, E.; Seo, M.; Hajishirzi, H.; and Farhadi, A. 2016. A diagram is worth a dozen images. In European Conference on Computer Vision, 235--251. Springer

    2016

  10. [11]

    T.; Moazam, H.; Miller, H.; Zaharia, M.; and Potts, C

    Khattab, O.; Singhvi, A.; Maheshwari, P.; Zhang, Z.; Santhanam, K.; Vardhamanan, S.; Haq, S.; Sharma, A.; Joshi, T. T.; Moazam, H.; Miller, H.; Zaharia, M.; and Potts, C. 2024. DSPy : Compiling Declarative Language Model Calls into State-of-the-Art Pipelines. In International Conference on Learning Representations

    2024

  11. [12]

    U.; Rasheed, H.; Maaz, M.; Khan, S.; and Khan, F

    Khattak, M. U.; Rasheed, H.; Maaz, M.; Khan, S.; and Khan, F. S. 2023. MaPLe : Multi-Modal Prompt Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19113--19122

    2023

  12. [13]

    J.; Gayen, S.; Ben Abacha, A.; and Demner-Fushman, D

    Lau, J. J.; Gayen, S.; Ben Abacha, A.; and Demner-Fushman, D. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific Data, 5(1): 180251

    2018

  13. [14]

    Li, C.; Wong, C.; Zhang, S.; Usuyama, N.; Liu, H.; Yang, J.; Naumann, T.; Poon, H.; and Gao, J. 2023. LLaVA-Med : Training a Large Language-and-Vision Assistant for Biomedicine in One Day. Advances in Neural Information Processing Systems, 36: 28541--28564

    2023

  14. [15]

    Liu, B.; Zhan, L.-M.; Xu, L.; Ma, L.; Yang, Y.; and Wu, X.-M. 2021. SLAKE : A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), 1650--1654. IEEE

    2021

  15. [16]

    Liu, H.; Wang, Z.; Guo, Y.; Shou, H.; and Tang, X. 2026. Adaptive Prompt Structure Factorization: A Framework for Self-Discovering and Optimizing Compositional Prompt Programs. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 11690--11714. Association for Computational Linguistics

    2026

  16. [17]

    Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.-C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507--2521

    2022

  17. [18]

    P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P

    Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Gupta, S.; Majumder, B. P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P. 2023. Self-Refine: Iterative Refinement with Self-Feedback. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 46534--46594

    2023

  18. [19]

    X.; Tan, J

    Masry, A.; Long, D. X.; Tan, J. Q.; Joty, S.; and Hoque, E. 2022. ChartQA : A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, 2263--2279

    2022

  19. [20]

    Mathew, M.; Karatzas, D.; and Jawahar, C. 2021. DocVQA : A Dataset for VQA on Document Images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2200--2209

    2021

  20. [21]

    Ou, Y.; Zhou, W.; Ding, S.; Li, L.; Wu, J.; Wang, T.; Chen, J.; Wang, S.; Xu, X.; Zhang, N.; Chen, H.; and Jiang, Y. E. 2025. Symbolic learning enables self-evolving agents. AI Open, 6: 314--322

    2025

  21. [22]

    Pryzant, R.; Iter, D.; Li, J.; Lee, Y.; Zhu, C.; and Zeng, M. 2023. Automatic Prompt Optimization with ``Gradient Descent'' and Beam Search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 7957--7968

    2023

  22. [23]

    Sclar, M.; Choi, Y.; Tsvetkov, Y.; and Suhr, A. 2024. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. In International Conference on Learning Representations

    2024

  23. [24]

    Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 8634--8652

    2023

  24. [25]

    Singh, A.; Natarajan, V.; Shah, M.; Jiang, Y.; Chen, X.; Batra, D.; Parikh, D.; and Rohrbach, M. 2019. Towards VQA Models That Can Read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8317--8326

    2019

  25. [26]

    Wang, P.; Bai, S.; Tan, S.; Wang, S.; Fan, Z.; Bai, J.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; et al. 2024. Qwen2-VL : Enhancing Vision-Language Model's Perception of the World at Any Resolution. arXiv preprint arXiv:2409.12191

    Pith/arXiv arXiv 2024

  26. [27]

    xAI . 2024. RealWorldQA . https://huggingface.co/datasets/xai-org/RealworldQA. Dataset on Hugging Face; accessed 2026-07-24

    2024

  27. [28]

    V.; Zhou, D.; and Chen, X

    Yang, C.; Wang, X.; Lu, Y.; Liu, H.; Le, Q. V.; Zhou, D.; and Chen, X. 2024. Large language models as optimizers. In International Conference on Learning Representations

    2024

  28. [29]

    Yue, X.; Ni, Y.; Zhang, K.; Zheng, T.; Liu, R.; Zhang, G.; Stevens, S.; Jiang, D.; Ren, W.; Sun, Y.; Wei, C.; Yu, B.; Yuan, R.; Sun, R.; Yin, M.; Zheng, B.; Yang, Z.; Liu, Y.; Huang, W.; Sun, H.; Su, Y.; and Chen, W. 2024. MMMU : A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI . In Proceedings of the IEEE/CVF Con...

    2024

  29. [30]

    Yuksekgonul, M.; Bianchi, F.; Boen, J.; Liu, S.; Lu, P.; Huang, Z.; Guestrin, C.; and Zou, J. 2025. Optimizing Generative AI by Backpropagating Language Model Feedback. Nature, 639(8055): 609--616

    2025

  30. [31]

    Zhang, P.; Jin, H.; Hu, L.; Li, X.; Kang, L.; Luo, M.; Song, Y.; and Wang, H. 2025. REVOLVE : Optimizing AI Systems by Tracking Response Evolution in Textual Optimization. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, 75216--75233. PMLR

    2025

  31. [32]

    C.; and Liu, Z

    Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022 a . Conditional Prompt Learning for Vision-Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 16816--16825

    2022

  32. [33]

    C.; and Liu, Z

    Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022 b . Learning to Prompt for Vision-Language Models. International Journal of Computer Vision (IJCV), 130(9): 2337--2348

    2022

  33. [34]

    I.; Han, Z.; Paster, K.; Pitis, S.; Chan, H.; and Ba, J

    Zhou, Y.; Muresanu, A. I.; Han, Z.; Paster, K.; Pitis, S.; Chan, H.; and Ba, J. 2023. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations

    2023

  34. [35]

    Advances in Neural Information Processing Systems , volume=

    Language models are few-shot learners , author=. Advances in Neural Information Processing Systems , volume=

  35. [36]

    Chen, Lichang and Chen, Jiuhai and Goldstein, Tom and Huang, Heng and Zhou, Tianyi , booktitle=

  36. [37]

    Proceedings of the 41st International Conference on Machine Learning , volume=

    Fernando, Chrisantha and Banarse, Dylan Sunil and Michalewski, Henryk and Osindero, Simon and Rockt. Proceedings of the 41st International Conference on Machine Learning , volume=

  37. [38]

    International Conference on Learning Representations , year=

    Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers , author=. International Conference on Learning Representations , year=

  38. [39]

    He, Xuehai and Zhang, Yichen and Mou, Luntian and Xing, Eric and Xie, Pengtao , journal=

  39. [40]

    European Conference on Computer Vision , pages=

    Visual prompt tuning , author=. European Conference on Computer Vision , pages=. 2022 , organization=

    2022

  40. [41]

    European Conference on Computer Vision , pages=

    A diagram is worth a dozen images , author=. European Conference on Computer Vision , pages=. 2016 , organization=

    2016

  41. [42]

    and Moazam, Hanna and Miller, Heather and Zaharia, Matei and Potts, Christopher , booktitle=

    Khattab, Omar and Singhvi, Arnav and Maheshwari, Paridhi and Zhang, Zhiyuan and Santhanam, Keshav and Vardhamanan, Sri and Haq, Saiful and Sharma, Ashutosh and Joshi, Thomas T. and Moazam, Hanna and Miller, Heather and Zaharia, Matei and Potts, Christopher , booktitle=

  42. [43]

    Khattak, Muhammad Uzair and Rasheed, Hanoona and Maaz, Muhammad and Khan, Salman and Khan, Fahad Shahbaz , booktitle=

  43. [44]

    Scientific Data , volume=

    A dataset of clinically generated visual questions and answers about radiology images , author=. Scientific Data , volume=. 2018 , publisher=

    2018

  44. [45]

    Li, Chunyuan and Wong, Cliff and Zhang, Sheng and Usuyama, Naoto and Liu, Haotian and Yang, Jianwei and Naumann, Tristan and Poon, Hoifung and Gao, Jianfeng , journal=

  45. [46]

    2021 , organization=

    Liu, Bo and Zhan, Li-Ming and Xu, Li and Ma, Lin and Yang, Yan and Wu, Xiao-Ming , booktitle=. 2021 , organization=

    2021

  46. [47]

    Advances in Neural Information Processing Systems , volume=

    Learn to explain: Multimodal reasoning via thought chains for science question answering , author=. Advances in Neural Information Processing Systems , volume=

  47. [48]

    Masry, Ahmed and Long, Do Xuan and Tan, Jia Qing and Joty, Shafiq and Hoque, Enamul , booktitle=

  48. [49]

    Mathew, Minesh and Karatzas, Dimosthenis and Jawahar, CV , booktitle=

  49. [50]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Automatic Prompt Optimization with ``Gradient Descent'' and Beam Search , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  50. [51]

    International Conference on Learning Representations , year=

    Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting , author=. International Conference on Learning Representations , year=

  51. [52]

    Singh, Amanpreet and Natarajan, Vivek and Shah, Meet and Jiang, Yu and Chen, Xinlei and Batra, Dhruv and Parikh, Devi and Rohrbach, Marcus , booktitle=. Towards

  52. [53]

    arXiv preprint physics/0004057 , year=

    The information bottleneck method , author=. arXiv preprint physics/0004057 , year=

    Pith/arXiv arXiv

  53. [54]

    2024 , publisher=

    Towards generalist biomedical AI , author=. 2024 , publisher=

    2024

  54. [55]

    Advances in Neural Information Processing Systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in Neural Information Processing Systems , volume=

  55. [56]

    International Conference on Learning Representations , year=

    Large language models as optimizers , author=. International Conference on Learning Representations , year=

  56. [57]

    Yang, Zhengyuan and Li, Linjie and Wang, Jianfeng and Lin, Kevin and Azarnasab, Ehsan and Ahmed, Faisal and Liu, Zicheng and Liu, Ce and Zeng, Michael and Wang, Lijuan , journal=

  57. [58]

    International Conference on Learning Representations , year=

    Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs , author=. International Conference on Learning Representations , year=

  58. [59]

    Du, Yingjun and Sun, Wenfang and Snoek, Cees G , journal=

  59. [60]

    Yue, Xiang and Ni, Yuansheng and Zhang, Kai and Zheng, Tianyu and Liu, Ruoqi and Zhang, Ge and Stevens, Samuel and Jiang, Dongfu and Ren, Weiming and Sun, Yuxuan and Wei, Cong and Yu, Botao and Yuan, Ruibin and Sun, Renliang and Yin, Ming and Zheng, Boyuan and Yang, Zhenzhu and Liu, Yibo and Huang, Wenhao and Sun, Huan and Su, Yu and Chen, Wenhu , booktitle=

  60. [61]

    Optimizing Generative

    Yuksekgonul, Mert and Bianchi, Federico and Boen, Joseph and Liu, Sheng and Lu, Pan and Huang, Zhi and Guestrin, Carlos and Zou, James , journal=. Optimizing Generative. 2025 , publisher=

    2025

  61. [62]

    Zhang, Peiyan and Jin, Haibo and Hu, Leyang and Li, Xinnuo and Kang, Liying and Luo, Man and Song, Yangqiu and Wang, Haohan , booktitle=

  62. [63]

    2025 , doi=

    A Multimodal Biomedical Foundation Model Trained from Fifteen Million Image--Text Pairs , author=. 2025 , doi=

    2025

  63. [64]

    Zhang, Xiaoman and Wu, Chaoyi and Zhao, Ziheng and Lin, Weixiong and Zhang, Ya and Wang, Yanfeng and Xie, Weidi , journal=

  64. [65]

    AI Open , volume=

    Symbolic learning enables self-evolving agents , author=. AI Open , volume=. 2025 , publisher=

    2025

  65. [66]

    The Eleventh International Conference on Learning Representations , year=

    Large language models are human-level prompt engineers , author=. The Eleventh International Conference on Learning Representations , year=

  66. [67]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Adaptive Prompt Structure Factorization: A Framework for Self-Discovering and Optimizing Compositional Prompt Programs , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=. 2026 , publisher=

    2026

  67. [68]

    doi:10.18653/v1/2022.emnlp-main.222 , year=

    Deng, Mingkai and Wang, Jianyu and Hsieh, Cheng-Ping and Wang, Yihan and Guo, Han and Shu, Tianmin and Song, Meng and Xing, Eric P and Hu, Zhiting , booktitle=. doi:10.18653/v1/2022.emnlp-main.222 , year=

    doi:10.18653/v1/2022.emnlp-main.222 2022

  68. [69]

    doi:10.18653/v1/2023.eacl-main.277 , year=

    Prasad, Archiki and Hase, Peter and Zhou, Xiang and Bansal, Mohit , booktitle=. doi:10.18653/v1/2023.eacl-main.277 , year=

    doi:10.18653/v1/2023.eacl-main.277 2023

  69. [70]

    Wang, Xinyuan and Li, Chenxi and Wang, Zhen and Bai, Fan and Luo, Haotian and Zhang, Jiayou and Jojic, Nebojsa and Xing, Eric P and Hu, Zhiting , booktitle=

  70. [71]

    International Journal of Computer Vision (IJCV) , volume=

    Learning to Prompt for Vision-Language Models , author=. International Journal of Computer Vision (IJCV) , volume=. doi:10.1007/s11263-022-01653-1 , year=

    doi:10.1007/s11263-022-01653-1

  71. [72]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

    Conditional Prompt Learning for Vision-Language Models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

  72. [73]

    Advances in Neural Information Processing Systems (NeurIPS) , volume=

    Self-Refine: Iterative Refinement with Self-Feedback , author=. Advances in Neural Information Processing Systems (NeurIPS) , volume=

  73. [74]

    Advances in Neural Information Processing Systems (NeurIPS) , volume=

    Reflexion: Language Agents with Verbal Reinforcement Learning , author=. Advances in Neural Information Processing Systems (NeurIPS) , volume=

  74. [75]

    Transactions on Machine Learning Research (TMLR) , year=

    Multimodal Chain-of-Thought Reasoning in Language Models , author=. Transactions on Machine Learning Research (TMLR) , year=

  75. [76]

    2025 , url=

    Bai, Shuai and Chen, Keqin and Liu, Xuejing and Wang, Jialin and Ge, Wenbin and Song, Sibo and Dang, Kai and Wang, Peng and Wang, Shijie and Tang, Jun and Zhong, Humen and Zhu, Yuanzhi and Yang, Mingkun and Li, Zhaohai and Wan, Jianqiang and Wang, Pengfei and Ding, Wei and Fu, Zheren and Xu, Yiheng and Ye, Jiabo and Zhang, Xi and Xie, Tianbao and Cheng, Z...

    2025

  76. [77]

    Wang, Peng and Bai, Shuai and Tan, Sinan and Wang, Shijie and Fan, Zhihao and Bai, Jinze and Chen, Keqin and Liu, Xuejing and Wang, Jialin and Ge, Wenbin and others , journal=

  77. [78]

    arXiv preprint arXiv:2404.14219 , year=

    Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone , author=. arXiv preprint arXiv:2404.14219 , year=

    Pith/arXiv arXiv