Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

Minor wording changes to a question — a few added sentences, a format switch, a persona — shift large multimodal models' accuracy by up to 15 points on visual multiple-choice benchmarks, with the most instruction-tuned models shifting most.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A study of 61 prompt variants across 10 vision-language models and 3 benchmarks finds accuracy swings of up to 15 points, with proprietary models more sensitive than open-source ones.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Promptception delivers a genuinely useful prompt taxonomy and a real sensitivity effect, but the proprietary-vs-open-source gap is partly a sample-size artifact and needs a matched re-analysis before the headline claim is trusted. the 4 major comments →

arxiv 2509.03986 v1 pith:PJLR7YHM submitted 2025-09-04 cs.CV cs.AIcs.CLcs.LG

Promptception: How Sensitive Are Large Multimodal Models to Prompts?

Mohamed Insaf Ismithdeen , Muhammad Uzair Khattak , Salman Khan This is my paper
classification cs.CV cs.AIcs.CLcs.LG
keywords prompt sensitivitylarge multimodal modelsmultiple-choice question answeringbenchmark stabilityinstruction followingzero-shot evaluationprompt engineeringmodel evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to prove that a large multimodal model's score on a multiple-choice visual benchmark is not a stable number: for the same image, the same question, and the same model, minor changes in the words that wrap the question can move accuracy by up to 15 percentage points. To make that variability measurable, it builds Promptception, a fixed grid of 61 prompt variations in 15 categories — cosmetic formatting changes, structured formats like JSON or Markdown, position shifts, typos, chain-of-thought requests, statistical hints, personas, penalties, rewards, and competitive framing — and runs 10 models of very different sizes and training regimes across three benchmarks covering single images, multiple images, and video. The study's central contrast is between model families: proprietary models (GPT-4o, Gemini 1.5 Pro) follow instruction changes so closely that accuracy swings widely with phrasing, while open-source models stay flatter but also fail to exploit helpful prompts such as chain-of-thought. If the finding holds, then single-prompt leaderboard numbers are fragile artifacts of the chosen wording, and evaluations need prompt-averaged aggregates, which is exactly the trimmed-mean estimator the paper proposes.

Core claim

On its own terms, the discovery is that prompt sensitivity in multimodal multiple-choice QA is large, systematic, and organized by how strongly a model is trained to follow instructions. Across the 61-prompt grid, minor variations produce accuracy deviations of up to 15% for certain prompts and models; for GPT-4o on MMMU-Pro, merely changing the sentence that requests the final answer letter (Category 12) moves accuracy by roughly ten points. Chain-of-thought prompts lift proprietary models by about ten points while leaving open-source models essentially flat. Negative personas can be destructive: asking the model to act as a careless student collapses GPT-4o and Gemini 1.5 Pro accuracy, wit

What carries the argument

Promptception — the paper's own evaluation framework, a fixed catalog of 61 prompt types grouped into 15 categories under 6 supercategories (choice formatting and presentation; linguistic and stylistic challenges; thought process and reasoning; context-aware and ethical guidance; task-specific instructions; performance, feedback, and penalty). Every prompt is appended to the same question-and-choices block on top of an identical minimal baseline instruction ('answer with the option letter from the given choices directly'), so any accuracy difference is attributable to the prompt alone. Two metrics carry the quantitative analysis: the 10%-trimmed mean, which discounts extreme prompts to give

Load-bearing premise

The central comparison — that proprietary models are more prompt-sensitive than open-source ones — assumes both groups were given the same prompts, the same questions, and the same scoring rules; in fact GPT-4o's prompts were modified (dollar signs removed, Appendix J), its safety refusals were excluded instead of counted wrong (Appendix E), and the proprietary models were tested on a 100-video MVBench subset while open-source models used all 4,000 videos (Section 2.3).

What would settle it

Run the full 61-prompt grid under perfectly symmetric conditions on a matched model pair — say InternVL2.5-38B and GPT-4o — on the complete MVBench and MMStar datasets, keeping every character of every prompt identical (including the dollar sign) and scoring every refusal or unparsable response as incorrect. If GPT-4o's standard deviation across the grid falls to the open-source level, the claimed sensitivity gap is an evaluation artifact; if it stays high, the models really do differ in prompt sensitivity.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A single reported accuracy number on an MCQA benchmark carries an intrinsic uncertainty of several points for the same model — up to 15 points in the paper's data — so best-case, carefully selected prompts can overstate and destabilize model comparisons.
  • Model rankings can change depending on which prompt the evaluator happens to use, because the prompt types that help open-source models (short, direct, image-focused) differ from those that help proprietary models (reasoning allowed, explicit answer format).
  • Benchmark reporting should move toward prompt-averaged scores: the paper shows the minimalist baseline is already near-optimal for open-source models but below the grid average for proprietary ones, so a single prompt systematically biases cross-family comparisons.
  • Concrete prompting guidance follows: for open-source models keep prompts terse and plain, avoid JSON/YAML/Markdown structuring, penalties, and competitive framing; for proprietary models allow reasoning room, choose the answer-request format carefully, and avoid negative personas, which can be catastrophic.
  • The Promptception grid doubles as a reusable sensitivity test — any model release can be scored across the 61 prompts and its spread reported alongside mean accuracy as a measure of how much of the result depends on wording.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The proprietary-versus-open-source contrast is not yet a clean model comparison, because the two groups were not evaluated identically: GPT-4o's prompts had the dollar sign removed, its safety refusals were excluded from accuracy rather than counted wrong, and the proprietary models saw only a 100-video MVBench slice. Re-running the grid with identical prompt text and identical scoring for both gr
  • The paper's own mechanism — that instruction-tuning strength drives sensitivity — predicts a smooth gradient: as an open-source family is scaled and instruction-tuned, its prompt-to-prompt spread should rise toward the proprietary level. The paper's InternVL 1B/8B/38B data already hint at this (the 1B model is erratic), so plotting standard deviation against model scale and tuning budget would con
  • Reporting a per-model spread across a prompt grid as a standard companion to accuracy would turn this measurement problem into a routine quality metric: benchmark readers could see at a glance how much of a reported number is prompt luck, and model builders could treat large spreads as a bug to be fixed.
  • The failure patterns — accuracy collapses under statistical hints, answer leakage, and competitive or persona framing — suggest that MCQ items carrying such text measure prompt compliance as much as visual reasoning, so benchmark builders who want capability scores should either strip those framings or control for them explicitly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Promptception, a taxonomy of 61 prompt types organized into 15 categories and 6 supercategories, and uses it to evaluate 8 open-source and 2 proprietary LMMs on three MCQA benchmarks (MMStar, MMMU-Pro, MVBench). The core empirical claim is that minor prompt phrasings produce accuracy deviations up to 15%, that proprietary models are more prompt-sensitive than open-source models, and that separate Prompting Principles are therefore needed for the two model families. The per-prompt accuracy tables (Tables 6–8) and the full prompt list (Appendix A) are potentially reusable assets, and the basic phenomenon—accuracy changes with prompt wording—is directly documented.

Significance. If the central claims hold, the paper has real significance: it would show that LMM benchmark scores are unstable under prompt rewording and that model comparisons can be artifacts of prompt choice. The prompt taxonomy and the per-prompt accuracy tables are a useful contribution and the paper is careful in several respects (e.g., manual verification of answer extraction, a persona ablation in Appendix C, and disclosure of the MVBench subset in Section 2.3). However, the headline proprietary-vs-open-source sensitivity comparison is not yet established because of evaluation asymmetries. The paper is therefore best viewed at present as a useful measurement study whose main interpretive claims require additional evidence or reanalysis.

major comments (4)
  1. [§2.3, Fig 12, Appendix D] The central claim that proprietary models are more prompt-sensitive is confounded by the MVBench subset size. GPT-4o and Gemini 1.5 Pro were evaluated on 100 MVBench items (5 per task), while open-source models used all 4,000. For a binomial accuracy with n=100, the standard error is roughly sqrt(p(1-p)/100) ≈ 5 points; for n=4,000 it is ≈0.8 points. The standard deviations reported in Fig 12 for GPT-4o (4.48) and Gemini 1.5 Pro (5.33) are close to this n=100 noise floor, while the open-source standard deviations (0.67–2.33) are near their smaller noise floor. The paper itself excludes MVBench for proprietary models in Fig 4 because only a subset was used, but Fig 12 and the abstract do not apply the same correction. Appendix D compares only trimmed means and baselines for the open-source full vs subset; it does not report subset standard deviations, so it cannot rule out the artifact. P
  2. [§4.2.2, Appendix J/E] The averaging and scoring protocols treat the two model groups asymmetrically. For open-source models, percentage relative accuracies below 80% are excluded from the averaging (Section 4.2.2); for proprietary models, large negative deviations are instead capped (prompt 9.2 is capped at -15% in Figure 6). In addition, Appendix J states that the '$' symbol was removed from all prompts for GPT-4o only, and Appendix E excludes GPT-4o safety refusals from accuracy rather than counting them as incorrect. These choices change the exact quantities being compared (average PRA and standard deviations) differently for the two groups. The paper should present an analysis with identical prompt text, identical handling of refusals, and with/without exclusion and capping, so that the claimed group difference is not an artifact of the evaluation pipeline.
  3. [§2.5.1/§4.1/§4.2.1/Appendix K] Several thresholds are selected after inspecting the data: the 10% trim rate is 'chosen based on empirical observation' (Section 4.1), the 0.78 high-sensitivity threshold is set to the median of the observed distribution (Appendix K), and the 80% PRA exclusion is discussed in Appendix J. These are post-hoc choices, not independent tests. The qualitative conclusions—e.g., that Categories 2, 6, 7, 8, 9, and 12 are high-sensitivity for open-source models, or that 'all prompt categories' are sensitive for proprietary models—depend on these cutoffs. Please report a sensitivity analysis over trim rates (e.g., 5–20%), exclusion thresholds (e.g., 85–75%), and cutoff values (e.g., 0.5–1.0) and indicate which conclusions reverse.
  4. [Appendix H / Fig 12] The reported standard deviations are not accompanied by uncertainty estimates. Even setting aside the MVBench sampling issue, the differences between GPT-4o (4.48) and open-source models such as InternVL2.5-38B (2.33) could be within sampling noise given only 61 prompt types. The paper should report bootstrap confidence intervals (resampling over prompts and, where relevant, over items) or a significance test before drawing the headline conclusion that proprietary models are more sensitive.
minor comments (4)
  1. [Throughout] There are several typos: 'texual' (Section 1), 'beacause' (Appendix H), 'Proprietery' (Figure 4 legend), 'dicussed' (Section 4.2.2), and 'Y AML' (Table 4).
  2. [§2.2] The reference for Llama-3.2-11B-Vision is given as Patterson et al. 2022, which is a paper on the carbon footprint of ML training, not on Llama-3.2. This is almost certainly the wrong citation.
  3. [Table 4] The 'Prompting Principles' are derived from the same experiments on which the paper evaluates them; they are post-hoc summaries of observed trends rather than independently validated guidelines. Consider reframing them as observations and validating on held-out prompts or datasets.
  4. [Appendix A] The color-coding legend (blue/orange/green/purple) is not accessible in grayscale or for color-blind readers. Add explicit textual labels to each prompt in the table or use symbols.

Circularity Check

0 steps flagged

No significant circularity; the sensitivity measurements are direct and the prompting principles are explicitly derived summaries, not independent predictions.

full rationale

The paper's central claims are direct measurements rather than predictions derived from fitted inputs. Figure 12 computes the standard deviation of measured accuracies across 61 prompts on the external MMStar, MMMU-Pro, and MVBench benchmarks; no parameter is fitted to a subset and then used to forecast the same subset. The Prompting Principles in Table 4 are explicitly described as 'Drawing from our comprehensive prompt sensitivity analysis (section 4)', so they are summaries of observed trends, not independent predictions being validated on the same data; no equation makes a principle equal to its input. The 0.78 median threshold and the 10% trimming rate are data-dependent descriptive choices, but they do not create a fitted-input-called-prediction loop for the central sensitivity claim. The MVBench 100-video subset for proprietary models (Sec 2.3) and the GPT-4o-specific exclusions (Appendix E/J) are genuine evaluation confounds that could bias the proprietary/open-source comparison, as is the unsupported Appendix D claim that the subset 'does not underestimate prompt sensitivity' when only trimmed means are compared; these are correctness risks, not circular reductions, because the measured standard deviation is not equal to the sample size or to the exclusion rule by construction. No load-bearing self-citation or imported uniqueness theorem is used; prior self-citations (e.g., GLAMM, VideoGLAMM in the introduction) are contextual only. Therefore no significant circularity is present.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central measurements rest on hand-chosen thresholds and trimming, plus benchmark/parsing assumptions. The prompt set and taxonomy are author-constructed with no formal coverage guarantee.

free parameters (5)
  • Trimming rate = 10%
    Chosen based on empirical observation (Sec 2.5.1); affects trimmed mean comparisons in Table 2.
  • High-sensitivity threshold = 0.78
    Set to the median of the observed standard-deviation distribution (Sec 4.2.1, Appendix K); determines which categories are called high-sensitivity.
  • PRA exclusion cutoff = 80%
    Open-source prompts with PRA below 80% excluded from averaging (Sec 4.2.2); changes which prompts appear as consistently good/bad in Figures 5/9.
  • MVBench proprietary subset size = 100 videos
    Proprietary models evaluated on 5 videos per task instead of full 4000 (Sec 2.3); Table 10 checks but cannot fully align question sets.
  • Temperature = 0
    Set for proprietary models only (Sec 2.4); open-source models use default sampling, though reported run variance is low.
axioms (5)
  • domain assumption The three benchmarks (MMStar, MMMU-Pro, MVBench) are valid and representative MCQA evaluations of LMM ability.
    The entire sensitivity analysis is built on these benchmarks without questioning their validity.
  • domain assumption GPT-4o mini answer-matching is a reliable oracle for extracting intended answers from free-form responses.
    Stage 2 of the extraction pipeline delegates judgment to GPT-4o mini (Appendix E); no independent verification beyond manual spot checks.
  • domain assumption The 61 prompts and 15-category taxonomy meaningfully sample the space of prompt formulations.
    Prompt set was hand-designed by authors; no coverage argument or prior taxonomy comparison is provided (Section 3).
  • domain assumption Zero-shot evaluation isolates prompt effects without interaction with sampling or few-shot context.
    Sec 2.1 argues zero-shot isolates prompt effects; no few-shot comparison is run to confirm.
  • domain assumption Accuracy standard deviation across prompts is a valid sensitivity measure independent of baseline accuracy level.
    Used throughout Sec 4 and Appendix H; models with different baseline accuracies are compared on raw std without normalization (Fig 12).

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Promptception: How Sensitive Are Large Multimodal Models to Prompts?." pith.science (2026). https://pith.science/paper/PJLR7YHM

@misc{pith2026250903986,
  author       = {Pith},
  title        = {Pith review of: Promptception: How Sensitive Are Large Multimodal Models to Prompts?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PJLR7YHM}},
  note         = {Machine review of arXiv:2509.03986}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Despite the success of Large Multimodal Models (LMMs) in recent years, prompt design for LMMs in Multiple-Choice Question Answering (MCQA) remains poorly understood. We show that even minor variations in prompt phrasing and structure can lead to accuracy deviations of up to 15% for certain prompts and models. This variability poses a challenge for transparent and fair LMM evaluation, as models often report their best-case performance using carefully selected prompts. To address this, we introduce Promptception, a systematic framework for evaluating prompt sensitivity in LMMs. It consists of 61 prompt types, spanning 15 categories and 6 supercategories, each targeting specific aspects of prompt formulation, and is used to evaluate 10 LMMs ranging from lightweight open-source models to GPT-4o and Gemini 1.5 Pro, across 3 MCQA benchmarks: MMStar, MMMU-Pro, MVBench. Our findings reveal that proprietary models exhibit greater sensitivity to prompt phrasing, reflecting tighter alignment with instruction semantics, while open-source models are steadier but struggle with nuanced and complex phrasing. Based on this analysis, we propose Prompting Principles tailored to proprietary and open-source LMMs, enabling more robust and fair model evaluation.

Figures

Figures reproduced from arXiv: 2509.03986 by Mohamed Insaf Ismithdeen, Muhammad Uzair Khattak, Salman Khan.

Figure 1
Figure 1. Figure 1: Categorization of prompts proposed in our [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Sensitivity of state-of-the-art LMMs to prompt variations. Examples from the MMStar benchmark [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Baseline Prompt. This serves as the simplest [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Standard Deviation of prompts within cat [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Average Prompt Performance for Proprietary [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average Prompt Performance for Open￾Source Models. PRA with respect to the Baseline Prompt Accuracy is averaged across Open-source Mod￾els and the 3 Benchmarks (MMStar, MMMU-Pro & MVBench) for each Prompt Type. For proprietary models, the majority of prompts enhanced performance relative to the baseline, with only a few exceptions, namely 4.5, 9.2 (actual drop –50%, capped at –15% for readability), 11.1, 1… view at source ↗
Figure 7
Figure 7. Figure 7: Answer Extraction Prompt used with GPT4o [PITH_FULL_IMAGE:figures/full_fig_p031_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Answer Extraction Prompt used with GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p031_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Best & Worst Prompts within each category [PITH_FULL_IMAGE:figures/full_fig_p031_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Accuracy fluctuations across different textual [PITH_FULL_IMAGE:figures/full_fig_p032_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Model sensitivity across all prompts, show [PITH_FULL_IMAGE:figures/full_fig_p032_12.png] view at source ↗
Figure 15
Figure 15. Figure 15: Model Sensitivity (Negative Prompts). Shows mean accuracy and standard deviation averaged over the three benchmarks: MMStar, MMMU-Pro, MVBench considering only the negative prompts. Closed-source models, such as GPT-4o and Gem￾ini 1.5 Pro, exhibit higher sensitivity. This could be due to refined instruction tuning, structured op￾timization for user queries, and meta-prompting mechanisms. These models are … view at source ↗
Figure 17
Figure 17. Figure 17: Sensitivity of Question Types in MMMU￾Pro. The Figure represents the Average Accuracy and Standard Deviation averaged across the models. The MMMU-Pro benchmark introduces three distinct types of multiple-choice questions: (1) S4, the standard format with four answer choices; (2) S10, an extended version with ten answer choices; and (3) V, a vision-based setting where the question is embedded within an ima… view at source ↗
Figure 16
Figure 16. Figure 16: Benchmark Sensitivity The bar chart ( [PITH_FULL_IMAGE:figures/full_fig_p033_16.png] view at source ↗
Figure 20
Figure 20. Figure 20: Sensitivity of Subjects in MMMU-Pro. The [PITH_FULL_IMAGE:figures/full_fig_p034_20.png] view at source ↗
Figure 18
Figure 18. Figure 18: Sensitivity of Constituent Benchmarks of [PITH_FULL_IMAGE:figures/full_fig_p034_18.png] view at source ↗
Figure 21
Figure 21. Figure 21: Sensitivity of Temporal Tasks in MVBench. [PITH_FULL_IMAGE:figures/full_fig_p034_21.png] view at source ↗
Figure 19
Figure 19. Figure 19: Sensitivity of Core Capabilities of MMStar. [PITH_FULL_IMAGE:figures/full_fig_p034_19.png] view at source ↗
Figure 23
Figure 23. Figure 23: Distribution of standard deviation values [PITH_FULL_IMAGE:figures/full_fig_p035_23.png] view at source ↗
Figure 22
Figure 22. Figure 22: GPT-4o Performance (Absolute Accuracy) for Category-12 Prompts on MMMU-Pro. This shows how even the slightest difference in how the answer is expected can result in significant fluctuations in perfor￾mance. Dataset Type Model ∆ Accuracy Response MMStar Type6.2 MiniCPM-v2.6-8B -12.6 "$LETTER" Type4.5 Molmo-7B-d-0924 -20.5 "A/A/A/A" Type7.6 Molmo-7B-d-0924 -21.8 "E" or "F" Type9.3 InternVL2.5-1B -21.3 "$LET… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models

    cs.CL 2026-06 unverdicted novelty 7.0

    All 18 audited MLLMs exhibit order sensitivity with per-facet flip rates of 24-50%, exceeding same-order decoder noise.

  2. MIRROR: Learning from the Other View for Multi-Modal Reasoning

    cs.AI 2026-07 conditional novelty 6.0

    An RL method that selects the best-performing view of each geometry problem as an internal teacher and distills it into weaker views improves VLM reasoning accuracy and consistency.

  3. Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMs

    cs.CL 2026-05 unverdicted novelty 6.0

    Domain specialization does not consistently improve clinical LLM robustness to meaning-preserving prompt variations, as shown by new sensitivity metrics on DiagnosisQA and MedQA.

Reference graph

Works this paper leans on

10 extracted references · 3 canonical work pages · cited by 3 Pith papers

  1. [3]

    **Question:** What design element best describes the visuals? **Options:** A

    Choose the most appropriate answer from the options provided. **Question:** What design element best describes the visuals? **Options:** A. Composition B. Perspective C. Balance D. Shape **Answer:** Type 2.4: ## ##Instructions##:

    work page

  2. [4]

    Preprint, arXiv:2311.17005

    Mvbench: A comprehensive multi- modal video understanding benchmark. Preprint, arXiv:2311.17005. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024. Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281. Pan Lu, Hritik Bansal, ...

    Pith/arXiv arXiv 2024

  3. [5]

    Preprint, arXiv:2411.04923

    Videoglamm: A large multimodal model for pixel-level visual grounding in videos. Preprint, arXiv:2411.04923. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex...

    Pith/arXiv arXiv 2024

  4. [10]

    Analyze the given image(s)/video and understand its content

    work page

  5. [11]

    Read the question carefully

  6. [12]

    Persona":

    Choose the most appropriate answer from the options provided. ##Question##: What design element best describes the visuals? ##Options##: A. Composition B. Perspective C. Balance D. Shape ##Answer##: Type 2.5: Compact TASK: Analyze the image(s)/video and pick the best option. QUESTION: What design element best describes the visuals? OPTIONS: A. Composition...

    work page 2059

  7. [2016]

    Preprint, arXiv:1603.07396

    A diagram is worth a dozen images. Preprint, arXiv:1603.07396. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yan- wei Li, Ziwei Liu, and Chunyuan Li. 2025. LLaV A- onevision: Easy visual task transfer. Transactions on Machine Learning Research. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and...

    Pith/arXiv arXiv 2025

  8. [2022]

    a helpful assistant

    The carbon footprint of machine learn- ing training will plateau, then shrink. Preprint, arXiv:2204.05149. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural langu...

    Pith/arXiv arXiv 2021

  9. [2024]

    Preprint, arXiv:2409.17146

    Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. Preprint, arXiv:2409.17146. Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Sas- try, Evangelos Milios, Sageev Oore, and Hassan Sajjad. 2024. Sensitivity of generative vlms to se- mantically and lexically altered prompts. Preprint, arXiv:2410.13030. Federico Errica, Gi...

    Pith/arXiv arXiv 2024

  10. [2025]

    Preprint, arXiv:2306.09996

    Investigating prompting techniques for zero- and few-shot visual question answering. Preprint, arXiv:2306.09996. Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. 2024. Artifacts or abduction: How do LLMs answer multiple-choice questions without the question? In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Lingu...

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.