Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensembles

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper shows that a Bayesian prompt ensemble whose prompt weights are conditioned on image clusters improves both accuracy and calibration of MLLM judges on text-to-image preference tasks.

desk verdict A solid, carefully tested extension of BPE to multimodal judging, but the cluster mechanism is never isolated—a random-cluster ablation is needed to know if the visual grouping is actually doing the work. read the letter →

arxiv 2509.08777 v1 pith:BYMASWWZ submitted 2025-09-10 cs.CV cs.CL

Eric Slyman , Mehrab Tanjim , Kushal Kafle , Stefan Lee This is my paper · ORCID
classification cs.CVcs.CL
keywords multimodallargelanguagemodelsMLLM-as-a-judgemodelcalibrationBayesianpromptensemblestext-to-imageevaluationensemblingCLIPimageclusteringselectiveprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal large language models are increasingly used to judge text-to-image outputs, but they are overconfident and their accuracy shifts with how a task is prompted. The paper claims that standard prompt ensembling—treating all prompts as equally important, or learning one global set of prompt weights—does not transfer well to image-based judgment tasks. It proposes Multimodal Mixture-of-Bayesian Prompt Ensembles (MMB), which learns separate prompt weights for groups of visually similar images and mixes them according to each new image's soft cluster assignment. On the HPSv2 and MJBench benchmarks, MMB improves agreement with human preferences and lowers calibration error relative to single-prompt baselines and to the Bayesian Prompt Ensemble method it extends. The practical payoff is that a calibrated judge can be trusted on high-confidence judgments while low-confidence cases are deferred to human review.

What carries the argument

The load-bearing object is the soft image-conditional mixture: a set of prompts treated as latent variables, a variational distribution $q(a|z)$ with learnable weights $w_{za}$ per visual cluster $z$, and a soft assignment $p(z|x)$ derived from cosine similarities between the image embedding and cluster centroids. Spherical k-means on a pretrained image encoder partitions an unlabeled support set into $K$ groups; temperature $\tau$ in the soft assignment interpolates between independent per-cluster BPE as $\tau\to0$ and a single global BPE as $\tau\to\infty$. The training objective balances, over validation data, the sum of soft-assignment-weighted per-cluster log-likelihoods minus per-cluster entropy, which is an evidence lower bound under the conditional-independence assumptions $a\perp x\mid z$ and $y\perp z\mid a$. Inference mixes the MLLM's per-prompt probabilities according to the learned weights and the test image's soft group membership.

What would settle it

Run MMB on a preference benchmark whose images all come from one narrow visual style, so every image lands in essentially one cluster, and compare it with BPE on the same data; if MMB's calibration improvement persists, the advantage is not caused by visual conditioning, undercutting the paper's explanation.

Watch

Extended reading notes

Core claim

The central claim is that an MLLM judge's reliability on a text-to-image preference question is not a global property of the prompt alone, but depends on the visual content being judged, and that this dependence can be captured with a mixture-of-experts-style ensemble. MMB forms $K$ groups by spherical k-means on image embeddings from a pretrained encoder, assigns each image a soft membership $p(z|x)$ based on cosine similarity to cluster centroids, and learns cluster-specific prompt weights $w^*_{za}$ by maximizing an evidence lower bound on validation pairs that balances per-group log-likelihood against an entropy regularizer. At test time the final preference probability is a soft mixture, $p(y|x)\approx\sum_z p(z|x)\sum_a w^*_{za}\,p(y|x,a)$, so images in different visual styles effectively use different prompt experts. At 10 prompts and 20 validation samples on HPSv2, the paper reports expected calibration error of 0.091 for MMB versus 0.114 for BPE and 0.142 for a simple average.

Load-bearing premise

The method's gains rest on the assumption that images which are close in the embedding space used for clustering also tend to agree on which evaluation prompt is most reliable; if prompt reliability does not vary coherently across these visual clusters, the cluster-specific weights add no useful signal.

Editorial extensions

If this is right

  • Selective evaluation becomes practical: a developer can set a confidence threshold and defer low-confidence judgments to human reviewers, and the reported coverage-error curves show MMB keeps error low even as coverage rises.
  • The method works on closed-source judges because it only queries the model and never needs weight access or fine-tuning.
  • MMB is data-efficient: calibration and accuracy gains hold with as few as five validation samples and five prompts, and performance saturates at moderate cluster counts, so the added machinery is cheap to tune.
  • The cluster-level weights are interpretable, revealing which prompt personas are most reliable for which visual styles, a diagnostic that single-prompt selection hides.
  • The same scaffold transfers to other multimodal judgment tasks, including VQA, content moderation, and ordinal Likert-style ratings, without modifying the base model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if prompt reliability really does vary coherently along embedding clusters, the learned cluster-to-prompt mapping could be reused as a cheap prompt-selection rule, letting a human read the highest-weighted prompt per cluster and build a specialized rubric without running the ensemble at test time.
  • Beyond the paper: the temperature $\tau$ and cluster count $K$ could themselves be learned from validation data; the paper grids over $K$ and treats $\tau$ as a fixed hyperparameter, so an adaptive version would remove the main tuning burden and sharpen comparisons with BPE.
  • Beyond the paper: on bias-focused benchmarks like MJBench, visual clustering may not align with the demographic axes along which judge bias varies, so the equal-preference confidence improvement may not transfer to other bias dimensions; a direct test would compare MMB's confidence on images that are visually similar but differ demographically.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Multimodal Mixture-of-Bayesian Prompt Ensembles (MMB), an extension of Bayesian Prompt Ensembles (BPE) for calibrating multimodal LLM judges on text-to-image preference tasks. MMB clusters CLIP image embeddings with spherical k-means, learns prompt weights per cluster by maximizing an ELBO on a validation set, and at inference time mixes prompt predictions using soft cluster assignments p(z|x). The authors evaluate MMB against single-prompt and ensemble baselines on HPSv2 pairwise human preferences and on a synthetically constructed MJBench no-preference task, reporting consistent improvements in ECE, MCE, F1, and related metrics on HPSv2, with extensive factorial experiments (multiple prompt counts, validation sizes, cluster counts, and random seeds) and permutation tests with FDR control.

Significance. If the reported results hold, MMB is a practically useful, black-box-compatible method for improving calibration and accuracy of closed-source MLLM judges, with a lightweight inference procedure. The HPSv2 experiments are unusually thorough (52.2k configurations, permutation tests with Benjamini-Yekutieli correction, multiple seeds), and the gains over BPE are consistent across prompt and validation-size settings. The ELBO derivation is self-contained and the method does not appear circular: weights are fit on a held-out split and evaluated on test splits. However, the central novelty, namely that image-cluster structure is what drives the improvement, is not directly tested; the paper lacks a random-cluster ablation, and the group-assignment procedure for pairwise inputs is underspecified. The MJBench evidence is synthetic and does not by itself demonstrate calibration to human preferences.

major comments (4)
  1. [Sec. 4, Eq. (7)] Eq. (7) states that the factorization applies conditional independencies a⊥x|z and y⊥z|a. The second independence is not the one actually used: the derivation retains p(y|x,a), so the required condition is y⊥z|x,a (along with a⊥x|z). As printed, y⊥z|a would not justify the factor p(y|x,a) with x conditioning retained, and it is not implied by the model described in the text. Please correct the stated independence and align the derivation in Sec. C with the correct condition.
  2. [Sec. 4.2, Eq. (10); Sec. 5] The end-to-end task is pairwise preference judgment with input x=(I1,I2), and Eq. (10) requires p(z|x), but the soft group assignment in Sec. 4 is defined only for a single image embedding ϕ_I(x). The paper never specifies how pairwise inputs are assigned to clusters: whether p(z|x) is an average of the two per-image soft assignments, a product, an assignment based on one image, or something else. This is a reproducibility gap in the central inference formula and must be stated explicitly.
  3. [Sec. 6, Table 3] The claim that visual clustering is the source of MMB's gains is not isolated. Table 3 varies K while keeping true k-means clusters, and the comparisons in Table 1 are against a global BPE that has N prompt weights rather than K×N. Because MMB has K times as many weights, the improvement could in principle come from the higher-capacity parameterization regularized by the entropy term rather than from meaningful image groups. Please add a random-cluster ablation (e.g., permuted cluster assignments with the same K and the same number of parameters) and, ideally, a test with cluster assignments based on non-visual nuisance features, to show that the visual structure of the clusters is load-bearing.
  4. [Sec. 5, MJBench; Table 4] The MJBench calibration evidence is synthetic: lower-preference variants are produced by deterministic aesthetic degradations, and the test pairs are constructed so that the unbiased prediction is 50% confidence. Reporting lower average confidence on this artificial no-preference task shows only that MMB is less overconfident on these synthetic pairs; it does not demonstrate calibration to human preferences, since no human labels are involved. Please either evaluate on real human preference data for MJBench or explicitly reframe Table 4 as an overconfidence/ambiguity analysis rather than as calibration evidence.
minor comments (4)
  1. [Sec. 4.2, Eq. (10)] In Eq. (10), the index i in p(y|x,a_i) is undefined; the sum is over a, so the term should be p(y|x,a).
  2. [Sec. 3.1, Eq. (5)] The relationship w_a = q(a)/N with the uniform prior p(a)≈C is not fully specified; please clarify the normalization and the role of N in the discrete reparameterization.
  3. [Sec. 5, experimental setup] The stated total of 52.2k unique experimental configurations does not match the factorial product of the listed factors (3 prompt levels × 4 sample levels × 5 cluster counts × 3 train seeds × 50 data seeds × 5 clustering seeds = 45,000, before accounting for baselines without clusters). Please clarify how the 52.2k figure is obtained.
  4. [Sec. 5, Table 4] Table 4 marks best and second-best methods but does not report significance tests for the MJBench confidence values; please state whether the differences are statistically significant, or mark the table accordingly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MMB's weights are fit on a held-out validation split and evaluated on held-out test splits; the ELBO derivation is self-contained.

full rationale

MMB's central mechanism is an empirical extension of external BPE: cluster-specific prompt weights w_za are optimized on D_val via the ELBO in Eq. (9) and then applied to test inputs through Eq. (10). The target quantities (test accuracy, ECE, MCE, AUC-PR, NLL, Brier) are measured on held-out test splits (HPSv2 14.4k pairwise comparisons; MJBench held-out folds) that were not used to select w_za. The variational derivation in Sec. C is standard and does not define p(y|x) in terms of fitted weights; p(z|x) uses only unlabeled support-set embeddings and CLIP similarities. The paper's limiting cases (K=1 collapses to BPE; |D_val|=0 reduces to AVG) are explicit boundaries, not hidden reuse of the target result. The absence of a random-cluster ablation is an experimental-control question, not circularity, because the reported gains are not forced by construction or by a self-citation chain. No load-bearing self-citations are present; reference [59] is external prior work.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The method's added machinery is mostly statistical (latent group variable z and variational weights w_za), not a new physical entity. The key postulates are the cluster-conditioned factorization and the choice of CLIP embeddings as the relevance space.

free parameters (2)
  • K (number of image clusters) = 4 to 64 swept; around 16 recommended
    Chosen by hand; experiments show performance saturates around K=32, but the method requires this hyperparameter (Sec 5, Table 3).
  • tau (temperature in soft group assignment) = not reported
    Controls sharpness of p(z|x) in Eq 9; authors discuss extremes but do not report the value used in main experiments (Sec 4).
assumptions (5)
  • domain assumption Conditional independence a⊥x|z and y⊥z|a used to factor p(y,a,z|x) in Eq 7.
    Sec 4.1 and Sec C. The factorization in Eq 7 requires p(y|x,z,a)=p(y|x,a) and p(a|x,z)=p(a|z); the stated y⊥z|a is stronger than needed and not generally true.
  • domain assumption Prompt reliability varies by image cluster.
    Core modeling premise that motivates cluster-conditioned weights (Sec 4).
  • domain assumption CLIP-ViT-B16 cosine similarity to k-means centroids is a valid relevance function for prompt weighting.
    Used in Sec 5 to define sim(.,.) and p(z|x).
  • standard math Uniform prior p(a|z) over prompts within each group.
    Assumed in Eq 9 and Sec C to derive the entropy regularizer.
  • domain assumption Validation annotations D_val are representative ground truth for prompt weighting.
    All calibration weights are inferred from D_val (Sec 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensembles." pith.science (2026). https://pith.science/paper/BYMASWWZ

@misc{pith2026250908777,
  author       = {Pith},
  title        = {Pith review of: Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensembles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BYMASWWZ}},
  note         = {Machine review of arXiv:2509.08777}
}
read the original abstract

Multimodal large language models (MLLMs) are increasingly used to evaluate text-to-image (TTI) generation systems, providing automated judgments based on visual and textual context. However, these "judge" models often suffer from biases, overconfidence, and inconsistent performance across diverse image domains. While prompt ensembling has shown promise for mitigating these issues in unimodal, text-only settings, our experiments reveal that standard ensembling methods fail to generalize effectively for TTI tasks. To address these limitations, we propose a new multimodal-aware method called Multimodal Mixture-of-Bayesian Prompt Ensembles (MMB). Our method uses a Bayesian prompt ensemble approach augmented by image clustering, allowing the judge to dynamically assign prompt weights based on the visual characteristics of each sample. We show that MMB improves accuracy in pairwise preference judgments and greatly enhances calibration, making it easier to gauge the judge's true uncertainty. In evaluations on two TTI benchmarks, HPSv2 and MJBench, MMB outperforms existing baselines in alignment with human annotations and calibration across varied image content. Our findings highlight the importance of multimodal-specific strategies for judge calibration and suggest a promising path forward for reliable large-scale TTI evaluation.

Figures

Figures reproduced from arXiv: 2509.08777 by the authors.

Figure 1
Figure 1. A high-level comparison on the HPSv2 [65] dataset between an average ensemble baseline (AVG.), current SOTA BPE [59], and our MMB method. We show F1 Score (higher is better ↑) and ECE (lower is better ↓). While BPE somewhat low￾ers the model’s discriminative ability (F1) relative to the baseline, MMB both recovers that loss (+1.8% F1 vs. AVG.) and achieves stronger calibration (-36% ECE vs. AVG.). All differences ar… view at source ↗
Figure 2
Figure 2. Overview of our MMB for multimodal prompt ensembles. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Error–coverage curves on HPSv2 [65] with 10 prompts across varying numbers of validation samples (5, 10, 20, 50). Each curve represents an average over multiple runs, with 95% confidence intervals (shaded regions) from bootstrapped sampling of the mean. Our MMB approach consistently achieves the lowest error across coverage levels and displays the narrowest intervals, indicating more stable and reliable performance.… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Clusters from MMB’s grouping, each matched to its best-performing prompt by largest weight in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Error-coverage curves for different models. HPSv2. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Image clusters and their corresponding highest weighted prompts (or lack thereof) when using [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Image clusters and their corresponding highest weighted prompts (or lack thereof) when using [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Synthetic images preference pairs generated from MJBench-Bias. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics

    cs.CV 2026-01 conditional novelty 6.0 of 10

    Prototypicality bias: common text-to-image metrics systematically prefer plausible-but-wrong images over correct non-prototypical ones; PROTOSCORE mitigates but does not eliminate the failure.

Reference graph

Works this paper leans on

75 extracted references · 57 canonical work pages · cited by 1 Pith paper

  1. [1]

    Spice: Semantic propositional image cap- tion evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image cap- tion evaluation. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, Octo- ber 11-14, 2016, Proceedings, Part V 14, pages 382–398. Springer, 2016. 1

    work page 2016

  2. [2]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering.ICCV, 2015. 1

    work page 2015

  3. [3]

    Openflamingo: An open-source frame- work for training large autoregressive vision-language mod- els.arXiv, 2023

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source frame- work for training large autoregressive vision-language mod- els.arXiv, 2023. 1

    work page 2023

  4. [4]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond.arXiv,

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond.arXiv,

    work page

  5. [5]

    The control of the false discovery rate in multiple testing under dependency

    Yoav Benjamini and Daniel Yekutieli. The control of the false discovery rate in multiple testing under dependency. Annals of statistics, pages 1165–1188, 2001. 6, 7, 3

    work page 2001

  6. [6]

    Improving image genera- tion with better captions.arXiv, 2023

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving image genera- tion with better captions.arXiv, 2023. 1

    work page 2023

  7. [7]

    Visit-bench: A benchmark for vision- language instruction following inspired by real-world use

    Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. Visit-bench: A benchmark for vision- language instruction following inspired by real-world use. NeurIPS, Datasets and Benchmarks, 2024. 2

    work page 2024

  8. [8]

    Andrew P. Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms.Pattern Recognition, 30(7):1145–1159, 1997. 6

    work page 1997

Show all 75 references
  1. [9]

    Glenn W. Brier. Verification of forecasts expressed in terms of probability.Monthly Weather Review, 78(1):1–3, 1950. 6

    1950

  2. [10]

    X-iqe: explainable image quality evaluation for text-to-image generation with visual large language models.arXiv, 2023

    Yixiong Chen, Li Liu, and Chris Ding. X-iqe: explainable image quality evaluation for text-to-image generation with visual large language models.arXiv, 2023. 2

    2023

  3. [11]

    Mj-bench: Is your multimodal reward model really a good judge for text-to- image generation?arXiv preprint arXiv:2407.04842, 2024

    Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, et al. Mj-bench: Is your multimodal reward model really a good judge for text-to- image generation?arXiv preprint arXiv:2407.04842, 2024. 5

    2024 arXiv

  4. [12]

    Dall-eval: Probing the reasoning skills and social biases of text-to- image generation models.ICCV, 2023

    Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to- image generation models.ICCV, 2023. 1

    2023

  5. [13]

    A coefficient of agreement for nominal scales

    Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46,

  6. [14]

    Ex- ploring GPT-4 vision for text-to-image synthesis evaluation

    Xiao Cui, Qi Sun, Wengang Zhou, and Houqiang Li. Ex- ploring GPT-4 vision for text-to-image synthesis evaluation. ICLR, Tiny Papers, 2024. 2

    2024

  7. [15]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. NeurIPS, 2024. 1

    2024

  8. [16]

    Improving selective visual question answering by learning from your peers

    Corentin Dancette, Spencer Whitehead, Rishabh Mahesh- wary, Ramakrishna Vedantam, Stefan Scherer, Xinlei Chen, Matthieu Cord, and Marcus Rohrbach. Improving selective visual question answering by learning from your peers. In Proceedings of the IEEE/CVF Conference on Computer V...

    2023

  9. [17]

    Moura, Devi Parikh, and Dhruv Batra

    Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jos ´e M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog.CVPR, 2017. 1

    2017

  10. [18]

    The relationship be- tween precision-recall and ROC curves

    Jesse Davis and Mark Goadrich. The relationship be- tween precision-recall and ROC curves. InProceedings of the 23rd International Conference on Machine Learning (ICML), pages 233–240, 2006. 6

    2006

  11. [19]

    Mllm-bench, evaluating multi- modal llms using gpt-4v.arXiv, 2023

    Wentao Ge, Shunian Chen, Guiming Chen, Junying Chen, Zhihong Chen, Shuo Yan, Chenghao Zhu, Ziyue Lin, Wenya Xie, Xidong Wang, et al. Mllm-bench, evaluating multi- modal llms using gpt-4v.arXiv, 2023. 2

    2023

  12. [20]

    Selective classification for deep neural networks.NeurIPS, 30, 2017

    Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks.NeurIPS, 30, 2017. 2

    2017

  13. [21]

    Practical variational inference for neural net- works.Advances in neural information processing systems, 24, 2011

    Alex Graves. Practical variational inference for neural net- works.Advances in neural information processing systems, 24, 2011. 3

    2011

  14. [22]

    A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xue- hao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024. 2

    2024 arXiv

  15. [23]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. InProceedings of the 34th International Conference on Machine Learning (ICML), pages 1321–1330, 2017. 2, 6

    2017

  16. [24]

    Springer, New York, NY , USA, 2001

    Trevor Hastie, Robert Tibshirani, and Jerome Friedman.The Elements of Statistical Learning. Springer, New York, NY , USA, 2001. 6

    2001

  17. [25]

    Glass, and Yulia Tsvetkov

    Tianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James R. Glass, and Yulia Tsvetkov. On the blind spots of model-based evaluation metrics for text generation.ACL, 2023. 2

    2023

  18. [26]

    Machine learning with a re- ject option: A survey.Machine Learning, pages 1–38, 2024

    Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. Machine learning with a re- ject option: A survey.Machine Learning, pages 1–38, 2024. 2

    2024

  19. [27]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in neural information processing systems, 30, 2017. 1

    2017

  20. [28]

    Promptboosting: Black-box text classification with ten forward passes

    Bairu Hou, Joe O’connor, Jacob Andreas, Shiyu Chang, and Yang Zhang. Promptboosting: Black-box text classification with ten forward passes. InInternational Conference on Ma- chine Learning, pages 13309–13324. PMLR, 2023. 2, 3 9

    2023

  21. [29]

    Are llm-based evaluators con- fusing nlg quality criteria?arXiv, 2024

    Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, and Xiaojun Wan. Are llm-based evaluators con- fusing nlg quality criteria?arXiv, 2024. 2

    2024

  22. [30]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 5

    2024 arXiv

  23. [31]

    Calibrating language models via augmented prompt ensembles.ICML Workshop on Deployable Generative AI, 2023

    Mingjian Jiang, Yangjun Ruan, Sicong Huang, Saifei Liao, Silviu Pitis, Roger Baker Grosse, and Jimmy Ba. Calibrating language models via augmented prompt ensembles.ICML Workshop on Deployable Generative AI, 2023. 2, 3

    2023

  24. [32]

    How can we know what language models know?Trans- actions of the Association for Computational Linguistics, 8: 423–438, 2020

    Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neu- big. How can we know what language models know?Trans- actions of the Association for Computational Linguistics, 8: 423–438, 2020. 3

    2020

  25. [33]

    Billion- scale similarity search with GPUs.IEEE Transactions on Big Data, 2019

    Jeff Johnson, Matthijs Douze, and Herv ´e J ´egou. Billion- scale similarity search with GPUs.IEEE Transactions on Big Data, 2019. 5

    2019

  26. [34]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Ma- tiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01569, 2023. 1

    2023 arXiv

  27. [35]

    Benchmarking cog- nitive biases in large language models as evaluators.arXiv,

    Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cog- nitive biases in large language models as evaluators.arXiv,

  28. [36]

    Improved precision and recall met- ric for assessing generative models.Advances in neural in- formation processing systems, 32, 2019

    Tuomas Kynk ¨a¨anniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall met- ric for assessing generative models.Advances in neural in- formation processing systems, 32, 2019. 1

    2019

  29. [37]

    Seed-bench: Bench- marking multimodal large language models.CVPR, 2024

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Bench- marking multimodal large language models.CVPR, 2024. 1

    2024

  30. [38]

    From generation to judgment: Op- portunities and challenges of llm-as-a-judge.arXiv preprint arXiv: 2411.16594, 2024

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. From generation to judgment: Op- portunities and challenges of llm-as-a-judge.arXiv preprint arXiv:...

    2024

  31. [39]

    Llms-as-judges: a com- prehensive survey on llm-based evaluation methods.arXiv preprint arXiv:2412.05579, 2024

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Llms-as-judges: a com- prehensive survey on llm-based evaluation methods.arXiv preprint arXiv:2412.05579, 2024. 2

    2024 arXiv

  32. [40]

    ROUGE: A package for automatic evaluation of summaries.Text Summarization Branches Out, 2004

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries.Text Summarization Branches Out, 2004. 1

    2004

  33. [41]

    Microsoft coco: Common objects in context.ECCV,

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context.ECCV,

  34. [42]

    Mitigating hallucination in large multi-modal models via robust instruction tuning.ICLR,

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning.ICLR,

  35. [43]

    Visual instruction tuning.NeurIPS, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.NeurIPS, 2023. 1, 2

    2023

  36. [44]

    Llms as narcissistic evaluators: When ego inflates evaluation scores

    Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. Llms as narcissistic evaluators: When ego inflates evaluation scores. arXiv, 2023. 2

    2023

  37. [45]

    Predict re- sponsibly: improving fairness and accuracy by learning to defer.NeurIPS, 31, 2018

    David Madras, Toni Pitassi, and Richard Zemel. Predict re- sponsibly: improving fairness and accuracy by learning to defer.NeurIPS, 31, 2018. 2

    2018

  38. [46]

    Gpt-4v(ision) technical work and authors.https: //openai.com/contributions/gpt-4v/, 2023

    OpenAI. Gpt-4v(ision) technical work and authors.https: //openai.com/contributions/gpt-4v/, 2023. 1

    2023

  39. [47]

    Openai o1 system card.https://openai

    OpenAI. Openai o1 system card.https://openai. com/index/openai-o1-system-card/, 2024. 1

    2024

  40. [48]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations.arXiv,

  41. [49]

    Bleu: a method for automatic evaluation of machine translation.ACL, 2002

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation.ACL, 2002. 1

    2002

  42. [50]

    Kosmos-2: Grounding multimodal large language models to the world.arXiv, 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world.arXiv, 2023. 1

    2023

  43. [51]

    Sdxl: improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. InInternational Con- ference on Learning Representations, pages 1–13, 2024. 1

    2024

  44. [52]

    Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever

    Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from nat- ural language supervision.ICML, 2021. 5, 1

    2021

  45. [53]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  46. [54]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  47. [55]

    Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information processing systems, 35:36479–36494, 2022

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information ...

    2022

  48. [56]

    Verbosity bias in preference labeling by large language models.arXiv preprint arXiv:2310.10076, 2023

    Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Aki- moto. Verbosity bias in preference labeling by large language models.arXiv preprint arXiv:2310.10076, 2023. 2

    2023 arXiv

  49. [57]

    Improved techniques for training gans.Advances in neural information processing systems, 29, 2016

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans.Advances in neural information processing systems, 29, 2016. 1

    2016

  50. [58]

    Gemini: a 10 family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a 10 family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 1

    2023 arXiv

  51. [59]

    Bayesian prompt ensembles: Model uncer- tainty estimation for black-box large language models

    Francesco Tonolini, Nikolaos Aletras, Jordan Massiah, and Gabriella Kazai. Bayesian prompt ensembles: Model uncer- tainty estimation for black-box large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 12229–12272, 2024. 1, 2, 3, 6

    2024

  52. [60]

    Large language models are not fair evaluators.arXiv, 2023

    Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators.arXiv, 2023. 2

    2023

  53. [61]

    A prompt pattern catalog to enhance prompt engineering with chatgpt.arXiv preprint arXiv:2302.11382, 2023

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Car- los Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer- Smith, and Douglas C Schmidt. A prompt pattern catalog to enhance prompt engineering with chatgpt.arXiv preprint arXiv:2302.11382, 2023. 3

    2023 arXiv

  54. [62]

    Reliable visual question answering: Abstain rather than answer incorrectly

    Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than answer incorrectly. InEuropean Conference on Computer Vision, pages 148–166. Springer, 2022. 2

    2022

  55. [63]

    Strength in numbers: Estimating confidence of large language models by prompt agreement

    Gwenyth Portillo Wightman, Alexandra Delucia, and Mark Dredze. Strength in numbers: Estimating confidence of large language models by prompt agreement. InProceedings of the 3rd Workshop on Trustworthy Natural Language Pro- cessing (TrustNLP 2023), pages 326–362, 2023. 2, 3

    2023

  56. [64]

    Gpt- 4v(ision) is a human-aligned evaluator for text-to-3d genera- tion.CVPR, 2024

    Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt- 4v(ision) is a human-aligned evaluator for text-to-3d genera- tion.CVPR, 2024. 2

    2024

  57. [65]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341,

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341,

    arXiv

  58. [66]

    Llava- critic: Learning to evaluate multimodal models.arXiv preprint arXiv:2410.02712, 2024

    Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava- critic: Learning to evaluate multimodal models.arXiv preprint arXiv:2410.02712, 2024. 2

    2024 arXiv

  59. [67]

    Vigor: Improving visual ground- ing of large vision language models with fine-grained reward modeling.arXiv, 2024

    Siming Yan, Min Bai, Weifeng Chen, Xiong Zhou, Qixing Huang, and Li Erran Li. Vigor: Improving visual ground- ing of large vision language models with fine-grained reward modeling.arXiv, 2024. 2

    2024

  60. [68]

    Jus- tice or prejudice? quantifying biases in llm-as-a-judge.arXiv preprint arXiv:2410.02736, 2024

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. Jus- tice or prejudice? quantifying biases in llm-as-a-judge.arXiv preprint arXiv:2410.02736, 2024. 2

    arXiv

  61. [69]

    Lamm: Language-assisted multi- modal instruction-tuning dataset, framework, and bench- mark.NeurIPS, Datasets and Benchmarks, 2023

    Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingn- ing Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al. Lamm: Language-assisted multi- modal instruction-tuning dataset, framework, and bench- mark.NeurIPS, Datasets and Benchmarks, 2023. 2

    2023

  62. [70]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions.TACL, 2014

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions.TACL, 2014. 1

    2014

  63. [71]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention.ICLR, 2024

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention.ICLR, 2024. 1

    2024

  64. [72]

    Gpt-4v (ision) as a generalist eval- uator for vision-language tasks.arXiv, 2023

    Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-4v (ision) as a generalist eval- uator for vision-language tasks.arXiv, 2023. 2

    2023

  65. [73]

    Xing, Haotong Zhang, Joseph Gon- zalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gon- zalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena.NeurIPS, 2023. 1

    2023

  66. [74]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.ICLR,

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.ICLR,

  67. [2024]

    aug- mented

    1 11 Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensembles Supplementary Material A. Additional Experimental Details Factor Levels/Value Notes Exps. prompts 5, 10, 20 Num. prompts ina samples 5, 10, 20, 50 Num. samples inD val clusters 4, 8, 16, 32, 64 Num. clus...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.