{"id":"2f84a253-de28-4be2-b625-bea2395b508b","arxiv_id":"2509.08777","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A cluster-conditioned Bayesian prompt ensemble improves the calibration and accuracy of multimodal LLM judges for text-to-image preference evaluation.","lead":"Multimodal AI models that judge generated images are often overconfident and biased. This paper introduces MMB, which blends multiple evaluation prompts using visual clusters of the images being judged, and reports better calibration and human agreement on two benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MMB's central mechanism is not isolated: without a random-cluster ablation, the reported gains could come from added parameterization rather than from meaningful image clusters.","rationale":"The reader's weakest assumption is that spherical k-means clusters of CLIP embeddings define groups within which prompt reliability is consistent, and that cluster-specific weights learned on Dval generalize through soft assignments. I agree: this is the load-bearing condition for the central claim. The empirical core on HPSv2 is otherwise strong: the factorial design, multiple seeds, permutation tests with FDR control, and consistent improvement across metrics are credible evidence that MMB performs well on this benchmark. The concern is not that the numbers are wrong, but that the numbers do not yet identify why they are right. Because MMB differs from BPE in two ways simultaneously, by conditioning on clusters and by increasing the number of learned weights, the reported improvement could in principle be explained by the latter alone. Table 3 shows MMB with K=4 already reaches ECE .090 versus BPE's .114, which makes the capacity-explanation plausible: even four coarse clusters add many weights without necessarily capturing prompt-reliability structure. A random-cluster ablation is the minimal experiment that would separate these explanations. I would not reject or accept without it; the current CONDITIONAL verdict remains appropriate. Secondary issues, including the Eq. 7 conditional-independence typo, the unreleased prompt pool and code, and the synthetic MJBench confidence results, are real but addressable and do not change the verdict.","tokens_in":17292,"tokens_out":7215,"duration_ms":66026,"concrete_test":"On HPSv2 with the paper's 10-prompt/20-sample configuration, rerun MMB for K=16 and K=32 under three variants: (a) original spherical k-means on CLIP embeddings; (b) random centroids or randomly permuted cluster labels over the same Dsup, with the same soft-assignment temperature and entropy regularization; (c) a single global BPE using the same objective but without cluster conditioning. If variant (b) matches variant (a) in ECE and F1 within the permutation-test significance used in Table 1, then the visual clusters are not the source of MMB's gains and the central claim is unsupported. If variant (b) degrades to BPE-level performance, the cluster-reliability assumption is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is Eq. 9-10: prompt weights w_za are learned per CLIP-embedding cluster, and p(z|x) soft-assigns each test sample. For the main claim to hold, prompt reliability must vary coherently along the k-means structure of Dsup, and the learned weights must transfer to test images through p(z|x). The paper never directly tests this. Table 3 varies K while keeping the true k-means clusters, and Table 1 only compares against global BPE, so the cluster mechanism is never isolated. A plausible alternative is that MMB's advantage comes from having K times more weights, regularized by entropy, rather than from cluster semantics: with K=4 MMB already reaches ECE .090 versus BPE's .114, which is surprisingly large for only four broad visual groups and suggests additional degrees of freedom may drive the gain. Two related under-specifications reinforce this concern: Eq. 7's decomposition requires the conditional independence y⊥z | a,x (not y⊥z|a as printed), and for pairwise inputs x=(I1,I2) the paper never states how p(z|x) is computed from two image embeddings. A random-cluster ablation would settle whether the visual grouping is actually load-bearing or whether MMB is essentially a higher-capacity reparameterization of BPE.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Multimodal Mixture-of-Bayesian Prompt Ensembles (MMB), an extension of Bayesian Prompt Ensembles (BPE) for calibrating multimodal LLM judges on text-to-image preference tasks. MMB clusters CLIP image embeddings with spherical k-means, learns prompt weights per cluster by maximizing an ELBO on a validation set, and at inference time mixes prompt predictions using soft cluster assignments p(z|x). The authors evaluate MMB against single-prompt and ensemble baselines on HPSv2 pairwise human preferences and on a synthetically constructed MJBench no-preference task, reporting consistent improvements in ECE, MCE, F1, and related metrics on HPSv2, with extensive factorial experiments (multiple prompt counts, validation sizes, cluster counts, and random seeds) and permutation tests with FDR control.","tokens_in":17557,"tokens_out":4514,"duration_ms":43583,"significance":"If the reported results hold, MMB is a practically useful, black-box-compatible method for improving calibration and accuracy of closed-source MLLM judges, with a lightweight inference procedure. The HPSv2 experiments are unusually thorough (52.2k configurations, permutation tests with Benjamini-Yekutieli correction, multiple seeds), and the gains over BPE are consistent across prompt and validation-size settings. The ELBO derivation is self-contained and the method does not appear circular: weights are fit on a held-out split and evaluated on test splits. However, the central novelty, namely that image-cluster structure is what drives the improvement, is not directly tested; the paper lacks a random-cluster ablation, and the group-assignment procedure for pairwise inputs is underspecified. The MJBench evidence is synthetic and does not by itself demonstrate calibration to human preferences.","major_comments":[{"comment":"Eq. (7) states that the factorization applies conditional independencies a⊥x|z and y⊥z|a. The second independence is not the one actually used: the derivation retains p(y|x,a), so the required condition is y⊥z|x,a (along with a⊥x|z). As printed, y⊥z|a would not justify the factor p(y|x,a) with x conditioning retained, and it is not implied by the model described in the text. Please correct the stated independence and align the derivation in Sec. C with the correct condition.","section":"Sec. 4, Eq. (7)"},{"comment":"The end-to-end task is pairwise preference judgment with input x=(I1,I2), and Eq. (10) requires p(z|x), but the soft group assignment in Sec. 4 is defined only for a single image embedding ϕ_I(x). The paper never specifies how pairwise inputs are assigned to clusters: whether p(z|x) is an average of the two per-image soft assignments, a product, an assignment based on one image, or something else. This is a reproducibility gap in the central inference formula and must be stated explicitly.","section":"Sec. 4.2, Eq. (10); Sec. 5"},{"comment":"The claim that visual clustering is the source of MMB's gains is not isolated. Table 3 varies K while keeping true k-means clusters, and the comparisons in Table 1 are against a global BPE that has N prompt weights rather than K×N. Because MMB has K times as many weights, the improvement could in principle come from the higher-capacity parameterization regularized by the entropy term rather than from meaningful image groups. Please add a random-cluster ablation (e.g., permuted cluster assignments with the same K and the same number of parameters) and, ideally, a test with cluster assignments based on non-visual nuisance features, to show that the visual structure of the clusters is load-bearing.","section":"Sec. 6, Table 3"},{"comment":"The MJBench calibration evidence is synthetic: lower-preference variants are produced by deterministic aesthetic degradations, and the test pairs are constructed so that the unbiased prediction is 50% confidence. Reporting lower average confidence on this artificial no-preference task shows only that MMB is less overconfident on these synthetic pairs; it does not demonstrate calibration to human preferences, since no human labels are involved. Please either evaluate on real human preference data for MJBench or explicitly reframe Table 4 as an overconfidence/ambiguity analysis rather than as calibration evidence.","section":"Sec. 5, MJBench; Table 4"}],"minor_comments":[{"comment":"In Eq. (10), the index i in p(y|x,a_i) is undefined; the sum is over a, so the term should be p(y|x,a).","section":"Sec. 4.2, Eq. (10)"},{"comment":"The relationship w_a = q(a)/N with the uniform prior p(a)≈C is not fully specified; please clarify the normalization and the role of N in the discrete reparameterization.","section":"Sec. 3.1, Eq. (5)"},{"comment":"The stated total of 52.2k unique experimental configurations does not match the factorial product of the listed factors (3 prompt levels × 4 sample levels × 5 cluster counts × 3 train seeds × 50 data seeds × 5 clustering seeds = 45,000, before accounting for baselines without clusters). Please clarify how the 52.2k figure is obtained.","section":"Sec. 5, experimental setup"},{"comment":"Table 4 marks best and second-best methods but does not report significance tests for the MJBench confidence values; please state whether the differences are statistically significant, or mark the table accordingly.","section":"Sec. 5, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The HPSv2 contribution is solid and the experiments are unusually thorough, but the paper's central mechanism, image-cluster-conditioned prompt weighting, is not isolated from the increase in parameter count. The pairwise p(z|x) specification and the misstated independence in Eq. (7) are fixable, and the MJBench section should be reframed. I would not reject on the current evidence, but these points need to be addressed before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take: this is a genuine extension of Bayesian Prompt Ensembles to multimodal judging, and the HPSv2 results are real and carefully tested. The main soft spot is that the paper never isolates the mechanism behind the gains: image clusters might be load-bearing, or the improvement might just come from having K times more prompt weights. A random-cluster ablation would settle it, and the paper lacks one.\n\nWhat's new: BPE from Tonolini et al. is text-only. MMB conditions prompt weights on CLIP-embedding clusters, with soft assignments for each test sample. That is a sensible black-box-compatible idea, and the application to TTI preference judging is new. The empirical core on HPSv2 is extensive: 52.2k configurations, permutation tests with FDR control, and consistent ECE/F1 gains across many settings. The error-coverage curves and the qualitative cluster-to-persona correspondence are nice supporting evidence.\n\nNow the soft spots, in proportion. The biggest one is the missing ablation: Table 3 varies K but keeps true k-means clusters, and Table 1 only compares against global BPE. With K=4 MMB already drops ECE from .114 to .090 at 10 prompts / 20 samples, which is a large jump for only four broad visual groups—extra degrees of freedom could easily explain that. A random-cluster control would tell us whether the visual grouping is actually load-bearing. That is a fixable gap, not a fatal one. There are also two under-specified details: Eq. 7 states the conditional independence as y⊥z|a, but the derivation needs y⊥z|a,x; and for pairwise inputs the paper never states how p(z|x) is computed from the two image embeddings. Both are easy to correct. The MJBench evidence is synthetic—degraded pairs and equal-preference confidence—so it is a weak secondary demonstration, not a real calibration benchmark. Code and the exact prompt pool are not released, which hurts reproducibility but is common.\n\nWho this is for: people working on MLLM-as-judge calibration or TTI evaluation. It deserves a serious referee; I would send it to review. The main request should be a random-cluster ablation and a clear statement of p(z|x) for image pairs. With those, the method's contribution would be solid.","headline":"A solid, carefully tested extension of BPE to multimodal judging, but the cluster mechanism is never isolated—a random-cluster ablation is needed to know if the visual grouping is actually doing the work.","tokens_in":18047,"tokens_out":2554,"would_cite":true,"duration_ms":390820,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that a Bayesian prompt ensemble whose prompt weights are conditioned on image clusters improves both accuracy and calibration of MLLM judges on text-to-image preference tasks.","keywords":["multimodal large language models","MLLM-as-a-judge","model calibration","Bayesian prompt ensembles","text-to-image evaluation","prompt ensembling","CLIP image clustering","selective prediction"],"falsifier":"Run MMB on a preference benchmark whose images all come from one narrow visual style, so every image lands in essentially one cluster, and compare it with BPE on the same data; if MMB's calibration improvement persists, the advantage is not caused by visual conditioning, undercutting the paper's explanation.","tokens_in":17121,"feed_emoji":"⚖️","tokens_out":14379,"duration_ms":115095,"temperature":0.7,"pith_summary":"Multimodal large language models are increasingly used to judge text-to-image outputs, but they are overconfident and their accuracy shifts with how a task is prompted. The paper claims that standard prompt ensembling—treating all prompts as equally important, or learning one global set of prompt weights—does not transfer well to image-based judgment tasks. It proposes Multimodal Mixture-of-Bayesian Prompt Ensembles (MMB), which learns separate prompt weights for groups of visually similar images and mixes them according to each new image's soft cluster assignment. On the HPSv2 and MJBench benchmarks, MMB improves agreement with human preferences and lowers calibration error relative to single-prompt baselines and to the Bayesian Prompt Ensemble method it extends. The practical payoff is that a calibrated judge can be trusted on high-confidence judgments while low-confidence cases are deferred to human review.","feed_headline":"Image-aware prompt mixtures improve AI judge calibration and accuracy","feed_subtitle":"On two text-to-image benchmarks the method lowers calibration error while raising agreement with human raters.","key_machinery":"The load-bearing object is the soft image-conditional mixture: a set of prompts treated as latent variables, a variational distribution $q(a|z)$ with learnable weights $w_{za}$ per visual cluster $z$, and a soft assignment $p(z|x)$ derived from cosine similarities between the image embedding and cluster centroids. Spherical k-means on a pretrained image encoder partitions an unlabeled support set into $K$ groups; temperature $\\tau$ in the soft assignment interpolates between independent per-cluster BPE as $\\tau\\to0$ and a single global BPE as $\\tau\\to\\infty$. The training objective balances, over validation data, the sum of soft-assignment-weighted per-cluster log-likelihoods minus per-cluster entropy, which is an evidence lower bound under the conditional-independence assumptions $a\\perp x\\mid z$ and $y\\perp z\\mid a$. Inference mixes the MLLM's per-prompt probabilities according to the learned weights and the test image's soft group membership.","core_discovery":"The central claim is that an MLLM judge's reliability on a text-to-image preference question is not a global property of the prompt alone, but depends on the visual content being judged, and that this dependence can be captured with a mixture-of-experts-style ensemble. MMB forms $K$ groups by spherical k-means on image embeddings from a pretrained encoder, assigns each image a soft membership $p(z|x)$ based on cosine similarity to cluster centroids, and learns cluster-specific prompt weights $w^*_{za}$ by maximizing an evidence lower bound on validation pairs that balances per-group log-likelihood against an entropy regularizer. At test time the final preference probability is a soft mixture, $p(y|x)\\approx\\sum_z p(z|x)\\sum_a w^*_{za}\\,p(y|x,a)$, so images in different visual styles effectively use different prompt experts. At 10 prompts and 20 validation samples on HPSv2, the paper reports expected calibration error of 0.091 for MMB versus 0.114 for BPE and 0.142 for a simple average.","pith_inferences":["Beyond the paper: if prompt reliability really does vary coherently along embedding clusters, the learned cluster-to-prompt mapping could be reused as a cheap prompt-selection rule, letting a human read the highest-weighted prompt per cluster and build a specialized rubric without running the ensemble at test time.","Beyond the paper: the temperature $\\tau$ and cluster count $K$ could themselves be learned from validation data; the paper grids over $K$ and treats $\\tau$ as a fixed hyperparameter, so an adaptive version would remove the main tuning burden and sharpen comparisons with BPE.","Beyond the paper: on bias-focused benchmarks like MJBench, visual clustering may not align with the demographic axes along which judge bias varies, so the equal-preference confidence improvement may not transfer to other bias dimensions; a direct test would compare MMB's confidence on images that are visually similar but differ demographically."],"forward_implications":["Selective evaluation becomes practical: a developer can set a confidence threshold and defer low-confidence judgments to human reviewers, and the reported coverage-error curves show MMB keeps error low even as coverage rises.","The method works on closed-source judges because it only queries the model and never needs weight access or fine-tuning.","MMB is data-efficient: calibration and accuracy gains hold with as few as five validation samples and five prompts, and performance saturates at moderate cluster counts, so the added machinery is cheap to tune.","The cluster-level weights are interpretable, revealing which prompt personas are most reliable for which visual styles, a diagnostic that single-prompt selection hides.","The same scaffold transfers to other multimodal judgment tasks, including VQA, content moderation, and ordinal Likert-style ratings, without modifying the base model."],"supporting_citations":[{"why":"the Bayesian Prompt Ensemble method that MMB extends from text-only classification to multimodal inputs","marker":"[59]"},{"why":"the HPSv2 human-preference benchmark used for the main calibration and accuracy experiments","marker":"[65]"},{"why":"the MJBench bias benchmark adapted into the synthetic equal-preference confidence test","marker":"[11]"},{"why":"supplies the pretrained image encoder whose embeddings define the visual clusters","marker":"[52]"},{"why":"the closed-source MLLM used as the judge model in all experiments","marker":"[30]"},{"why":"defines the calibration metrics (ECE and MCE) used to measure improvement","marker":"[23]"},{"why":"provides the fast spherical k-means implementation used to build the clusters","marker":"[33]"},{"why":"the false-discovery-rate correction that underpins the statistical significance comparisons","marker":"[5]"}],"fun_headline_variants":["AI judge calibration gets a multimodal boost","Image clusters refine AI judge prompts","Visual context keys to AI judge accuracy","Multimodal ensembles calibrate AI judges","Calibrating AI judges with visual prompt groups"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains rest on the assumption that images which are close in the embedding space used for clustering also tend to agree on which evaluation prompt is most reliable; if prompt reliability does not vary coherently across these visual clusters, the cluster-specific weights add no useful signal.","fun_headline_variants_meta":{"raw":{"variants":["AI judge calibration gets a multimodal boost","Image clusters refine AI judge prompts","Visual context keys to AI judge accuracy","Multimodal ensembles calibrate AI judges","Calibrating AI judges with visual prompt groups"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1644,"prompt_tokens":981,"completion_tokens":663,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":600}},"tokens_in":597,"tokens_out":663,"duration_ms":6197,"temperature":1.0,"reasoning_tokens":600,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:59:14.726633+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MMB on a preference benchmark whose images all come from one narrow visual style, so every image lands in essentially one cluster, and compare it with BPE on the same data; if MMB's calibration improvement persists, the advantage is not caused by visual conditioning, undercutting the paper's explanation.","supporting_citations":[{"cited_title":"Bayesian prompt ensembles: Model uncer- tainty estimation for black-box large language models","cited_arxiv_id":null,"evidence_quote":"the Bayesian Prompt Ensemble method that MMB extends from text-only classification to multimodal inputs"},{"cited_title":"Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever","cited_arxiv_id":null,"evidence_quote":"supplies the pretrained image encoder whose embeddings define the visual clusters"},{"cited_title":"Weinberger","cited_arxiv_id":null,"evidence_quote":"defines the calibration metrics (ECE and MCE) used to measure improvement"},{"cited_title":"The control of the false discovery rate in multiple testing under dependency","cited_arxiv_id":null,"evidence_quote":"the false-discovery-rate correction that underpins the statistical significance comparisons"}],"review_version":2}