{"id":"c41bdf66-f6b6-408f-af22-749b9e9aa660","arxiv_id":"2412.03178","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"PUNC scores text-to-image uncertainty by comparing the prompt with an LVLM caption of the generated image, and this text-space similarity outperforms image-space baselines for OOD and ambiguous prompts.","lead":"This paper presents PUNC, a method that measures uncertainty in text-to-image generation by using a vision-language model to caption generated images and then comparing that caption with the original prompt in text space. The authors report that PUNC outperforms image-space uncertainty baselines on a new benchmark, and they demonstrate applications in bias detection, copyright protection, and out-of-distribution detection.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PUNC feeds the original prompt to the LVLM when generating the caption, so S(c, c-hat) may reflect LVLM prompt-following rather than T2I fidelity; reported superiority could be an artifact of this protocol.","rationale":"Good-faith reading: the paper introduces PUNC as a prompt-space uncertainty method and evaluates it on constructed ID/OOD and aleatoric datasets. The central claim is the outperformance of image-space baselines. The most load-bearing condition is that S(c*, c-hat) measures alignment between the prompt and the image's semantic content. The Step 2 protocol violates that condition by conditioning the caption on the prompt itself, creating a direct path from c* to c-hat that bypasses the image entirely. This is independent of whether the LVLM is 'accurate': even a well-calibrated LVLM will use the prompt as a strong prior when asked to caption with it. The proposed test is cheap and decisive: it changes only the captioning prompt, so any drop in performance is attributable to the confound. If the drop is small, PUNC survives and the conditional verdict stands; if the drop is large, the paper's headline claim needs substantial qualification. I agree partially with the reader: the reader's weakest assumption (LVLM misdescription confounding the score) anticipates this, but the specific mechanism of prompt leakage is sharper and more actionable. I do not see a stronger concern: the benchmark construction and metric selection are secondary and could be addressed by the same conditional revision. Therefore I leave the reader's conditional verdict unchanged, with the added condition that the prompt-leakage ablation be performed.","tokens_in":33339,"tokens_out":5638,"duration_ms":55601,"concrete_test":"Re-run the OOD (Table 2) and aleatoric (Table 3) benchmarks with one change: in PUNC Step 2, remove the original prompt from the LVLM input and caption with a neutral instruction like 'Describe this image in detail.' Keep generated images, LVLM, and similarity metrics identical. If average AUROC/fpr95 gaps over the best baseline shrink by more than about 5-10 points, or PUNC stops outperforming, the reported superiority is substantially driven by prompt-conditioned captioning rather than T2I uncertainty. A secondary check: hold generated images fixed and compute S(c*, c-hat) under the true prompt vs a randomly mismatched prompt; if scores barely separate, the uncertainty score mostly tracks the prompt, not image content.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is hidden in PUNC Step 2 (Sec. 4.2): the LVLM receives the original prompt c*, not just the generated image, and produces caption c-hat; the score is then S(c*, c-hat). Modern LVLMs are instruction followers, so c-hat is anchored to c* independently of what the T2I model actually drew. This is not the generic 'LVLM makes mistakes' concern: it is a systematic shortcut. For Corrupted prompts the LVLM can repair spelling and grammar using the prompt itself; for OOD prompts, the prompt is a LLaVA caption from the same domain, so the LVLM can reproduce that domain's vocabulary; for Vague prompts the LVLM receives the class name and can elaborate on it. In all these cases a high AUROC can be achieved by text-prompt copying rather than by detecting that the generated image fails to satisfy the prompt. The large variation across LVLMs in Appendix C (Molmo vs LLaVA vs Llama) is consistent with this: what varies is the LVLM's prompt-following behavior, not the T2I model's uncertainty. The central claim that PUNC outperforms state-of-the-art therefore requires excluding this confound; without code, data, or error bars, the Tables 2 and 3 numbers cannot currently be attributed to genuine uncertainty estimation. The paper's Limitations paragraph mentions LVLM biases in general terms but does not address this prompt-anchoring mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PUNC, a method for quantifying uncertainty of text-to-image (T2I) models with respect to the input prompt. PUNC generates an image from the prompt, uses a large vision-language model (LVLM) to caption that image, and then computes a text-space similarity (ROUCE or BERTScore) between the original prompt and the LVLM caption, with precision and recall variants intended to separate aleatoric and epistemic uncertainty. The authors construct prompt datasets (Normal, three OOD domains, Vague, Adversarial, and two Corrupted levels), adapt image-space baselines (DDPM-OOD, LMD, 2XDM) to the prompt-conditioned setting, and evaluate AUROC/AUPR/FPR95 across four T2I models. They also present applications to politician recognition, copyrighted characters, and bias detection. The central claim is that PUNC outperforms existing uncertainty estimation techniques.","tokens_in":33606,"tokens_out":6676,"duration_ms":59708,"significance":"If the central claim were established, PUNC would be a valuable contribution: it is a simple, computationally light method that leverages LVLMs to move uncertainty quantification from image space to semantic text space, and it is the first systematic study of prompt-space uncertainty for T2I models. The paper also provides a useful benchmark structure and explores interesting downstream applications. However, the current experimental evidence is weakened by a systematic confound in the PUNC protocol (the LVLM receives the original prompt) and by benchmark construction that may separate prompt styles rather than model uncertainty. These issues are fixable with additional controlled experiments, but they are load-bearing for the paper's main claim.","major_comments":[{"comment":"The PUNC protocol feeds the original prompt c* to the LVLM together with the generated image when producing the caption c-hat. Because modern LVLMs are instruction followers, c-hat can be anchored to c* even when the T2I output is poor, so the score S(c*, c-hat) may measure LVLM prompt-following rather than T2I uncertainty. This affects all benchmark families: corrupted prompts can be auto-corrected from the text, OOD prompts (which are LLaVA captions) can be echoed in domain vocabulary, and vague prompts can be elaborated from the class name. The large spread across LVLMs in Appendix C (Molmo vs. Llava vs. Llama) is consistent with this shortcut, and the perfect 100% AUROC on Vague in Table 3 is suspicious. The Limitations paragraph mentions LVLM biases in general but does not address this specific anchoring mechanism. Please run a controlled ablation in which the LVLM captions the image without receiving the prompt (e.g., a neutral instruction), and include a text-only baseline that computes the score from c* alone without the image. This is necessary to attribute the reported AUROC values to genuine T2I uncertainty.","section":"Sec. 4.2"},{"comment":"The benchmark labels are defined by prompt source and construction, not by independently verified T2I uncertainty. The Normal prompts are GPT-4 descriptions of ImageNet images, while the OOD prompts are LLaVA-Next captions of remote-sensing, texture, and microscopic images; these sets differ in vocabulary, sentence length, and captioner identity. Vague prompts are short templates, and Corrupted prompts are truncated or grammatically damaged Normal prompts. As a result, the AUROC values may reflect these stylistic differences rather than the model's uncertainty about the prompt. The paper does not provide an independent check that the constructed OOD prompts actually induce high epistemic uncertainty in the T2I models (e.g., human or LVLM ratings of image-prompt alignment). Please include a control experiment in which prompt style is matched across ID and OOD (same captioner, similar length and vocabulary), or otherwise demonstrate that the separation is driven by model uncertainty.","section":"Sec. 5"},{"comment":"The claim that 'PUNC outperforms state-of-the-art uncertainty estimation techniques' is not supported across all reported settings. On the Texture OOD dataset, PUNC's AUROC is about 54.95-57.01% with Molmo, which is substantially lower than DDPM-OOD mse (72.64%) and LMD mse (69.68%). Conversely, on the Vague dataset, PUNC achieves near-perfect AUROC (99.99-100%) and 0% FPR95, which is implausibly high for an uncertainty measure and likely reflects a trivial separation of short template prompts from long descriptive prompts. The paper should report global averages over all OOD datasets with per-dataset breakdowns, and temper the abstract and Section 5.1 claims to match the actual per-dataset results.","section":"Table 2/3"},{"comment":"The evaluation reports point estimates of AUROC, AUPR, and FPR95 with no confidence intervals, no number of repetitions, and no significance tests. The paper also does not release code, the prompt dataset, or the exact LVLM instruction used in Step 2 at the time of writing, which hampers reproducibility and makes it difficult to assess whether the observed differences (often only a few AUROC points) are robust. Please provide the exact prompt template for the LVLM, release the dataset/code, and add error bars or significance tests for the main tables.","section":"Tables 2-3"}],"minor_comments":[{"comment":"The notation 'PUNC BERT' is ambiguous: please state explicitly whether BERTScore precision or recall is used in each table, since the choice is not evident without context.","section":"Tables 2-3"},{"comment":"The LVLM prompt template used in Step 2 of PUNC is not specified; please provide the exact instruction given to the LVLM, as this is essential for reproducibility and for interpreting the anchoring effect discussed above.","section":"Appendix B"},{"comment":"The claim of being 'the first' to quantify uncertainty in T2I generation appears several times; it would be more measured to say 'to our knowledge' and to discuss any concurrent or related work on uncertainty in generative text-to-image models more thoroughly.","section":"Section 1"},{"comment":"There are minor typos, including 'Corrup Lvl2' in Table 3 and 'choosen' in Appendix D; please proofread the manuscript.","section":"Table 3"},{"comment":"The applications listed in Figure 1 (deepfake prevention, bias detection, copyright protection) are only covered in the appendix; consider adding a short pointer in the main text so readers know where these results are discussed.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper proposes a useful task and a simple, appealing method, but the prompt-anchoring confound in PUNC's Step 2 and the benchmark construction are serious enough that the central empirical claim is not yet established. The authors should be asked to provide the controlled ablations and the dataset/code. I also note that the novelty claims ('first') are stated more strongly than the related-work discussion warrants; the editor may wish to ask the authors to moderate these claims during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know: (1) this is the first paper to treat prompt-space uncertainty for text-to-image models as its own task, and PUNC is a simple, plausible method that outperforms adapted image-space baselines on its own benchmark; (2) the headline numbers are weaker than they look because the LVLM that captions the generated image is given the original prompt as input, so the similarity score may be measuring the LVLM's prompt-following, not the T2I model's uncertainty.\n\nWhat's genuinely new: the task definition is reasonable, the precision/recall decomposition of aleatoric vs epistemic uncertainty is a neat idea, and the paper runs a wide evaluation across four T2I models and three LVLMs. The applications (bias, copyright, deepfake) are speculative but show the method has practical reach. I think the reader's conditional verdict is fair.\n\nThe soft spots are real. The stress-test concern holds up on reading: in Step 2, c-hat = f_txt(c*, ...), so the caption is conditioned on the prompt. For corrupted prompts the LVLM can repair spelling; for OOD prompts it can echo the LLaVA vocabulary; for vague prompts it can elaborate on the class name. The large variation across LVLMs in Appendix C is consistent with this. The paper's Limitations paragraph mentions LVLM biases in general but doesn't address this specific mechanism.\n\nAlso, the metric is chosen per uncertainty type (ROUGE recall for epistemic, BERT precision for aleatoric) after seeing the results, and there are no error bars, code, or data. The benchmark construction has confounds: Normal uses GPT-4 ImageNet captions, while OOD uses LLaVA captions of different domains, so stylistic differences alone could separate them.\n\nThat said, the central idea isn't invalidated. A simple ablation—captioning with and without the prompt, or with a blind captioner—would settle it. If that holds, PUNC would be a solid contribution.\n\nWho's this for? People working on T2I reliability, OOD detection, or generative model evaluation. It deserves a serious referee. I'd send it out with a request for the ablation and for error bars/release.","headline":"First real attempt at prompt-space uncertainty for T2I, but the LVLM's access to the original prompt confounds the central result.","tokens_in":34166,"tokens_out":2183,"would_cite":false,"duration_ms":19413,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the uncertainty of a text-to-image model about a prompt can be measured by captioning the generated image with a vision-language model and comparing the caption to the prompt, and that this text-space score…","keywords":["uncertainty quantification","text-to-image generation","vision-language models","aleatoric uncertainty","epistemic uncertainty","out-of-distribution detection","prompt-space uncertainty","semantic similarity"],"falsifier":"Take a set of ambiguous prompts that human raters agree are ambiguous, plus crisp prompts matched for vocabulary; if PUNC's scores do not separate the two while human ratings do, the similarity proxy fails. A sharper check: hand-write captions for the same generated images without seeing the prompt, and see whether PUNC's score changes when the caption wording changes; if it does, the captioner's phrasing, not the generator's uncertainty, is driving the score.","tokens_in":33147,"feed_emoji":"🖼️","tokens_out":8109,"duration_ms":71566,"temperature":0.7,"pith_summary":"This paper establishes a new task: quantifying how uncertain a text-to-image model is about a prompt, and argues that the right place to measure that uncertainty is in semantic space rather than pixel space. It introduces PUNC, which generates an image from a prompt, uses a large vision-language model (LVLM) to caption that image, and scores similarity between the caption and the prompt. The paper claims that PUNC outperforms most adapted image-space methods on out-of-distribution, vague, corrupted, and adversarially altered prompts. It also claims that, because text similarity metrics carry precision and recall components, PUNC can separate aleatoric uncertainty (imprecise or corrupted prompts) from epistemic uncertainty (concepts the model does not know). If the claim holds, uncertainty scores for text-to-image systems can be obtained from a single generation and used for out-of-distribution detection, bias detection, deepfake monitoring, and copyright-relevant content detection.","feed_headline":"Comparing prompt and caption quantifies text-to-image uncertainty","feed_subtitle":"A single LVLM caption plus text-similarity beats pixel-space baselines on OOD, vague, and corrupted prompts.","key_machinery":"The central machinery is the prompt-caption similarity score. Given a prompt $c^*$, the text-to-image model draws one image $x \\sim p_\\theta(x|c^*)$; a large vision-language model turns the image into a caption $\\hat{c} = f^{\\mathrm{txt}}_\\omega(c^*, f^{\\mathrm{img}}_\\omega(x))$; and the uncertainty score is the similarity $S(c^*,\\hat{c})$ computed by ROUGE or BERTScore. Those text similarity metrics carry precision and recall at the token or embedding level, which is how PUNC claims to separate aleatoric uncertainty (extra concepts in the image lower precision) from epistemic uncertainty (missing concepts lower recall). This replaces the image-space similarity used by adapted baselines such as noising/reconstruction or two-sample ensembling.","core_discovery":"On its own terms, the paper's central discovery is that prompt-space semantic uncertainty can be captured by comparing the original prompt with an LVLM-generated caption of the output image, and that this text-space comparison outperforms image-space methods that noise, mask, or ensemble generations. Uncertainty is defined semantically: aleatoric uncertainty arises when a prompt allows multiple concepts (vagueness, typos, corruption), and epistemic uncertainty arises when the model lacks knowledge of a concept in the prompt (out-of-distribution concepts). PUNC computes $S(c^*, \\hat{c}) = \\mathrm{sim}(c^*, \\hat{c})$ between the prompt and the caption, and uses the precision/recall asymmetry of ROUGE or BERTScore to attribute low recall to epistemic uncertainty and low precision to aleatoric uncertainty. The paper's experiments span four text-to-image models and report that PUNC outperforms most existing uncertainty techniques on average across the out-of-distribution, vague, adversarial, and corrupted prompt datasets.","pith_inferences":["A testable extension is to replace the caption-similarity step with a direct LVLM answer to the question \"does this image match the prompt?\" and compare that score with PUNC, which would show whether the caption bottleneck is the limiting factor.","Because PUNC's score depends on the LVLM's wording, prompts whose meaning is faithfully preserved but phrased differently may be scored as uncertain; calibrating the similarity metric on paraphrases of the same prompt would test this sensitivity.","The same prompt-caption protocol could be applied to video or audio generation by captioning the output modality and comparing the caption with the conditioning text.","The paper's own limitations section concedes that the LVLM may inject its own biases or inaccuracies inherited from training data and that the analysis is English-only; a robust deployment would need captioner calibration and multilingual coverage."],"forward_implications":["Single-generation scoring: PUNC needs only one image per prompt plus a caption, so it can score any pretrained text-to-image model without retraining or ensembling.","Precision/recall separation: using ROUGE recall to detect missing concepts and precision to detect extra concepts lets a practitioner tell whether a failure comes from an under-specified or corrupted prompt (aleatoric) or from an unfamiliar concept (epistemic).","Benchmark availability: the released prompt datasets covering normal, out-of-distribution, vague, adversarial, and corrupted prompts let other text-to-image models be scored on the same uncertainty benchmark.","Application signal: the uncertainty score can flag generated images containing named politicians or copyrighted characters, giving a concrete signal for deepfake-style and copyright-relevant content monitoring."],"supporting_citations":[{"why":"Supplies the Normal in-distribution prompt dataset used as the reference set for all uncertainty detectors.","marker":"[13]"},{"why":"Captions the remote-sensing, texture, and microscopic images that form the out-of-distribution epistemic prompt datasets.","marker":"[36]"},{"why":"Baseline adapted from image-space out-of-distribution detection via noising and reconstruction; the comparison PUNC must beat.","marker":"[23]"},{"why":"Baseline adapted from manifold-denoising out-of-distribution detection; the second image-space comparison.","marker":"[39]"},{"why":"Provides the vision-language captioner at the core of PUNC's prompt-caption comparison.","marker":"[11]"},{"why":"ROUGE supplies the token-level precision/recall similarity used to disentangle aleatoric and epistemic uncertainty.","marker":"[37]"},{"why":"BERTScore supplies the embedding-level semantic similarity alternative to ROUGE in PUNC.","marker":"[69]"},{"why":"Source of the Adversarial prompt dataset used to test aleatoric uncertainty.","marker":"[70]"}],"fun_headline_variants":["Text-space comparison wins for T2I uncertainty","PUNC uses LVLM captions to gauge text-to-image uncertainty","Prompt-caption similarity separates T2I uncertainty types","Caption-prompt matching outperforms pixel-based T2I uncertainty"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"PUNC assumes that the similarity between the original prompt and a caption of the generated image, written by an AI model that describes images, truly reflects how unsure the image generator is; if the describer makes mistakes or uses different wording, those errors get counted as generator uncertainty, and the benchmark also assumes that prompts about rare subjects, vague wording, and corrupted text really do produce the two kinds of uncertainty they are meant to test.","fun_headline_variants_meta":{"raw":{"variants":["Text-space comparison wins for T2I uncertainty","PUNC uses LVLM captions to gauge text-to-image uncertainty","Prompt-caption similarity separates T2I uncertainty types","Caption-prompt matching outperforms pixel-based T2I uncertainty"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1249,"prompt_tokens":984,"completion_tokens":265,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":198}},"tokens_in":600,"tokens_out":265,"duration_ms":3320,"temperature":1.0,"reasoning_tokens":198,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:40:38.695215+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of ambiguous prompts that human raters agree are ambiguous, plus crisp prompts matched for vocabulary; if PUNC's scores do not separate the two while human ratings do, the similarity proxy fails. A sharper check: hand-write captions for the same generated images without seeing the prompt, and see whether PUNC's score changes when the caption wording changes; if it does, the captioner's phrasing, not the generator's uncertainty, is driving the score.","supporting_citations":[{"cited_title":"Gpt4image: Can large pre-trained models help vision models on perception tasks? arXiv e-prints , pages arXiv–2306, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the Normal in-distribution prompt dataset used as the reference set for all uncertainty detectors."},{"cited_title":"Denoising diffusion models for out-of-distribution detection","cited_arxiv_id":null,"evidence_quote":"Baseline adapted from image-space out-of-distribution detection via noising and reconstruction; the comparison PUNC must beat."},{"cited_title":"Unsupervised out-of-distribution detection with diffusion inpainting","cited_arxiv_id":null,"evidence_quote":"Baseline adapted from manifold-denoising out-of-distribution detection; the second image-space comparison."},{"cited_title":"Rouge: A package for automatic evaluation of summaries","cited_arxiv_id":null,"evidence_quote":"ROUGE supplies the token-level precision/recall similarity used to disentangle aleatoric and epistemic uncertainty."},{"cited_title":"To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images","cited_arxiv_id":null,"evidence_quote":"Source of the Adversarial prompt dataset used to test aleatoric uncertainty."}],"review_version":1}