{"id":"6cb7ebc2-e601-4ce2-8522-19519481f35a","arxiv_id":"2608.04750","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A YOLO-based evaluation framework shows that text-to-image models consistently render the literal vehicle of a simile, a failure that CLIPScore and PickScore miss.","lead":"This paper measures how often text-to-image models draw the literal object named in a simile, such as drawing a stone for bread as hard as stone, instead of transferring only its properties. It introduces a YOLO-based evaluation framework and finds that all five tested models show this literalization bias, with Qwen-Image failing most often.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"YOLO-Det counts any appearance of the vehicle category as literalization, but in visual similes the vehicle's form may be correctly transferred to the target, so the reported bias rates may be inflated.","rationale":"The reader's weakest-assumption analysis identifies exactly the same load-bearing concern: YOLO presence is treated as literalization failure without validating that presence is a failure. This is the single most important threat to the central claim because the claim is about a specific failure mode, while the measurement captures a broader category-presence event. The concern is not resolved by the reported human correlations. Q2 validates presence, not failure; Q3 is negatively correlated with Q2, but that only shows that visible vehicles tend to reduce overall relevance in these prompts, which could reflect annotators' assumptions about what 'reflects the sentence meaning' means rather than an independent judgment of literalization. The paper's own examples of cloud-horse and branch-bat similes show that at least some intended renderings naturally contain the vehicle's form, so the operational definition in Sections 2.2.1 and 2.4.1 is not merely a theoretical edge case. The proposed test would settle the issue directly: if annotators judge most detected vehicles to be apt visual metaphors, the abstract's 'confuse it with the object' claim is overstated, and the framework would need to distinguish literal objects from form-transferred vehicles before the bias rates can be interpreted. Other concerns raised by the reader, such as regeneration circularity and missing code/data, are real but secondary; they affect reproducibility and interpretation of mitigation, not the validity of the core measurement. Because the reader already made acceptance conditional on this construct-validity issue, the verdict does not need to change.","tokens_in":18810,"tokens_out":3071,"duration_ms":41064,"concrete_test":"Sample 100 YOLO-detected images per model (500 total), stratified by model and simile template. Have annotators blind to model and hypothesis classify each detected vehicle as (i) literal vehicle appearing as an unrelated object in the scene, (ii) vehicle form applied to the target (e.g., horse-shaped clouds, bat-shaped branch), or (iii) ambiguous/other. Recompute YOLO-Det after excluding category (ii). If the adjusted detection rate drops materially (e.g., by more than 20% relative) or the model ordering changes, then the headline claim of systematic object confusion is not supported by YOLO-Det alone and the metric must be redefined or supplemented.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that models systematically 'misinterpret the metaphorical vehicle and confuse it with the object.' The operational evidence is YOLO-Det: the proportion of images in which YOLO detects the vehicle category (Section 2.4.1). This treats any detected instance of the vehicle category as literalization bias. That inference is too strong. For a simile such as 'The clouds drifted exactly like horses across the sky' (used in the paper's own Diffusion Lens examples), a correct rendering would depict clouds shaped like horses, and YOLO detecting 'horse' in that image would be a sign of successful attribute transfer, not literalization. Similarly, 'The tree branch looks like a baseball bat' can be correctly rendered as a branch shaped like a bat; a bat-shaped branch is not a literal baseball bat. The framework's design in Section 2.2.1 deliberately restricts vehicles to YOLO-detectable categories, so the detector cannot distinguish 'literal object of that category' from 'target rendered with the vehicle's form.' The human validation in Section 2.4.2 does not close this gap: Q2 asks only whether the vehicle is clearly present, not whether its presence is a failure. Q3 captures overall relevance but is a coarse judgment and does not separately score whether a detected vehicle is an apt visual metaphor. The paper acknowledges in Section 4.4 that absence of detection does not prove correct interpretation, but it never addresses the converse: presence of the vehicle category does not prove literalization. If a substantial fraction of YOLO-positive images are apt vehicle-form transfers, then YOLO-Det overstates literalization bias and may even change the model ranking.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an evaluation framework for measuring 'literalization bias' in text-to-image models, defined as the tendency to depict the metaphorical vehicle of a simile as a literal object. It constructs simile prompts whose vehicles are drawn from YOLO's 80 object categories, generates images with five t2i models (Dreamlike, PixArt, FLUX, SD3.5, Qwen-Image), measures YOLO detection of the vehicle category in generated images, validates this metric against human ratings of vehicle presence, and uses Diffusion Lens to track vehicle appearances across text encoder layers. The paper reports detection rates of 0.298–0.614 across models and proposes random and layer-based regeneration to reduce the bias.","tokens_in":19108,"tokens_out":7708,"duration_ms":84787,"significance":"The strength of the paper is the construction of a controlled, multi-model evaluation pipeline and the high correlation (r=0.826, ρ=0.798) between YOLO detection and human judgments of vehicle presence, which shows the metric reliably measures one thing: whether the vehicle appears. The paper also honestly states in Section 4.4 that absence of detection does not prove correct interpretation. However, the central claim that models 'misinterpret' or 'confuse' the vehicle with the object is not established, because the paper never tests whether detected vehicles are literal objects or the result of apt attribute transfer (e.g., clouds shaped like horses). The regeneration experiments are also circular because they select against the same YOLO metric. If the definitional gap is closed, the framework could be a useful diagnostic, but as it stands the headline claim is overstated.","major_comments":[{"comment":"The paper defines literalization bias as the literal appearance of the metaphorical vehicle and operationalizes it as YOLO detection of the vehicle category (Section 2.4.1), but it never establishes that a YOLO detection cannot arise from a correct visual simile. For example, a correct rendering of 'The clouds drifted exactly like horses' could show horse-shaped clouds, and YOLO would detect 'horse'; a branch shaped like a baseball bat would likewise trigger detection of 'baseball bat'. The human validation (Section 2.4.2) only asks whether the vehicle is clearly present (Q2) or overall relevance (Q3); it does not ask whether a detected vehicle is a literal object or a transferred attribute. Section 4.4 acknowledges that absence of detection does not prove correct interpretation, but it never addresses the converse. The reported rates (0.298–0.614) therefore conflate literalization with any depiction of the vehicle category, and the abstract's claim that models 'confuse it with the object' is not supported. Please re-annotate a sample of detected images with a judgment of literalness (is the vehicle depicted as an independent object, or is its form transferred to the target?) and report the proportion of detections that are true failures.","section":"§2.2.1, §2.4.1, §4.4"},{"comment":"Random regeneration and layer-based regeneration both select the first image in which YOLO does not detect the vehicle; consequently, the reported reductions in YOLO-Det are largely guaranteed by construction and are not independent evidence that literalization bias has decreased. For any prompt whose initial generation contains the vehicle, sampling repeatedly and stopping at the first non-detection will reduce the per-prompt detection rate unless the detection probability is 1. The paper provides no human evaluation or independent metric showing that the regenerated images have better simile understanding; the CLIPScore and PickScore results are acknowledged to be insensitive. The claim that these methods 'reduce literalization bias' should be re-framed as a procedure for reducing YOLO-Det specifically, or validated with human judgments of whether the regenerated image better reflects the simile's meaning.","section":"§3.4.1, §3.4.2"},{"comment":"The model-level YOLO-Det values in Table 1 are reported without confidence intervals or statistical tests. With approximately 1,576 prompts per model, the differences among Dreamlike (0.298), SD3.5 (0.325), and FLUX (0.349) may or may not be reliable; adding binomial confidence intervals or a paired test would substantiate the claim of 'substantial' variation. Similarly, Figure 4 presents mean first-emergence layers and coverage ratios without error bars, making it hard to evaluate whether the cross-model differences in these metrics are meaningful.","section":"Table 1, Figure 4"}],"minor_comments":[{"comment":"References [43] and [44] are the same paper (Su et al., Neural Processing Letters) and should be merged.","section":"References [43] and [44]"},{"comment":"The paper does not include a data/code availability statement; to support reproducibility, please release the G5 simile dataset and generation/inference scripts.","section":"Reproducibility"},{"comment":"In Table 2, Q1 (Context) is defined in Section 2.4.2 but results for Q1 are not reported; clarify whether they appear in the appendices or were excluded.","section":"Table 2"},{"comment":"Figure 4 would benefit from error bars or confidence intervals; in addition, the sample sizes behind the counts (e.g., 550 Persistent cases) should be stated in the caption or text.","section":"Figure 4"},{"comment":"Section 3.1 reports a non-significant ANOVA (p=0.11) and then a significant focused comparison; the focused test should be labeled as exploratory and the possibility of multiple comparisons should be addressed.","section":"§3.1"},{"comment":"In Section 3.2.2, the statement that 'higher Q2 scores tend to coincide with lower Q3 scores' is supported by the model-level pattern, but the within-model relationship is not shown; consider reporting per-model correlations.","section":"§3.2.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of ACM MM. The definitional issue is fundamental and could lead to rejection if not addressed; however, I believe it is fixable with additional annotation and a careful reframing. The circularity of the regeneration experiments is more problematic, but it too can be addressed by adding human validation. I recommend major revision and would expect the authors to either substantially revise the central claim or provide evidence that YOLO detections are indeed literal objects in a large majority of cases."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — this paper is worth a look, but read its main number with a skeptical eye. The contribution is a genuinely useful measurement instrument: a controlled simile dataset where the metaphorical vehicle is always a YOLO-detectable category, a detector-based metric (YOLO-Det) for whether the vehicle appears in the generated image, and a Diffusion Lens analysis that shows where in the text encoder the vehicle first emerges. The cross-model comparison across five architectures is well executed, and the human validation is real evidence: YOLO-Det correlates strongly with human judgments of vehicle presence (r=0.826), and the annotators agree with each other. The paper also does the right thing by checking CLIPScore and PickScore and showing they miss this failure mode.\n\nThe soft spot is the construct. YOLO-Det treats any detection of the vehicle category as literalization bias. That conflates 'the vehicle appears as a literal object' with 'the vehicle category appears at all.' For a simile like 'clouds drifted exactly like horses,' a correct rendering would show clouds shaped like horses, and YOLO would happily flag 'horse.' The human Q2 question only asks whether the vehicle is clearly present; it does not ask whether that presence is a failure. So the reported rates (Qwen-Image at 0.614, for instance) may be inflated, and the model ranking could shift if apt form-transfers were counted as successes rather than failures. The paper honestly says that absence of detection does not prove correct interpretation, but it never addresses the converse, which is exactly the case that matters.\n\nThe regeneration experiments are also weaker than they look: they select the first image in which YOLO does not detect the vehicle, then measure YOLO-Det on that selected set. That reduction is partly by construction. Whether the regenerated images actually transfer the intended attributes is not measured.\n\nMinor: no code or data released, Table 1 has no confidence intervals, and the human evaluation covers 100 images per model, which is acceptable but not generous.\n\nStill, the central phenomenon — that T2I models often render the vehicle of a simile literally — is plausible, and the paper gives the community a working benchmark for studying it. The flaw is in the interpretation of the metric, not in the experimental backbone. I would send this to a serious referee. Ask them to require an additional human study that distinguishes literal object from attribute-transferred form, and to release the dataset and code. With that, it becomes a solid and citable benchmark.","headline":"A useful new benchmark for measuring when T2I models literally render the vehicle of a simile, but the core YOLO metric conflates category presence with literalization and likely inflates the reported rates.","tokens_in":19692,"tokens_out":3428,"would_cite":true,"duration_ms":38038,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Text-to-image models systematically draw the literal object named in a simile instead of transferring its attributes.","keywords":["text-to-image generation","simile understanding","literalization bias","metaphorical vehicle","object detection","Diffusion Lens","evaluation benchmark","figurative language"],"falsifier":"Generate the same G5 prompts with the five models, take all images where YOLO flags the vehicle, and have annotators label each one as an accidental literal object, a deliberate visual metaphor, or an appropriate scene element; if a substantial share of flagged images are judged deliberate or appropriate, the YOLO-Det rate overstates literalization bias and the reported model ordering would not survive.","tokens_in":18632,"feed_emoji":"🖼️","tokens_out":7455,"duration_ms":80969,"temperature":0.7,"pith_summary":"The paper tries to establish that text-to-image models have a systematic, measurable failure with simile prompts: instead of using the metaphorical vehicle as a source of attributes for the target, they often render the vehicle itself as a literal object in the image. It names this failure literalization bias and builds an evaluation framework around it, using a controlled dataset of 1,576 simile sentences over 80 object-detectable vehicle categories and 14 templates, automatic detection of the vehicle in generated images, and Diffusion Lens tracing of when the vehicle appears in the text encoder. Across five architecturally diverse models, the vehicle is detected in images roughly 30 to 61 percent of the time, in 78 of 80 vehicle categories and all 14 templates, and the detection rate tracks human judgments of vehicle presence far better than standard similarity metrics do. If the finding holds, it means figurative language is a distinct blind spot of current text-to-image models that general-purpose alignment scores hide.","feed_headline":"Image models literalize simile prompts up to 61 percent of the time","feed_subtitle":"An object-detector benchmark finds the metaphor object drawn literally in 78 of 80 categories and all 14 templates.","key_machinery":"The load-bearing mechanism is the YOLO-Det rate: the fraction of generated images in which a pretrained object detector finds the object category named as the metaphorical vehicle. The framework's controlled construction, with vehicles restricted to 80 YOLO-detectable categories and 14 simile templates, makes each failure countable and avoids reliance on subjective aesthetic judgments. Diffusion Lens visualizations then expose the layer at which the vehicle first becomes visible in the text encoder, allowing the paper to separate failures that persist into the final image from those that appear in intermediate layers but are suppressed.","core_discovery":"The central discovery is that literalization bias is real, widespread, and measurable: given simile prompts, current text-to-image models frequently depict the metaphorical vehicle as a concrete object. Using a dataset of 1,576 maximum-agreement simile sentences covering all 80 YOLO-detectable categories and all 14 templates, the paper reports YOLO detection rates of the vehicle from 0.298 for Dreamlike to 0.614 for Qwen-Image, with detections appearing in 78 of 80 categories and every template. The YOLO-based detection rate correlates strongly with human ratings of vehicle presence (r = 0.826, rho = 0.798), while CLIPScore and PickScore correlate weakly with both vehicle presence and overall prompt-image relevance. Diffusion Lens analysis shows the literal vehicle tends to emerge early and persist in CLIP-based branches, while appearing later but still persisting in Qwen-Image, and the paper shows that both random regeneration and layer-based regeneration can reduce detection rates without modifying the prompt, though the reductions vary by model and text encoder branch.","pith_inferences":["A natural extension is to pair vehicle detection with attribute-transfer probes, for example asking whether the target actually looks harder or differently shaped; if regeneration removes the vehicle without producing the attribute, then literalization suppression is not comprehension.","The 78-of-80 coverage suggests testing the same framework on other figurative constructions, such as metonymy and hyperbole, and on multilingual similes, where the YOLO vocabulary may not be culturally portable.","The layer-based results imply that intervention at shallow text-encoder layers might reduce literalization more directly than seed sampling in CLIP-based models, though the paper only tests selecting existing layers rather than modifying representations.","If literalization bias is as general as reported, image-generation benchmarks that reward only overall text-image similarity will systematically rank literalizing models too high for figurative prompts; a literalization rate could be added as a standard reporting axis."],"forward_implications":["Simile-based prompting cannot be assumed to transfer attributes; a prompt like 'as hard as stone' tends to put a stone in the image instead of conveying hardness.","General-purpose alignment metrics such as CLIPScore and PickScore will not surface this failure, so object-level grounding checks are needed to benchmark figurative prompts.","Literalization is not confined to a few templates or vehicles: it appears in all 14 templates and 78 of 80 YOLO categories, indicating a general tendency rather than an artifact of a small prompt set.","Both random seed variation and layer-based regeneration can lower detection rates without changing the prompt, giving a cheap partial mitigation, but their effectiveness is model- and text-encoder-dependent.","Lower detection of the vehicle does not by itself prove simile comprehension; the intended attribute transfer must be evaluated separately."],"supporting_citations":[{"why":"Supplies the YOLO detector whose object categories become the operational definition of metaphorical-vehicle literal appearance.","marker":"[24]"},{"why":"Provides the fixed list of 80 YOLO-detectable categories used as the controlled metaphorical-vehicle vocabulary.","marker":"[36]"},{"why":"Diffusion Lens is the method that produces layer-wise text-encoder visualizations used to trace when the vehicle emerges.","marker":"[47]"},{"why":"Defines the CLIP text encoder whose layer structure underlies the CLIP-branch emergence and regeneration analysis.","marker":"[34]"},{"why":"Defines the T5 text encoder whose layer structure underlies the T5-branch emergence and regeneration analysis.","marker":"[35]"}],"fun_headline_variants":["Image models literalize simile prompts up to 61%","Simile metaphors drawn literally by AI image models up to 61%","T2I models mistake simile vehicles for objects up to 61%","Benchmark: Image AI literalizes similes in up to 61% of cases","Object detector shows image models literalize similes up to 61%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measurement treats any YOLO detection of the vehicle category as a literalization failure, which presupposes that no correct rendering of these similes may contain an object of that category; a deliberately visual metaphor or a scene where the vehicle object is part of the intended image would be counted as a failure.","fun_headline_variants_meta":{"raw":{"variants":["Image models literalize simile prompts up to 61%","Simile metaphors drawn literally by AI image models up to 61%","T2I models mistake simile vehicles for objects up to 61%","Benchmark: Image AI literalizes similes in up to 61% of cases","Object detector shows image models literalize similes up to 61%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000653,"raw_usage":{"total_tokens":2986,"prompt_tokens":929,"completion_tokens":2057,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1961}},"tokens_in":545,"tokens_out":2057,"duration_ms":15261,"temperature":1.0,"reasoning_tokens":1961,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:26:10.373967+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate the same G5 prompts with the five models, take all images where YOLO flags the vehicle, and have annotators label each one as an accidental literal object, a deliberate visual metaphor, or an appropriate scene element; if a substantial share of flagged images are judged deliberate or appropriate, the YOLO-Det rate overstates literalization bias and the reported model ordering would not survive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the YOLO detector whose object categories become the operational definition of metaphorical-vehicle literal appearance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the fixed list of 80 YOLO-detectable categories used as the controlled metaphorical-vehicle vocabulary."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Diffusion Lens is the method that produces layer-wise text-encoder visualizations used to trace when the vehicle emerges."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the T5 text encoder whose layer structure underlies the T5-branch emergence and regeneration analysis."}],"review_version":1}