{"id":"d7f5dff0-e6a1-42f7-9352-37eeca715c13","arxiv_id":"2412.15396","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Recaptioning training images with a multimodal model and replacing CLIP's text tower with a large language model makes a standard contrastive model state-of-the-art on composition and retrieval benchmarks.","lead":"The authors show that an ordinary CLIP model can learn to understand visual relationships if it is trained on richer, automatically generated image captions and given a stronger language model for text. The recipe beats more complex specialized systems on image retrieval and composition benchmarks, suggesting data quality may matter more than architecture.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ARO gains in Table 1 are largely driven by hard-negative fine-tuning that mimics ARO's perturbation style, not by the proposed semantic guidance; without it, ARO relations drop from 94% to 65% (Table 8).","rationale":"The reader's weakest assumption concerns the factual grounding of recaptioned captions. That is a real concern, but Table 6 indicates that removing alt-text grounding costs only about 1 point on DOCCI-full (90.3 vs. 89.3), so it is not the most load-bearing issue. The hard-negative ablation in Table 8 is far more consequential: the difference between no-hard-negative and hard-negative fine-tuning is 29 percentage points on ARO relations and 11 points on attributes, while DOCCI slightly worsens. This shows that the headline ARO numbers, which are the basis for 'surpasses all bespoke architectures' and 'achieves state of the art results on compositional benchmarks,' are largely attributable to a benchmark-specific data augmentation rather than to the paper's proposed semantic guidance. The paper is transparent about this ablation, but the framing in the abstract and introduction overstates the role of recaptioning and the strong text encoder. The DOCCI retrieval results remain strong and provide independent support for the data-centric hypothesis, so a full rejection is not warranted. However, the ARO claim should be re-scoped or accompanied by evidence that the hard-negative benefits transfer to differently-constructed composition benchmarks. The verdict should remain CONDITIONAL, with the condition that the ARO results be reported without the benchmark-matched hard-negative stage or with a demonstration of transfer to held-out composition tasks.","tokens_in":16181,"tokens_out":3692,"duration_ms":31869,"concrete_test":"Train the same model (recaptioning + Gemini-8B text tower) with hard negatives generated by generic sentence scrambling (random word-order permutations) instead of the ARO-style few-shot prompt, and report ARO relations/attributes. If accuracy falls from 94%/93% toward the no-hard-negative levels of 65%/82% (Table 8), the Table 1 ARO claim is a benchmark-matched artifact rather than a general compositional understanding.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that improved semantic guidance (recaptioning + pretrained text encoder) suffices to make a standard CLIP model surpass bespoke architectures on compositional benchmarks rests heavily on ARO results (Table 1). The paper's own ablation (Table 8) shows that the hard-negative fine-tuning stage, whose prompts are specifically designed to match ARO relations/attributes evaluation style (Figure 3, Section 2.1), is the dominant factor: ARO relations accuracy jumps from 65% to 93% and attributes from 82% to 94% when this stage is added, while DOCCI-full retrieval slightly declines (91.6 to 88.1). This means the SOTA ARO numbers are not a clean demonstration of semantic guidance; they reflect training on synthetic negatives crafted to the exact perturbation taxonomy of the evaluation benchmark. The recaptioning and strong-text-encoder contributions are more visible on DOCCI (Table 5), but the abstract and introduction generalize the ARO result to 'compositional learning' broadly. Because no held-out composition test is reported for the hard-negative stage, one cannot separate a genuine improvement in relational understanding from overfitting to ARO-style negation patterns. This concern is quantitative and internal to the paper, making it more decisive than the reader's weaker assumption about caption grounding: Table 6 shows grounded vs. ungrounded captions differ by only 1 point on DOCCI (90.3 vs. 89.3).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two simple, scalable modifications to a standard two-tower CLIP model: (1) recaption 1B WebLI images with Gemini 1.5 Flash using grounded prompts that incorporate alt-text and page title, and (2) replace the randomly initialized text tower with a pretrained LLM (Gemini Flash-8B or Gemma-2B), fine-tuning only the last four layers. The model is trained with a single contrastive loss, plus two data augmentations: random sentence sampling and 64M synthetic hard negatives generated to mimic ARO-style perturbations. The authors report state-of-the-art or highly competitive results on ARO, SugarCrepe, COCO, and DOCCI retrieval, with the largest gains on DOCCI (94.5% recall@1 on test). A series of ablations (Tables 5-8) isolates the contributions of recaptioning, text-encoder pretraining, sentence sampling, and hard-negative fine-tuning.","tokens_in":16682,"tokens_out":7737,"duration_ms":66042,"significance":"If the claims hold, this is an important demonstration that supervision quality can substitute for architectural complexity in compositional vision-language learning. The paper's strengths include the clean ablation structure, the use of DOCCI as a detailed retrieval benchmark, the human-annotation analysis showing that many COCO retrieval failures are actually acceptable matches, and the open Gemma-2B text-encoder variant that partially mitigates reliance on proprietary components. The central ARO claim, however, is entangled with a hard-negative fine-tuning stage that is explicitly crafted to match ARO's perturbation taxonomy, so the significance is conditional: the recaptioning and strong-text-encoder contributions are convincingly demonstrated on DOCCI (Table 5), but the broad claim of surpassing bespoke architectures on compositional benchmarks needs a cleaner separation of the hard-negative effect.","major_comments":[{"comment":"The hard-negative fine-tuning stage is the dominant contributor to the ARO results in Table 1. Table 8 shows that adding hard negatives raises ARO relations from 65% to 93% and attributes from 82% to 94%, while slightly lowering DOCCI-full retrieval from 91.6 to 88.1. Since Figure 3 states that the synthetic negatives were produced with few-shot examples 'matching the style of ARO relations and attributes evaluation,' the Table 1 ARO numbers are partly trained to the evaluation distribution rather than being a clean demonstration of the proposed semantic guidance. The paper should either report the hard-negative ablation on a held-out composition benchmark whose perturbation style was not used to construct the negatives (SugarCrepe is a natural candidate and is not reported in Table 8), or explicitly limit the ARO claim.","section":"Section 3.5, Table 8"},{"comment":"The paper's central claim is that two changes, recaptioning and a pretrained text encoder, suffice to close the compositional gap, but the final model also includes 64M synthetic hard negatives, and the ablations show that this third ingredient is required for the ARO headline. The abstract and introduction ('simple approach boosts performance of CLIP substantially and surpasses all bespoke architectures') should be aligned with this decomposition. In addition, the hard-negative ablation in Table 8 is reported only on DOCCI-full, COCO, and ARO; no DOCCI-test numbers are given for the hard-negative stage, so it is unclear whether the strong DOCCI-test result in Table 3 is affected by the same trade-off. Please report a complete configuration-to-benchmark matrix for the main results.","section":"Section 3.5, Tables 5-8"},{"comment":"The recaptioning step is motivated by reducing hallucination, but the paper provides no quantitative evidence that the generated captions are factually grounded in the images. The only indirect evidence is Table 6, where removing alt-text grounding costs 7.7 points on COCO (39.2 vs 31.5) but only 1.0 point on DOCCI-full (90.3 vs 89.3). Because the large DOCCI gains are the strongest support for the recaptioning claim, a sample-based verification of caption-image factuality (e.g., human ratings or an object-consistency check) would materially strengthen the paper. Without it, one cannot exclude the possibility that the visual encoder learned spurious correlations that happen to match DOCCI-style captions.","section":"Section 2.1, Appendix A"}],"minor_comments":[{"comment":"The caption states 'Default and short caption length contain 133.4 and 354.8 words, respectively,' which appears to have the two values reversed or mislabeled, since the 'short' condition is intended to be more concise and Figure 4 reports a mean of 57 words for the default captions; please clarify.","section":"Table 6 caption"},{"comment":"All results are reported from single training runs without seeds, confidence intervals, or error bars; given the large claimed differences (e.g., 65% to 93% in Table 8), reporting variance across a few seeds would help assess reliability.","section":"All tables"},{"comment":"The paper relies on internal 1B WebLI data, Gemini 1.5 Flash recaptioning, and the Gemini Flash-8B text encoder, none of which are publicly available. The open Gemma-2B variant only partially mitigates this. Please state explicitly what artifacts (captions, checkpoints, or code) can be released to support reproducibility.","section":"Section 2"},{"comment":"There is a typo in the reference to Figure 2: 'a strong multimodal foundation model (Figure 2.' is missing a closing parenthesis, and the sentence should end properly.","section":"Section 2.1"}],"recommendation":"major_revision","confidential_remarks":"The ARO hard-negative concern is substantial and should be addressed before acceptance: the current Table 1 headline conflates benchmark-specific training with the proposed semantic guidance. I recommend requesting the SugarCrepe ablation for the hard-negative stage and a revised claims decomposition. The heavy reliance on proprietary data and models may also be a scope and fit consideration for the journal, though the Gemma-2B variant helps."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. The DOCCI retrieval result is real and large: a vanilla ViT-B/16 CLIP with recaptioned WebLI and a Gemini text tower gets 94.5% recall@1 on DOCCI-test, up from 58.4 for stock CLIP. That is a clean demonstration that caption quality and text encoder strength matter more than architecture. The ARO result, however, is mostly a hard-negative fine-tuning artifact. Table 8 is the tell: without the synthetic hard negatives, ARO relations sit at 65%; with them they jump to 93%. Those negatives are generated with few-shot prompts that mimic ARO's perturbation taxonomy, so the model is being trained to the evaluation distribution. The paper is transparent about this trade-off, but the abstract and intro still push \"surpasses all bespoke architectures\" as a semantic-guidance win, which is not what the ablation supports.\n\nWhat is genuinely new: combining recaptioning with an LLM text tower in a standard CLIP, and showing the additive gains on a non-hackable retrieval benchmark. The ablation is clean: recaptioning alone takes DOCCI-full from 53.5 to 75.6; the Gemini text tower from 53.5 to 67.2; together they reach 91.6. Sentence sampling adds another point. These are large, monotonic effects. The COCO analysis, where 70% of retrieval ``failures'' were judged acceptable by human raters, is also thoughtful and likely correct.\n\nSoft spots: no error bars or multiple seeds; internal WebLI data and Gemini models; no code or checkpoints. The caption-grounding worry is less of an issue than I would have guessed—Table 6 shows grounded vs ungrounded captions differ by only 1 point on DOCCI (90.3 vs 89.3). The real confound is hard negatives. Also, ImageNet zero-shot at 68.4% is below stock CLIP; the authors explain this by data distribution, which is fair, but it does mean the improvements are not universal.\n\nWho should read it: anyone working on vision-language pretraining or compositionality benchmarks. The DOCCI retrieval setup is a good new yardstick. The paper deserves a serious referee. The revision should (a) de-emphasize ARO or report the with/without hard-negative split more prominently, (b) test on a held-out composition set that is not styled like the training negatives, and (c) release at least a sample of the recaptioned captions. I would send it to review with those requests.","headline":"The DOCCI gain is real; the ARO gain is mostly hard-negative fine-tuning, and the paper knows it.","tokens_in":17063,"tokens_out":3105,"would_cite":true,"duration_ms":25906,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A standard CLIP model, trained with recaptioned images and a pretrained language-model text tower under a single contrastive loss, achieves state-of-the-art scores on compositional benchmarks and detailed image retrieval, outperforming…","keywords":["visual composition","contrastive learning","CLIP","recaptioning","image retrieval","semantic guidance","text encoder","compositional benchmarks"],"falsifier":"Train the identical recipe but swap one attribute or relation in half of the recaptioned captions so the caption no longer describes the image (human-verified), then measure ARO relations and attributes accuracy: if accuracy stays near 90%, the model is exploiting language statistics rather than grounded composition, while a sharp drop would confirm the model is reading image content.","tokens_in":16009,"feed_emoji":"🖼️","tokens_out":12172,"duration_ms":85258,"temperature":0.7,"pith_summary":"The paper's thesis is that contrastive vision-language models behave like bag-of-words reasoners not because of their architecture but because their training text is too weak, and that fixing the text is sufficient to fix the model. It shows that recaptioning 1B web images with a grounded multimodal foundation model and swapping in a pretrained language model as the text tower lifts a standard CLIP model from 59%/63% to 92%/94% on ARO relations/attributes and from 58.4% to 94.5% recall@1 on DOCCI retrieval, surpassing bespoke architectures with a single contrastive loss and no architectural changes. If this is right, the bottleneck for compositional understanding is data quality, and scaling caption quality is a cheaper and more scalable path than designing specialized losses or cross-modal encoders. The paper also argues that COCO caption retrieval is too noisy to measure such progress: human annotation of its apparent failures shows 70.2% are actually acceptable matches.","feed_headline":"Richer captions push standard CLIP to state-of-the-art composition","feed_subtitle":"Recaptioned data plus a pretrained text tower hit 92% on ARO relations and 94.5% on DOCCI retrieval.","key_machinery":"The load-bearing mechanism is the pairing of a grounded recaptioning pipeline with a mostly-frozen pretrained language-model text tower. Recaptioning replaces an average 7-word alt-text caption with a 57-word description generated by Gemini 1.5 Flash from the image plus the original alt-text and page title, with a prompt that instructs the model not to invent details; this turns a bag-of-words training target into one that names objects, attributes, relations, and spatial layout in natural language. The text tower is a pretrained Gemini 1.5 Flash-8B or Gemma-2B model whose last four layers are fine-tuned (with bidirectional attention) while the rest stay frozen, giving the contrastive objective a rich compositional language space to align image patches to. A supplementary mechanism is a set of 64M synthetic hard-negative captions generated in the style of the ARO benchmark, used for a short fine-tuning stage.","core_discovery":"The central claim is that enriching the target semantic embedding is sufficient to make a standard contrastive learning model capture visual composition. Two minimal changes accomplish this: first, replacing noisy web alt-text with long, grounded captions generated by Gemini 1.5 Flash using the image, the alt-text, and the page title as grounding, with a prompt instructing the model not to invent details; second, using a pretrained Gemini 1.5 Flash-8B (or Gemma-2B) language model as the text tower, fine-tuning only the last four layers and switching to bidirectional attention. With these changes alone, a ViT-Base CLIP model trained with a single contrastive loss surpasses bespoke architectures on ARO and SugarCrepe and achieves 94.5% recall@1 on DOCCI-test retrieval without fine-tuning on DOCCI. Ablations on 100M images show the two changes are additive, that sentence sampling helps, and that hard-negative fine-tuning trades off DOCCI for ARO.","pith_inferences":["If caption quality is the main lever, the same recaptioning recipe should also improve other contrastive training families and filtering pipelines; the paper does not test this, but it is a direct consequence of the claim that the target text is the bottleneck.","The frozen LLM text tower itself encodes language plausibility, so part of the ARO/SugarCrepe gains may come from a text-side prior rather than from image grounding; the paper's ablations do not fully separate these two sources.","The 70.2% mislabeled-failure finding suggests that other models' COCO retrieval numbers would also rise under the same human re-scoring protocol, so some of the apparent gap between this model and prior work on COCO may be benchmark noise.","Because the model was trained at 256x256 without DOCCI fine-tuning while a compared method gains over 10 points from 448x448 fine-tuning, combining the recaptioned data with higher resolution and fine-tuning is an obvious deferred experiment with likely headroom."],"forward_implications":["The gains transfer across text encoders: both Gemini 1.5 Flash-8B and Gemma-2B deliver large DOCCI improvements over a from-scratch text encoder, so the effect is not specific to one language model.","COCO is a saturated retrieval benchmark: human raters judged 70.2% of the model's apparent failures to be acceptable matches, so detailed-caption benchmarks such as DOCCI are needed to measure compositional gains.","Adding a 50% JFT mixture raises zero-shot ImageNet accuracy from 68.4% to 79.1%, indicating that the lower ImageNet score reflects the training distribution rather than a failure of fine-grained discrimination.","Hard-negative fine-tuning lifts ARO relations from 65% to 93% and attributes from 82% to 94% while slightly hurting DOCCI, showing a trade-off between compositional discrimination and detailed retrieval.","The improved alignment transfers to a practical application: the model predicts human preference for text-to-image generation better than an open-source CLIP baseline (81.1% vs 71.3% weighted accuracy)."],"supporting_citations":[{"why":"Provides the standard two-tower CLIP architecture and the open-source checkpoints whose bag-of-words behavior the paper diagnoses and improves on.","marker":"[26]"},{"why":"The multimodal model used to generate the grounded, longer captions that replace alt-text in the training data.","marker":"[27]"},{"why":"The pretrained Gemini 1.5 Flash-8B model used as the text tower, with only the last four layers fine-tuned.","marker":"[30]"},{"why":"The alternative pretrained Gemma-2B text tower used to show the approach transfers across text encoder families.","marker":"[31]"},{"why":"Supplies the English-only WebLI subset of 1B image-alt-text pairs that are recaptioned for training.","marker":"[4]"},{"why":"The ARO benchmark whose relations and attributes splits quantify the departure from bag-of-words behavior, and whose NegCLIP method is a data-side baseline.","marker":"[36]"},{"why":"The DOCCI benchmark with long human-written captions, used to measure detailed image retrieval; 94.5% recall@1 is the headline result.","marker":"[22]"},{"why":"The SugarCrepe benchmark with harder, non-hackable negatives, where the model also reaches state-of-the-art or competitive accuracy.","marker":"[11]"},{"why":"The COCO retrieval benchmark whose failures the paper re-examines with human annotation, finding most apparent errors are acceptable matches.","marker":"[19]"}],"fun_headline_variants":["Better captions push plain CLIP past specialized models","Data upgrade makes vanilla CLIP dominate composition benchmarks","Standard CLIP with grounded captions tops composition tasks","Improved captions alone lift CLIP to state-of-the-art composition","Recaptioned data and pretrained text tower: CLIP excels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The recaptioned training captions are factually accurate and grounded enough in the image that the contrastive objective learns true image-text correspondence rather than patterns in the generated language.","fun_headline_variants_meta":{"raw":{"variants":["Better captions push plain CLIP past specialized models","Data upgrade makes vanilla CLIP dominate composition benchmarks","Standard CLIP with grounded captions tops composition tasks","Improved captions alone lift CLIP to state-of-the-art composition","Recaptioned data and pretrained text tower: CLIP excels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1309,"prompt_tokens":961,"completion_tokens":348,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":281}},"tokens_in":577,"tokens_out":348,"duration_ms":3932,"temperature":1.0,"reasoning_tokens":281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:27:28.189411+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical recipe but swap one attribute or relation in half of the recaptioned captions so the caption no longer describes the image (human-verified), then measure ARO relations and attributes accuracy: if accuracy stays near 90%, the model is exploiting language statistics rather than grounded composition, while a sharp drop would confirm the model is reading image content.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Provides the standard two-tower CLIP architecture and the open-source checkpoints whose bag-of-words behavior the paper diagnoses and improves on."},{"cited_title":"Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context, 2024","cited_arxiv_id":null,"evidence_quote":"The pretrained Gemini 1.5 Flash-8B model used as the text tower, with only the last four layers fine-tuned."},{"cited_title":"Gemma 2: Improving open language models at a practical size, 2024","cited_arxiv_id":null,"evidence_quote":"The alternative pretrained Gemma-2B text tower used to show the approach transfers across text encoder families."},{"cited_title":"When and why vision- language models behave like bags-of-words, and what to do about it? In International Conference on Learning Repre- sentations, 2023","cited_arxiv_id":null,"evidence_quote":"The ARO benchmark whose relations and attributes splits quantify the departure from bag-of-words behavior, and whose NegCLIP method is a data-side baseline."},{"cited_title":"Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality","cited_arxiv_id":null,"evidence_quote":"The SugarCrepe benchmark with harder, non-hackable negatives, where the model also reaches state-of-the-art or competitive accuracy."}],"review_version":1}