{"id":"d84fe610-1152-44ba-89df-6c82e075e8f3","arxiv_id":"2412.15632","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"YUKINO reports state-of-the-art scores on SugarCREPE and Winoground by injecting image-derived pseudo-tokens into captions and comparing yes versus no prompts, a protocol that likely inflates the results.","lead":"A new method, YUKINO, uses textual inversion to add image-specific pseudo-tokens to captions and claims large gains on compositional benchmarks for CLIP. The reported gains appear to come from a changed evaluation protocol rather than from improved compositional understanding.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported SugarCREPE/Winoground gains are likely an artifact of the evaluation protocol: the image-specific pseudo-token plus the yes/no template creates a shortcut margin that does not test caption content.","rationale":"The reader's weakest assumption identifies the same load-bearing flaw: the inference protocol compares a yes-augmented caption against a no-augmented caption using a pseudo-token derived from the target image, so the margin is nearly independent of caption content. My reading of the methodology confirms this. In OTI (Section III-B), the triplet loss Eq. (1) pulls 'A photo of a $' toward the image and pushes 'A photo of no $' away; at inference this same pair is inserted into every candidate caption. Thus the 'no' token acts as a global negative bias that can make any caption in the yes form beat any caption in the no form. The absence of any control baseline (random token, fixed token, or standard CLIP scoring) means the reported numbers cannot be attributed to compositional understanding. I found no independent support, such as released code or machine-checked proofs, that would mitigate this. The paper's internal ablation does not address the protocol shortcut, because all ablations use the same yes/no comparison. Therefore the reader's REJECT verdict is appropriate, and my analysis does not change it.","tokens_in":14903,"tokens_out":3732,"duration_ms":32831,"concrete_test":"Re-run the SugarCREPE and Winoground evaluations with Θ's predicted pseudo-token replaced by the pseudo-token of a randomly chosen different image, keeping all yes/no prompts and the decision rule otherwise identical. If accuracy remains near the reported levels (e.g., ~95% on SugarCREPE), the result is driven by the yes/no template rather than by image-specific compositional knowledge.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that YUKINO improves CLIP's compositional understanding without hard negatives. That claim rests entirely on the evaluation protocol in Section III-A and Eq. (7), and that protocol is invalid for measuring compositionality. At inference, the pseudo-token v* is predicted from the query image I itself, then inserted into every candidate caption via 'A photo of a $ that {caption}' (the yes version) and 'A photo of no $ that {caption}' (the no version). SugarCREPE accuracy is then computed as s(yes version of the true caption, I) > s(no version of the false caption, I). Because v* is optimized so that 'A photo of a $' is close to I and 'A photo of no $' is far from I, the sign of this inequality is dominated by the yes/no template plus the image-specific token. A semantically wrong caption placed in the yes template will typically beat a correct caption placed in the no template, regardless of caption content. The paper provides no control with a random pseudo-token, a fixed pseudo-token, a pseudo-token from a different image, or the unmodified CLIP scoring function. The reported >8% SugarCREPE improvement and the Winoground gains therefore do not establish compositional understanding; they measure a shortcut in the scoring rule.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new task, Zero-Shot Compositional Understanding (ZS-CU), and a method, YUKINO, that uses textual inversion to map a query image to a pseudo-token in CLIP's text-embedding space. In a first stage, pseudo-tokens are optimized with a triplet loss that pulls 'A photo of a $' toward the image and pushes 'A photo of no $' away, together with GPT-generated regularization captions. In a second stage, a three-layer network is distilled from the optimized tokens so that any image can be inverted quickly. At test time, the predicted pseudo-token is inserted into 'yes' and 'no' versions of candidate captions, and image-to-text retrieval is decided by comparing the similarity of a 'yes' caption with the image against the similarity of a 'no' caption with the same image. The paper reports large gains on SugarCREPE and Winoground and presents ablations of the regularization and distillation losses.","tokens_in":15080,"tokens_out":10426,"duration_ms":85135,"significance":"If the reported results were valid, the paper would offer a lightweight, test-time way to improve CLIP's compositional understanding without hard negatives, and the distillation of iterative textual inversion into a feed-forward network is a practical speed-up. The 'no' regularization idea is a plausible alternative to hard negatives. However, the headline numbers are produced by an evaluation protocol that is circular with respect to the method: the image-specific pseudo-token is trained to make exactly the yes/no comparison used at test time. The paper does not release code, does not report a standard image-retrieval benchmark, and reports only the image-to-text direction on SugarCREPE. The significance of the contribution can therefore not be assessed from the current evidence.","major_comments":[{"comment":"The evaluation protocol is circular with respect to the method and does not measure caption composition. At inference, the pseudo-token v* is predicted from the query image I and inserted into a 'yes' template ('A photo of a $ that {caption}') and a 'no' template ('A photo of no $ that {caption}'). The OTI objective in Eq. (1) explicitly trains v* so that 'A photo of a v*' is close to I and 'A photo of no v*' is far from I. The accuracy rule in Eq. (7), s(T_t,I) > s(T_n,I), therefore compares a caption in the token-favorable template with a caption in the token-unfavorable template; the outcome is dominated by the image-specific token and the yes/no wording, not by whether the caption correctly describes the image's compositional content. A semantically wrong caption in the 'yes' form can beat a correct caption in the 'no' form. The reported SugarCREPE and Winoground gains in Tables I and II therefore do not establish compositional understanding. The paper provides no control with a random pseudo-token, a fixed pseudo-token, a pseudo-token from a different image, or both candidate captions in the same template.","section":"Section III-A and Eq. (7)"},{"comment":"The Winoground text, image, and group scores inherit the same shortcut. In Eq. (8), s(T_t,I_t) uses the pseudo-token for I_t in the 'yes' template while s(T_n,I_t) uses the same token in the 'no' template; the first quantity is trained to be large and the second to be small. In Eq. (9), s(T_t,I_t) compares a caption carrying the I_t-specific token against the other image I_n, so the margin is again aided by the token rather than by caption content. The large Winoground scores in Tables II and VI and the separation in Fig. 3 are the expected consequence of the yes/no training objective, not evidence of compositional matching.","section":"Equations (8)-(10) and Section VI-B"},{"comment":"Only the image-to-text direction of SugarCREPE is reported, although SugarCREPE supports both image-to-text and text-to-image evaluations. The image-to-text direction is exactly the one in which an image-specific pseudo-token can be inserted into every candidate caption; the text-to-image direction would require a different protocol and is not reported. Reporting only the favorable direction further obscures whether the method improves compositionality or merely exploits the test-time token.","section":"Section IV-A3 and Table I"},{"comment":"The abstract claims 'significant improvements in image retrieval tasks,' but no standard image-text retrieval benchmark (for example, COCO or Flickr30K) is reported. The only retrieval-style result is the Winoground Image Score, which is subject to the same circular evaluation described above. The image-retrieval claim is therefore unsupported by the experiments in the manuscript.","section":"Section IV-C and the abstract"}],"minor_comments":[{"comment":"There are typographical errors, including 'preformance' in the introduction and 'Dtaset' in the Table II caption.","section":"Section I and Table II"},{"comment":"The notation dT_no and dg_no is used without definition; the 'no' regularization loss is not specified as precisely as the 'yes' loss, which makes the objective in Eq. (3) incomplete as written.","section":"Section III-B, after Eq. (2)"},{"comment":"The contrastive loss notation is ambiguous: γ is written as a vector in the numerator but as a set with index j in the denominator, and the index k in the second sum is not clearly tied to the batch position.","section":"Equation (5)"},{"comment":"The method is called zero-shot, but Θ is trained on the ImageNet-1K test split as unlabeled data and hyperparameters are selected on COCO validation retrieval performance; this should be clarified and the term 'zero-shot' qualified.","section":"Section IV-B and Section VI-A1"},{"comment":"The text says 'We trained CLIP on the same pretraining dataset, LAION-2B,' which conflicts with the description of CLIP as frozen; presumably an OpenCLIP checkpoint is used, and this should be stated precisely.","section":"Section IV-A1"},{"comment":"Under the ViT-L-14 backbone, YUKINO-OTI has lower Single Image and Single Text scores than the CLIP baseline (54.75 vs. 60.13 and 52.50 vs. 55.00), which is not discussed in the text claiming improvements across all backbones.","section":"Table VI"}],"recommendation":"reject","confidential_remarks":"The core problem is the evaluation protocol. If the authors were to reformulate the evaluation so that the same pseudo-token appears in both candidate captions (or in neither), the reported SugarCREPE and Winoground numbers could change substantially; as submitted, the headline claim is not supported. The paper would also benefit from a standard retrieval benchmark and a comparison with Pic2Word/PALAVRA-style baselines, but these are secondary to the protocol issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper proposes YUKINO, a textual-inversion network that maps images to pseudo-tokens and uses a 'no' template to improve CLIP's compositional understanding without hard negatives. That combination is genuinely new, and the two-phase design (optimization-based inversion distilled into a feed-forward net) is reasonable. The paper also cites the relevant inversion literature (PALAVRA, Pic2Word) and includes ablations of the regularization losses. So the method itself is not nonsense.\n\nThe problem is the evaluation. In Section III-A and Eq. (7), the authors compute the score by first predicting the pseudo-token from the query image, then appending it to both the 'yes' and 'no' versions of every candidate caption. The matching rule then asks whether the yes version of one caption beats the no version of another. Since the pseudo-token was optimized so that 'A photo of a $' is close to the image and 'A photo of no $' is far, the comparison is dominated by the template plus the token, not by the caption content. A semantically wrong caption placed in the yes template will routinely beat a correct caption in the no template. The paper provides no control with a random token, a fixed token, a token from a different image, or the unmodified CLIP score. So the >8% SugarCREPE gain and the Winoground numbers do not measure compositionality; they measure the shortcut in the scoring rule.\n\nA second issue: the paper trains on the ImageNet test split and calls the method zero-shot. That is at least a mislabel. And the reported SOTA numbers are therefore not interpretable.\n\nWhat is salvageable? The 'no' regularization idea and the distillation setup could be useful if evaluated honestly. The representation distribution analysis in Fig. 3 is a good diagnostic. But the current experimental support is absent.\n\nMy recommendation: this should not go to peer review in its current form. The central claim is invalidated by the protocol, and a serious referee would have to do the authors' control experiments for them. I would desk-reject, with an invitation to resubmit after adding proper controls and fixing the data split. It is a useful cautionary example for the reading group.","headline":"Interesting idea, but the evaluation protocol leaks the test image into the text and the reported gains are not about compositionality.","tokens_in":15681,"tokens_out":3161,"would_cite":false,"duration_ms":27196,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a frozen CLIP model can gain compositional understanding by inverting each image into a pseudo-token, without any hard-negative training data.","keywords":["compositional understanding","vision-language models","textual inversion","zero-shot learning","image-text retrieval","knowledge distillation","pseudo-token","CLIP"],"falsifier":"At inference, replace the image-derived pseudo-token with a random token or with the pseudo-token of a different image in both the 'yes' and 'no' prompts; if accuracy on SugarCREPE or Winoground stays near the reported level, the margin is not coming from the image-specific token and the claimed mechanism is not what drives the scores.","tokens_in":14570,"feed_emoji":"🧩","tokens_out":7021,"duration_ms":55689,"temperature":0.7,"pith_summary":"This paper tries to establish that compositional understanding—telling apart captions that differ only in object, attribute, or relation order—can be added to a frozen CLIP model without hard-negative training data. Its method, YUKINO, learns a single pseudo-token per image in CLIP's text-embedding space and injects that token into both a 'yes' and a 'no' version of each caption at inference. The paper reports that this beats existing models on SugarCREPE by more than eight percentage points in key comparisons and lifts CLIP's Winoground group score from 7.5 to 42.75 on a ViT-B/32 backbone. If true, the result matters because it offers an unlabeled, lightweight route to compositional reasoning that does not depend on expensive curated negatives.","feed_headline":"Pseudo-token trick lifts CLIP's compositional score by 8","feed_subtitle":"An unlabeled-image inversion network lets a frozen CLIP tell swapped captions apart, beating trained baselines.","key_machinery":"The load-bearing object is the pseudo-token: a learned embedding vector that stands in for an image inside CLIP's text encoder, written as $ in prompt templates such as 'A photo of a $' and 'A photo of no $'. The 'no' logical caption replaces hard negatives: the triplet loss pulls the image closer to 'a photo of a $ {caption}' than to 'a photo of no $ {negative caption}', so the token must encode enough structure to separate yes from no. A language-model-generated context loss regularizes the token so it combines naturally with ordinary words rather than collapsing into a sparse embedding. The two-stage pipeline first optimizes pseudo-tokens per image (OTI), then distills them into a feed-forward network Θ that maps any CLIP image feature to a token in about two seconds, making the procedure usable at scale.","core_discovery":"On its own terms, the paper's discovery is that inverting an image into a single pseudo-token—a vector inserted into CLIP's token-embedding space as if it were a word—lets a frozen CLIP make compositional distinctions it could not make before. The authors define the match between an image and a caption by comparing the similarity of the image to the caption augmented with 'a photo of a $' against the similarity of the image to rival captions augmented with 'a photo of no $'. The pseudo-token is optimized with a triplet loss on unlabeled images, using language-model-generated context sentences to regularize token interaction, and a distilled three-layer network then produces the token in about two seconds. Reported results include 94.93 average accuracy on SugarCREPE with ViT-B/32 (versus 88.48 for the best compared method on the same backbone) and a Winoground group score of 42.75 versus 7.50 for the original CLIP.","pith_inferences":["The same yes/no pseudo-token comparison could be applied to any frozen vision-language model or any benchmark that asks whether a text describes an image, turning compositional evaluation into a two-prompt retrieval problem.","One testable extension is to check whether the pseudo-token alone can retrieve or reconstruct its source image; if it can, some of the benchmark margin may come from token-image correlation rather than caption structure.","The reliance on language-model-generated context suggests that the diversity and quality of that context, not just the inversion loss, may be a hidden driver of the gains; a deliberately weak language model could reveal how much headroom the regularization actually provides."],"forward_implications":["YUKINO achieves the highest reported scores on SugarCREPE and Winoground across several CLIP backbones in the paper's comparisons.","The method improves compositional reasoning without hard-negative training data or fine-tuning, using only unlabeled images to train the inversion network.","Models fine-tuned on hard negatives can show inflated SugarCREPE numbers yet still fail Winoground's mixed-pairing test, while YUKINO improves on both, which the paper presents as evidence that its gain is compositional rather than artifact-based.","Knowledge distillation cuts inversion time from about 108 seconds per image to about 2 seconds with accuracy at least matching the optimization-based version."],"supporting_citations":[{"why":"The frozen CLIP model whose image and text encoders define the shared embedding space and whose compositional weaknesses the method targets.","marker":"[9]"},{"why":"The textual inversion procedure that maps an image to a learned pseudo-token in the text-embedding space, the foundation of YUKINO.","marker":"[10]"},{"why":"Provides the 'photo of a $' prompt template and the idea of personalizing frozen vision-language representations through a learned token.","marker":"[17]"},{"why":"The SugarCREPE benchmark supplies the main compositional-accuracy measurements and the hard-negative bias analysis the paper builds on.","marker":"[3]"},{"why":"The Winoground benchmark, whose text/image/group scores test whether gains survive mixed pairings.","marker":"[4]"},{"why":"Structure-CLIP, the strongest hard-negative-supervised baseline compared on SugarCREPE and Winoground.","marker":"[7]"},{"why":"TripletCLIP, a generative-hard-negative baseline whose reported numbers are used as a comparison point.","marker":"[6]"},{"why":"The Neg-CLIP baseline, used to show that hard-negative fine-tuning inflates SugarCREPE scores without fixing Winoground.","marker":"[8]"}],"fun_headline_variants":["Pseudo-token inversion lifts CLIP compositionality by 8%","Zero-shot CLIP boost: no hard negatives needed","CLIP learns compositionality from unlabeled images alone","Image inversion unlocks CLIP's compositional blind spot","Frozen CLIP gets 8% better at composition via pseudo-tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a pseudo-token learned from the test image does not itself leak which caption is correct, so the yes-over-no margin measures caption comprehension rather than image recognition.","fun_headline_variants_meta":{"raw":{"variants":["Pseudo-token inversion lifts CLIP compositionality by 8%","Zero-shot CLIP boost: no hard negatives needed","CLIP learns compositionality from unlabeled images alone","Image inversion unlocks CLIP's compositional blind spot","Frozen CLIP gets 8% better at composition via pseudo-tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000587,"raw_usage":{"total_tokens":2742,"prompt_tokens":916,"completion_tokens":1826,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1742}},"tokens_in":532,"tokens_out":1826,"duration_ms":11740,"temperature":1.0,"reasoning_tokens":1742,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:14:33.904470+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"At inference, replace the image-derived pseudo-token with a random token or with the pseudo-token of a different image in both the 'yes' and 'no' prompts; if accuracy on SugarCREPE or Winoground stays near the reported level, the margin is not coming from the image-specific token and the claimed mechanism is not what drives the scores.","supporting_citations":[{"cited_title":"An image is worth one word: Personalizing text-to-image generation using textual inversion,","cited_arxiv_id":null,"evidence_quote":"The textual inversion procedure that maps an image to a learned pseudo-token in the text-embedding space, the foundation of YUKINO."},{"cited_title":"“this is my unicorn, fluffy","cited_arxiv_id":null,"evidence_quote":"Provides the 'photo of a $' prompt template and the idea of personalizing frozen vision-language representations through a learned token."},{"cited_title":"Sugarcrepe: Fixing hackable benchmarks for vision- language compositionality,","cited_arxiv_id":null,"evidence_quote":"The SugarCREPE benchmark supplies the main compositional-accuracy measurements and the hard-negative bias analysis the paper builds on."},{"cited_title":"Winoground: Probing vision and language models for visio-linguistic compositionality,","cited_arxiv_id":null,"evidence_quote":"The Winoground benchmark, whose text/image/group scores test whether gains survive mixed pairings."},{"cited_title":"Structure-clip: Towards scene graph knowledge to enhance multi-modal structured representations,","cited_arxiv_id":null,"evidence_quote":"Structure-CLIP, the strongest hard-negative-supervised baseline compared on SugarCREPE and Winoground."}],"review_version":1}