REVIEW 4 major objections 6 minor 35 references
A New Method to Capturing Compositional Knowledge in Linguistic Space
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that a frozen CLIP model can gain compositional understanding by inverting each image into a pseudo-token, without any hard-negative training data.
desk verdict Interesting idea, but the evaluation protocol leaks the test image into the text and the reported gains are not about compositionality. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the pseudo-token: a learned embedding vector that stands in for an image inside CLIP's text encoder, written as $ in prompt templates such as 'A photo of a $' and 'A photo of no $'. The 'no' logical caption replaces hard negatives: the triplet loss pulls the image closer to 'a photo of a $ {caption}' than to 'a photo of no $ {negative caption}', so the token must encode enough structure to separate yes from no. A language-model-generated context loss regularizes the token so it combines naturally with ordinary words rather than collapsing into a sparse embedding. The two-stage pipeline first optimizes pseudo-tokens per image (OTI), then distills them into a feed-forward network Θ that maps any CLIP image feature to a token in about two seconds, making the procedure usable at scale.
What would settle it
At inference, replace the image-derived pseudo-token with a random token or with the pseudo-token of a different image in both the 'yes' and 'no' prompts; if accuracy on SugarCREPE or Winoground stays near the reported level, the margin is not coming from the image-specific token and the claimed mechanism is not what drives the scores.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that inverting an image into a single pseudo-token—a vector inserted into CLIP's token-embedding space as if it were a word—lets a frozen CLIP make compositional distinctions it could not make before. The authors define the match between an image and a caption by comparing the similarity of the image to the caption augmented with 'a photo of a $' against the similarity of the image to rival captions augmented with 'a photo of no $'. The pseudo-token is optimized with a triplet loss on unlabeled images, using language-model-generated context sentences to regularize token interaction, and a distilled three-layer network then produces the token in about two seconds. Reported results include 94.93 average accuracy on SugarCREPE with ViT-B/32 (versus 88.48 for the best compared method on the same backbone) and a Winoground group score of 42.75 versus 7.50 for the original CLIP.
Load-bearing premise
The load-bearing premise is that a pseudo-token learned from the test image does not itself leak which caption is correct, so the yes-over-no margin measures caption comprehension rather than image recognition.
Editorial extensions
If this is right
- YUKINO achieves the highest reported scores on SugarCREPE and Winoground across several CLIP backbones in the paper's comparisons.
- The method improves compositional reasoning without hard-negative training data or fine-tuning, using only unlabeled images to train the inversion network.
- Models fine-tuned on hard negatives can show inflated SugarCREPE numbers yet still fail Winoground's mixed-pairing test, while YUKINO improves on both, which the paper presents as evidence that its gain is compositional rather than artifact-based.
- Knowledge distillation cuts inversion time from about 108 seconds per image to about 2 seconds with accuracy at least matching the optimization-based version.
Reading between the lines
- The same yes/no pseudo-token comparison could be applied to any frozen vision-language model or any benchmark that asks whether a text describes an image, turning compositional evaluation into a two-prompt retrieval problem.
- One testable extension is to check whether the pseudo-token alone can retrieve or reconstruct its source image; if it can, some of the benchmark margin may come from token-image correlation rather than caption structure.
- The reliance on language-model-generated context suggests that the diversity and quality of that context, not just the inversion loss, may be a hidden driver of the gains; a deliberately weak language model could reveal how much headroom the regularization actually provides.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a new task, Zero-Shot Compositional Understanding (ZS-CU), and a method, YUKINO, that uses textual inversion to map a query image to a pseudo-token in CLIP's text-embedding space. In a first stage, pseudo-tokens are optimized with a triplet loss that pulls 'A photo of a $' toward the image and pushes 'A photo of no $' away, together with GPT-generated regularization captions. In a second stage, a three-layer network is distilled from the optimized tokens so that any image can be inverted quickly. At test time, the predicted pseudo-token is inserted into 'yes' and 'no' versions of candidate captions, and image-to-text retrieval is decided by comparing the similarity of a 'yes' caption with the image against the similarity of a 'no' caption with the same image. The paper reports large gains on SugarCREPE and Winoground and presents ablations of the regularization and distillation losses.
Significance. If the reported results were valid, the paper would offer a lightweight, test-time way to improve CLIP's compositional understanding without hard negatives, and the distillation of iterative textual inversion into a feed-forward network is a practical speed-up. The 'no' regularization idea is a plausible alternative to hard negatives. However, the headline numbers are produced by an evaluation protocol that is circular with respect to the method: the image-specific pseudo-token is trained to make exactly the yes/no comparison used at test time. The paper does not release code, does not report a standard image-retrieval benchmark, and reports only the image-to-text direction on SugarCREPE. The significance of the contribution can therefore not be assessed from the current evidence.
major comments (4)
- [Section III-A and Eq. (7)] The evaluation protocol is circular with respect to the method and does not measure caption composition. At inference, the pseudo-token v* is predicted from the query image I and inserted into a 'yes' template ('A photo of a $ that {caption}') and a 'no' template ('A photo of no $ that {caption}'). The OTI objective in Eq. (1) explicitly trains v* so that 'A photo of a v*' is close to I and 'A photo of no v*' is far from I. The accuracy rule in Eq. (7), s(T_t,I) > s(T_n,I), therefore compares a caption in the token-favorable template with a caption in the token-unfavorable template; the outcome is dominated by the image-specific token and the yes/no wording, not by whether the caption correctly describes the image's compositional content. A semantically wrong caption in the 'yes' form can beat a correct caption in the 'no' form. The reported SugarCREPE and Winoground gains in Tables I and II therefore do not establish compositional understanding. The paper provides no control with a random pseudo-token, a fixed pseudo-token, a pseudo-token from a different image, or both candidate captions in the same template.
- [Equations (8)-(10) and Section VI-B] The Winoground text, image, and group scores inherit the same shortcut. In Eq. (8), s(T_t,I_t) uses the pseudo-token for I_t in the 'yes' template while s(T_n,I_t) uses the same token in the 'no' template; the first quantity is trained to be large and the second to be small. In Eq. (9), s(T_t,I_t) compares a caption carrying the I_t-specific token against the other image I_n, so the margin is again aided by the token rather than by caption content. The large Winoground scores in Tables II and VI and the separation in Fig. 3 are the expected consequence of the yes/no training objective, not evidence of compositional matching.
- [Section IV-A3 and Table I] Only the image-to-text direction of SugarCREPE is reported, although SugarCREPE supports both image-to-text and text-to-image evaluations. The image-to-text direction is exactly the one in which an image-specific pseudo-token can be inserted into every candidate caption; the text-to-image direction would require a different protocol and is not reported. Reporting only the favorable direction further obscures whether the method improves compositionality or merely exploits the test-time token.
- [Section IV-C and the abstract] The abstract claims 'significant improvements in image retrieval tasks,' but no standard image-text retrieval benchmark (for example, COCO or Flickr30K) is reported. The only retrieval-style result is the Winoground Image Score, which is subject to the same circular evaluation described above. The image-retrieval claim is therefore unsupported by the experiments in the manuscript.
minor comments (6)
- [Section I and Table II] There are typographical errors, including 'preformance' in the introduction and 'Dtaset' in the Table II caption.
- [Section III-B, after Eq. (2)] The notation dT_no and dg_no is used without definition; the 'no' regularization loss is not specified as precisely as the 'yes' loss, which makes the objective in Eq. (3) incomplete as written.
- [Equation (5)] The contrastive loss notation is ambiguous: γ is written as a vector in the numerator but as a set with index j in the denominator, and the index k in the second sum is not clearly tied to the batch position.
- [Section IV-B and Section VI-A1] The method is called zero-shot, but Θ is trained on the ImageNet-1K test split as unlabeled data and hyperparameters are selected on COCO validation retrieval performance; this should be clarified and the term 'zero-shot' qualified.
- [Section IV-A1] The text says 'We trained CLIP on the same pretraining dataset, LAION-2B,' which conflicts with the description of CLIP as frozen; presumably an OpenCLIP checkpoint is used, and this should be stated precisely.
- [Table VI] Under the ViT-L-14 backbone, YUKINO-OTI has lower Single Image and Single Text scores than the CLIP baseline (54.75 vs. 60.13 and 52.50 vs. 55.00), which is not discussed in the text claiming improvements across all backbones.
Circularity Check
SugarCREPE/Winoground gains are forced by the evaluation protocol: the image-specific pseudo-token is trained to produce the same yes/no margin that the benchmark then measures.
-
self definitional
[Section III-A inference protocol; Section VI-B Eq. (7); training objective Eq. (1) in Section III-B]
"In inference, given a query (I, T), we add “yes” prompt and “no” prompt to each caption T to generate “yes” caption T t and “no” caption T n. After that we predict the pseudo-token corresponding to I by Θ and concatenate it to T t and T n. ... acc(I, Tt, Tn) = 1 if s(T t, I) > s(T n, I) ... Ltri = max{∥f − gyes∥2 − ∥f − gno∥2 + 1, 0}"
The benchmark decision rule is the same inequality that OTI trains v* to satisfy: Eq. (1) minimizes the distance from the image to 'A photo of a $' relative to 'A photo of no $', and Eq. (7) scores a correct match when the yes-template caption beats the no-template caption. Because the pseudo-token is predicted from the very image being scored (and is optimized/distilled to make this margin), the sign of the inequality is dominated by the image-specific token and the yes/no template, not by whether the caption content is compositionally correct. The paper offers no control with a random or fixed pseudo-token, so the reported >8% SugarCREPE and Winoground improvements are a by-construction consequence of the training objective rather than evidence of compositional understanding.
full rationale
The central empirical claim of YUKINO is that it improves compositional understanding of a frozen CLIP without hard negatives. Tracing the derivation chain, the method's training objective (Eq. 1) explicitly optimizes the pseudo-token v* so that the image is closer to 'A photo of a $' than to 'A photo of no $'. The evaluation (Section III-A and Eq. 7) then decides a match by comparing a yes-template caption containing that same image-specific v* against a no-template caption containing it. This makes the measured accuracy a re-statement of the training margin: for any caption pair, the yes/no template plus the image-tuned token tends to fix the inequality regardless of caption content. Winoground equations (8)-(10) inherit the same yes/no construction. No control experiments (random pseudo-token, fixed pseudo-token, pseudo-token from another image, or raw CLIP scoring on the same template) are reported, so the gains cannot be attributed to compositionality. This is not a matter of self-citation or imported uniqueness; it is a structural equivalence between the fitted objective and the evaluation metric. The non-circular parts of the paper (OTI-to-network distillation, regularization ablations) do not rescue the headline benchmark claim. Score 8: the central claim reduces by construction to the trained yes/no margin, though the method itself is not vacuous as a retrieval-oriented image-to-token mapping.
Assumptions & free parameters
free parameters (8)
- lambda_tri =
1.0
- lambda_OTI_gpt =
0.5
- lambda_CLR =
1.0
- lambda_gpt =
0.5
- temperature tau =
0.25
- k for OTI =
15
- k for Theta =
150
- learning rates =
swept
assumptions (5)
- domain assumption CLIP's text encoder can meaningfully compose the word 'no' with a pseudo-token to express logical negation.
- domain assumption A single pseudo-token is sufficient to capture the compositionally relevant structure of an image.
- ad hoc to paper Appending an image-derived pseudo-token to a caption does not dominate the caption's semantic content during retrieval.
- domain assumption The unlabeled test split of ImageNet1K is an appropriate training distribution for zero-shot compositional understanding.
- domain assumption Captions generated by GPT-Neo-2.7B are a valid context for regularizing pseudo-token interaction.
invented entities (1)
-
Pseudo-token v*
Cite this review
Pith. "Pith review of A New Method to Capturing Compositional Knowledge in Linguistic Space." pith.science (2026). https://pith.science/paper/KDFJTJ6E
@misc{pith2026241215632,
author = {Pith},
title = {Pith review of: A New Method to Capturing Compositional Knowledge in Linguistic Space},
year = {2026},
howpublished = {\url{https://pith.science/paper/KDFJTJ6E}},
note = {Machine review of arXiv:2412.15632}
}
read the original abstract
Compositional understanding allows visual language models to interpret complex relationships between objects, attributes, and relations in images and text. However, most existing methods often rely on hard negative examples and fine-tuning, which can overestimate improvements and are limited by the difficulty of obtaining hard negatives. In this work, we introduce Zero-Shot Compositional Understanding (ZS-CU), a novel task that enhances compositional understanding without requiring hard negative training data. We propose YUKINO (Yielded Compositional Understanding Knowledge via Textual Inversion with NO), which uses textual inversion to map unlabeled images to pseudo-tokens in a pre-trained CLIP model. We propose introducing "no" logical regularization to address the issue of token interaction in inversion. Additionally, we suggest using knowledge distillation to reduce the time complexity of textual inversion. Experimental results show that YUKINO outperforms the existing multi-modal SOTA models by over 8% on the SugarCREPE benchmark, and also achieves significant improvements in image retrieval tasks.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Crepe: Can vision-language foundation models reason compositionally?,
Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna, “Crepe: Can vision-language foundation models reason compositionally?,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 10910–10921
work page 2023
-
[2]
When and why vision-language models behave like bags-of-words, and what to do about it?,
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou, “When and why vision-language models behave like bags-of-words, and what to do about it?,” in The Eleventh International Conference on Learning Representations , 2022
work page 2022
-
[3]
Sugarcrepe: Fixing hackable benchmarks for vision- language compositionality,
Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna, “Sugarcrepe: Fixing hackable benchmarks for vision- language compositionality,” Advances in Neural Information Processing Systems, vol. 36, 2024
work page 2024
-
[4]
Winoground: Probing vision and language models for visio-linguistic compositionality,
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross, “Winoground: Probing vision and language models for visio-linguistic compositionality,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5238–5248
work page 2022
-
[5]
COLA: A Benchmark for Compositional Text-to-image Retrieval
Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan A Plummer, Ranjay Krishna, and Kate Saenko, “Cola: How to adapt vision-language models to compose objects localized with attributes?,” arXiv preprint arXiv:2305.03689, 2023
work page Pith review arXiv 2023
-
[6]
Tripletclip: Improving compositional reasoning of clip via synthetic vision-language negatives,
Maitreya Patel, Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, and Yezhou Yang, “Tripletclip: Improving compositional reasoning of clip via synthetic vision-language negatives,” arXiv preprint arXiv:2411.02545 , 2024
arXiv 2024
-
[7]
Structure-clip: Towards scene graph knowledge to enhance multi-modal structured representations,
Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, et al., “Structure-clip: Towards scene graph knowledge to enhance multi-modal structured representations,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2024, pp. 2417–2425
work page 2024
-
[8]
Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, and Amir Globerson, “Incorporating structured representations into pretrained vision & language models using scene graphs,” arXiv preprint arXiv:2305.06343 , 2023
arXiv 2023
Show all 35 references
-
[9]
Learning transferable visual models from natural language supervision,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning....
2021
-
[10]
An image is worth one word: Personalizing text-to-image generation using textual inversion,
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or, “An image is worth one word: Personalizing text-to-image generation using textual inversion,” in The Eleventh International Conference on Learning Representations , 2023
2023
-
[11]
Learn” no
Jaisidh Singh, Ishaan Shrivastava, Mayank Vatsa, Richa Singh, and Aparna Bharati, “Learn” no” to say” yes” better: Improving vision- language models via negations,” arXiv preprint arXiv:2403.20312, 2024
2024 arXiv
-
[12]
Iterated learning improves compositionality in large vision- language models,
Chenhao Zheng, Jieyu Zhang, Aniruddha Kembhavi, and Ranjay Kr- ishna, “Iterated learning improves compositionality in large vision- language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 13785–13795
2024
-
[13]
Teaching structured vision & language concepts to vision & language models,
Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky, “Teaching structured vision & language concepts to vision & language models,” in Proceedings of the IEEE/CVF Conference on Comp...
2023
-
[14]
Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs,
Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo, “Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs,” arXiv preprint arXiv:2401.02582, 2024
2024 arXiv
-
[15]
What you see is what you read? improving text-image alignment evaluation,
Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor, “What you see is what you read? improving text-image alignment evaluation,” Advances in Neural Information Processing Systems , vol. 36, 2024
2024
-
[16]
Multi-concept customization of text-to-image diffusion,
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu, “Multi-concept customization of text-to-image diffusion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1931–1941
2023
-
[17]
“this is my unicorn, fluffy
Niv Cohen, Rinon Gal, Eli A Meirom, Gal Chechik, and Yuval Atzmon, ““this is my unicorn, fluffy”: Personalizing frozen vision-language representations,” in European conference on computer vision . Springer, 2022, pp. 558–577
2022
-
[18]
Zero-shot composed image retrieval with textual inversion,
Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo, “Zero-shot composed image retrieval with textual inversion,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15338–15347
2023
-
[19]
isearle: Improving textual inversion for zero-shot composed image retrieval,
Lorenzo Agnolucci, Alberto Baldrati, Marco Bertini, and Alberto Del Bimbo, “isearle: Improving textual inversion for zero-shot composed image retrieval,” arXiv preprint arXiv:2405.02951 , 2024
2024 arXiv
-
[20]
Pic2word: Mapping pictures to words for zero-shot composed image retrieval,
Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister, “Pic2word: Mapping pictures to words for zero-shot composed image retrieval,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 1...
2023
-
[21]
Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)...
2018
-
[22]
Distilling the knowledge in a neural network,
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531 , 2015
2015 arXiv
-
[23]
Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes,
Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister, “Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes,” arXiv preprint arXiv...
2023 arXiv
-
[24]
Dinov2: Learning robust visual features without supervision,
Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al., “Dinov2: Learning robust visual features without supervision,” Transactions on Machine Learning Research ...
2024
-
[25]
Clip-kd: An empirical study of clip model distillation,
Chuanguang Yang, Zhulin An, Libo Huang, Junyu Bi, Xinqiang Yu, Han Yang, Boyu Diao, and Yongjun Xu, “Clip-kd: An empirical study of clip model distillation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024
2024
-
[26]
Laion-5b: An open large- scale dataset for training next generation image-text models,
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gor- don, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al., “Laion-5b: An open large- scale dataset for training next generation image-text models,” Advances in Neural I...
2022
-
[27]
ImageNet Large Scale Visual Recognition Challenge,
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision (I...
2015
-
[28]
Language models are few-shot learners,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al., “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877– 1901, 2020
1901
-
[29]
Im- proved baselines with visual instruction tuning,
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, “Im- proved baselines with visual instruction tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 26296–26306
2024
-
[30]
The llama 3 herd of models,
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024
2024 arXiv
-
[31]
Visual genome: Connecting language and vision using crowdsourced dense image anno- tations,
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei, “Visual genome: Connecting language and vision using crowdsourced dense image anno- tations,...
2017
-
[32]
Optimization-based Textual Inversion (OTI): We con- ducted 350 iterations on the test split of the ImageNet dataset and validated on the validation split of the COCO dataset. Learning rates were swept over {2e − 3, 2e − 2, 5e − 2}, and loss weights λOT Igpt were explored in {0...
-
[33]
The batch size is set to 256, and loss weights λgpt are swept over {0.5, 0.75, 1} to determine the best model
Textual Inversion Network Θ: We train YUKINO for 50 epochs, sweeping learning rates over {1e − 6, 1e − 5, 1e − 4} and selecting models based on retrieval performance on the COCO validation set. The batch size is set to 256, and loss weights λgpt are swept over {0.5, 0.75, 1} t...
-
[35]
A little girl sitting on top of a bed next to a lamp
Both OTI and Θ: We used the GPT-Neo-2.7B model, which has 2.7 billion parameters and was developed by EleutherAI, to generate the phrases for regularization. For each of the 20,932 class names in the Open Images V7 dataset , we pre-generated 256 phrases, with a temperature of ...
-
[150]
Training the textual inversion network Θ takes 18 hours in total on a single A6000 GPU
Using a single A6000 GPU, YUKINO (with a ViT-B/32 backbone) processes approximately 2 seconds per image with a batch size of 256. Training the textual inversion network Θ takes 18 hours in total on a single A6000 GPU. Details of the textual inversion network Θ architecture are...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.