{"id":"966d6dbb-6d44-4d31-accf-eed9efe35b48","arxiv_id":"2412.01814","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"COSMOS improves zero-shot vision-language performance by adding random text cropping and a cross-attention module to a student-teacher self-distillation loop, outperforming CLIP-style baselines trained on comparable data.","lead":"COSMOS is a training method that makes vision-language models look beyond the main object in an image by cropping both images and captions, then distilling local details into a global representation. Across retrieval, classification, segmentation, and compositional benchmarks, it reports consistent gains over CLIP-style baselines at the same or smaller image-text pair budgets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main-table CLIP/SigLIP baselines may omit the synthetic long captions used to train COSMOS, confounding the claimed consistent outperformance.","rationale":"I read the paper in good faith. The core proposal—text cropping, cross-attention, and cross-modality self-distillation—is well-motivated and the ablations provide internal support. However, the single most load-bearing threat to the central claim is not the local-global text alignment assumption (which the authors themselves discuss in Sec. 3.1 and test in Supp E.7), but the lack of disclosure about whether the reproduced CLIP/SigLIP baselines in Tables 1 and 2 were trained with the same DreamLIP synthetic long captions as COSMOS. Table 5 quantifies the effect of text augmentation: on CC3M it raises MSCOCO I2T R@1 by roughly 30 points and ImageNet top-1 by about 12 points. Without knowing the caption input for the baselines, the reported superiority over CLIP/SigLIP could be an artifact. This concern is concrete and testable from the released code. The comparison to DreamLIP, which uses long captions, is fair and shows a real advantage, so rejection is not warranted; the correct remedy is to disclose and, if needed, rerun baselines. The reader's verdict of CONDITIONAL is thus unchanged, and my stress-test does not alter it.","tokens_in":31785,"tokens_out":5331,"duration_ms":52682,"concrete_test":"Inspect the released code/configs for the CLIP and SigLIP reproductions in Table 1; if they load original captions rather than the DreamLIP synthetic captions, rerun them with the same synthetic captions (e.g., on CC3M) and compare to COSMOS. If their MSCOCO I2T R@1 increases by roughly 10 points (cf. Table 5, rows 4-5), the reported advantage over CLIP/SigLIP is primarily caption augmentation, not the cross-modality self-distillation. Also verify the same for SigLIP and for CC12M/YFCC15M/Merged-30M.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that COSMOS 'consistently outperforms previous strong baselines' (Abstract, Sec. 4). Tables 1 and 2 compare against CLIP [63] and SigLIP [88] trained on the same pre-training datasets, with footnote 'Results are reproduced with our setup for fair comparison.' However, the paper does not state whether these reproduced baselines were trained with the original short alt-text captions or with the DreamLIP synthetic long captions used to train COSMOS (Sec. 3.1, Sec. B). This matters because the component ablation in Table 5 shows that simply switching from no text augmentation to long synthetic captions (row 4 vs row 5) improves MSCOCO image-to-text R@1 from 20.6 to 50.4 and ImageNet top-1 from 22.8 to 34.4 on CC3M. If the CLIP/SigLIP rows in Tables 1 and 2 use original captions, the reported gaps (e.g., CC3M MSCOCO I2T R@1: CLIP 40.2 vs COSMOS 53.1) are mostly due to caption enrichment, not the proposed cross-modality self-distillation. The comparison to DreamLIP, which also uses long synthetic captions, is fair and supports the method, but the broad claim of outperforming all previous strong baselines is overstated if CLIP/SigLIP are not given the same caption data. This is a load-bearing ambiguity because it directly affects the headline empirical claim and is easily fixable by disclosing the exact caption inputs for each baseline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes COSMOS, a vision-language pre-training method that combines text cropping, a cross-attention module, and cross-modality self-distillation. Global and local views of both images and long synthetic captions are used to train a student model against an EMA teacher, with a cross-attention module generating cross-modal embeddings for the student. The authors evaluate on zero-shot retrieval, classification, semantic segmentation, and compositionality benchmarks, reporting gains over CLIP, SigLIP, and DreamLIP on 3M-30M training sets, and in several cases exceeding models trained on hundreds of millions or billions of pairs.","tokens_in":32092,"tokens_out":6752,"duration_ms":65829,"significance":"If the reported results hold, COSMOS is a meaningful advance because it demonstrates that cross-modal self-distillation with text augmentation can substantially improve CLIP-style representations on fine-grained and compositional tasks, with consistent gains over the direct DreamLIP baseline and even over models trained on much larger data. The paper's strengths include a clear component ablation (Table 5), reimplementation of previous SSL baselines on the same long-caption data (Supp. E.1, Table 9), additional experiments on PixelProse (Supp. E.3), and a public code release, all of which support the reproducibility of the method.","major_comments":[{"comment":"The headline claim that COSMOS 'consistently outperforms previous strong baselines' is confounded unless the CLIP/SigLIP rows in Tables 1 and 2 were trained with the same DreamLIP long synthetic captions used for COSMOS. The table footnotes say 'Results are reproduced with our setup for fair comparison,' but Sec. C.1 and Supp. D describe only batch sizes and OpenCLIP model names, not whether the reproduced CLIP/SigLIP models were trained on original alt-text or on the long captions. This matters because the component ablation in Table 5 shows that switching from no text augmentation to long synthetic captions (row 4 vs row 5) improves MSCOCO image-to-text R@1 from 20.6 to 50.4 on CC3M, and ImageNet top-1 from 22.8 to 34.4. If the CLIP/SigLIP baselines in Tables 1 and 2 use original captions, a large part of the reported gap (e.g., CC3M MSCOCO I2T R@1: CLIP 40.2 vs COSMOS 53.1) is due to caption enrichment rather than the proposed cross-modality self-distillation. The authors must disclose the exact caption inputs for each reproduced baseline and, if necessary, retrain the CLIP/SigLIP baselines with the same long captions to make the comparison fair.","section":"Sec. 4, Tables 1-2, and Supp. D"},{"comment":"The abstract and introduction state that COSMOS 'consistently outperforms previous strong baselines,' but the per-dataset classification results in Table 2 do not support 'consistently' in the sense of every dataset. On Merged-30M, DreamLIP outperforms COSMOS on Food-101 (76.4 vs 73.9), FGVCAircraft (8.3 vs 5.9), Oxford Pets (66.3 vs 62.9), Flowers102 (59.1 vs 54.6), and ImageNet (58.4 vs 57.6). COSMOS has a higher average over the 11 datasets (58.6 vs 58.4), but the difference is only 0.2 points. The claim should be qualified to say COSMOS outperforms on average or on most datasets, rather than 'consistently outperforms.'","section":"Sec. 4.2, Table 2 (Merged-30M rows)"},{"comment":"All experimental results are single-run point estimates without any measure of variance, number of seeds, or significance testing. Some of the claimed advantages are small (e.g., 57.6 vs 58.4 ImageNet, 58.6 vs 58.4 average in Table 2; 25.9 vs 25.9 MMVP in Table 4), and these differences could easily be within run-to-run noise for large-scale pre-training. The authors should either provide error bars or multiple-seed results for at least the key tables, or explicitly state that the reported numbers are single runs and avoid overinterpreting small gaps.","section":"Tables 1-4"}],"minor_comments":[{"comment":"The subscripts on the cross-attention modules appear swapped relative to the textual description: Eq. (1) uses C_T^theta to produce the image embedding h_I by attending to text tokens, while Eq. (2) uses C_I^theta to produce the text embedding h_T by attending to image tokens. The definitions in the text make the behavior clear, but the notation C_T for an image-side module and C_I for a text-side module is confusing; consider renaming to C_{I<-T} and C_{T<-I} or adding a clarifying sentence.","section":"Sec. 3.2, Eqs. (1)-(2)"},{"comment":"The row 1 'CLIP' baseline in Table 5 is explicitly trained without any augmentation (as stated in Sec. 4.5), which is a much weaker setup than the CLIP baseline reported in Table 1 (e.g., MSCOCO I2T R@1: 15.0 in Table 5 vs 40.2 in Table 1 on CC3M). The large jump from row 1 to row 5 should not be interpreted as the effect of text augmentation relative to the standard CLIP baseline; this discrepancy should be clarified in the text.","section":"Sec. 4.5, Table 5"},{"comment":"There is a typo: 'we presents results' should be 'we present results.'","section":"Sec. 4.4"},{"comment":"The phrase 'four 4-GPU machines' is ambiguous. Please specify whether the training used four machines each with four GPUs (16 GPUs total) or a single machine with four GPUs.","section":"Supp. E.4, Table 14"},{"comment":"The SILC-C baseline is marked with an asterisk indicating results are copied from its paper, but SILC-C may use a different vision encoder (e.g., ViT-L) than the ViT-B/16 used for COSMOS. Please confirm the architecture of SILC-C or add a note about the architecture mismatch.","section":"Sec. 4.3, Table 3"},{"comment":"The COSMOS loss in Eq. (6) applies symmetric InfoNCE between h_I/h_T and both CLS_t and EOT_t, which means the student cross-modal embeddings are pulled toward both teacher class tokens and teacher text tokens. The formulation is valid, but a brief intuition for why both teacher tokens are needed (rather than using only the corresponding modality tokens) would improve readability.","section":"Sec. 3.3, Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The central methodological idea is solid and the comparison to DreamLIP (which also uses long synthetic captions) is fair and favorable. The main concern is the baseline-caption ambiguity: if the reproduced CLIP/SigLIP models in Tables 1-2 were trained on original alt-text, the headline 'consistently outperforms' is largely a caption-effect result. This is easily fixable by disclosing the caption inputs and, if needed, retraining those baselines. The second issue is the overstatement of 'consistently outperforms' given the per-dataset classification results. If the authors address these, the paper would be a valuable contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Natasha, you should read this one if you care about data-efficient VLP pre-training. COSMOS takes the DINO multi-crop idea and applies it to text: global and local crops of long synthetic captions, an EMA teacher, and a small cross-attention module that lets the [cls] token read text tokens and the [eot] token read image tokens. The cross-modality self-distillation loss pulls student crops toward teacher global tokens. The pieces are known, but the specific combination is new for vision-language pre-training, and the component ablation shows each piece earns its keep. The gains are large and consistent across retrieval, classification, segmentation, and compositionality benchmarks. The paper ships code and is transparent about dataset URL losses. That is real work.\n\nThe soft spots are two. The first is load-bearing: the ablation (Table 5) shows that on CC3M, simply adding long synthetic captions as text augmentation jumps MSCOCO I2T R@1 from 20.6 to 50.4. The main tables compare COSMOS against CLIP and SigLIP 'reproduced with our setup,' but the paper never states whether those baselines were trained with the original alt-text or with the same DreamLIP long captions COSMOS uses. If the former, a large chunk of the reported gap is caption enrichment, not the cross-modality distillation. The comparison to DreamLIP (official weights, long captions) is fair and still shows COSMOS ahead, so the method itself has support; but the broad claim of 'consistently outperforms previous strong baselines' needs the baseline inputs disclosed. This is a one-paragraph fix.\n\nSecond, all results are single-run point estimates. No seeds, no variance. For the size of the claimed margins that is a minor issue, but it would be good to see at least one seed for the main table.\n\nThe E.7 ablation on text cropping strategies actually addresses the independence concern: independent sampling beats 'local within global,' so the worry about misaligned crops is less damning than it first appears. Also, the paper doesn't overclaim by avoiding loss rescaling, which is a nice simplification over SILC-type recipes.\n\nNet: this deserves peer review. The method is novel enough, the experiments are extensive, and the baseline ambiguity is fixable. I'd want the authors to specify caption inputs for every reproduced baseline, add a seed or two, and tone down the 'data efficiency' framing since MLLM-generated captions carry their own cost. With those, this is a solid paper.","headline":"A strong empirical recipe for cross-modal self-distillation, but the main-table CLIP/SigLIP baselines may not have seen the same long captions, and the ablation shows that matters a lot.","tokens_in":32658,"tokens_out":4107,"would_cite":true,"duration_ms":39448,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cropped captions plus cross-modal self-distillation beat much larger vision-language pretraining runs.","keywords":["vision-language pretraining","contrastive learning","self-distillation","text cropping","cross-attention","zero-shot retrieval","semantic segmentation","compositional understanding"],"falsifier":"A direct test is to keep all of COSMOS's machinery but replace the local text crops with sentences sampled from other images' captions; if retrieval and segmentation gains persist, the local-to-global text correspondence is not the source of the improvement, and if they collapse, the assumption that captions decompose regionally is load-bearing.","tokens_in":31548,"feed_emoji":"✂️","tokens_out":8323,"duration_ms":81165,"temperature":0.7,"pith_summary":"Contrastive vision-language models align whole images with whole captions, and this global objective tends to make them lock onto the largest foreground object while ignoring small objects, background details, and word order. This paper argues that the fix is to pretrain with both global and local views of images and of text: a teacher network sees global views, a student network sees cropped views, and the student must reproduce the teacher's richer global representation. The paper's central claim is that when this self-distillation signal passes through a cross-attention module that pools image tokens with text tokens and vice versa, both the image and text encoders learn finer-grained, more composable representations. If true, the payoff is data efficiency: models trained on tens of millions of pairs can match or beat models trained on billions on retrieval, segmentation, and compositional understanding.","feed_headline":"Caption cropping beats billion-scale vision-language pretraining","feed_subtitle":"A 30M-pair model with cross-modality distillation outdoes billion-pair baselines on fine-grained tasks.","key_machinery":"The load-bearing mechanism is a student-teacher pretraining loop with three interacting pieces. First, multi-crop augmentation is applied to both modalities: two global image views plus six local image views, and text views formed by randomly sampling one to five sentences (global) or a single sentence (local) from long synthetic captions. Second, a cross-attention module makes each modality attend to the other, using the image's [cls] token as a query over text tokens and the text's [eot] token as a query over image tokens, then adding the attended result back to the query token. Third, a cross-modality self-distillation loss matches these student cross-modal embeddings to the teacher's global [cls] and [eot] tokens under a symmetric contrastive loss. Because the teacher is a moving average of the student and only sees global views, the student is trained to reconstruct rich global context from local inputs, and the gradient flows into both encoders at once.","core_discovery":"On its own terms, COSMOS establishes that randomly cropping long synthetic captions into multi-sentence global views and single-sentence local views, and doing the same for images, creates a workable self-distillation task for vision-language models. The student processes all crops and uses a cross-attention module to produce image and text embeddings that are each conditioned on the other modality; these are matched, through a symmetric contrastive loss, to the teacher's global image and text tokens. The teacher is an exponential moving average of the student, so the supervision is fully self-generated. The reported consequence is consistent gains across retrieval, classification, segmentation, and compositional benchmarks: for instance, on the COCO retrieval benchmark the merged-30M model reaches 68.0% image-to-text R@1, against 62.3% for the strongest previous long-caption method and 59.3% for a model trained on two billion pairs.","pith_inferences":["Editorial inference: the method's dependence on sentence-level long captions means its advantage over plain contrastive training should shrink as captions become shorter or less decomposable; an experiment training on single-sentence captions without any cropping would isolate this factor.","Editorial inference: because image and text crops are sampled independently, the objective sometimes asks local text about the background to align with a local image crop of the foreground; the method's robustness to this mismatch suggests it learns conceptual co-occurrence rather than strict region-level grounding, a property worth probing directly by measuring alignment when local crops are know","Editorial inference: the cross-attention module can be read as a learned pooling operation; replacing its query tokens with averaged tokens or adding a second layer would test whether the gains come from the attention pooling itself or from the extra capacity it adds to the student."],"forward_implications":["Retrieval: with 30 million pretraining pairs, the model reaches 68.0% image-to-text R@1 on COCO, exceeding a 2-billion-pair baseline at 59.3%.","Segmentation: the 30M model averages 20.0% mIoU across six zero-shot segmentation benchmarks, above the 16.5% of a 1-billion-pair baseline.","Compositional understanding: on SugarCrepe it scores 86.6% average, and on MMVP-VLM it matches a 1-billion-pair model at 25.9%.","The cross-attention module is used only during training; at inference the model is evaluated as a standard contrastive model using [cls] and [eot] tokens.","The two losses are added at the same scale, so the method needs no grid search over loss weights, a tuning burden that prior self-distillation methods carry."],"supporting_citations":[{"why":"Provides the multi-crop and student-teacher self-distillation formulation that COSMOS extends from images to text.","marker":"[7]"},{"why":"Supplies the long synthetic captions used for text cropping and serves as the strongest same-data baseline in most tables.","marker":"[91]"},{"why":"Establishes the contrastive alignment objective and the zero-shot evaluation protocol that COSMOS builds on and compares against.","marker":"[63]"},{"why":"Supplies the billion-scale pretrained models whose retrieval, classification, and segmentation numbers COSMOS is measured against.","marker":"[13]"},{"why":"Represents the earlier self-supervised-plus-contrastive approach that improves only the image encoder, the limitation COSMOS targets.","marker":"[56]"},{"why":"Provides the prior self-distillation baseline whose loss scaling problem COSMOS avoids, and a comparison point on segmentation.","marker":"[57]"},{"why":"Provides a strong contrastive baseline with a sigmoid objective that COSMOS outperforms under matched data.","marker":"[88]"},{"why":"Defines the zero-shot semantic segmentation evaluation protocol and the enhancement module applied to COSMOS for dense prediction.","marker":"[77]"},{"why":"Supplies the SugarCrepe compositional-understanding benchmark used to show the model does not treat captions as bag-of-words.","marker":"[33]"},{"why":"Supplies the MMVP-VLM visual-perception benchmark where COSMOS matches far larger models.","marker":"[74]"}],"fun_headline_variants":["Self-distillation with caption cropping tops billion-scale VLMs","30M pairs beat billion: cross-modal self-distillation wins","Cross-modal self-distillation: smaller data, finer results","Text cropping and cross-attention drive VLM self-distillation","Global-local multi-modal views: VLM outperforms bigger models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a randomly chosen sentence from a long caption and a randomly chosen crop of the paired image describe overlapping content, so that pulling them together in embedding space teaches genuine fine-grained alignment rather than gluing unrelated details together.","fun_headline_variants_meta":{"raw":{"variants":["Self-distillation with caption cropping tops billion-scale VLMs","30M pairs beat billion: cross-modal self-distillation wins","Cross-modal self-distillation: smaller data, finer results","Text cropping and cross-attention drive VLM self-distillation","Global-local multi-modal views: VLM outperforms bigger models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"total_tokens":1569,"prompt_tokens":932,"completion_tokens":637,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":553}},"tokens_in":548,"tokens_out":637,"duration_ms":6783,"temperature":1.0,"reasoning_tokens":553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:54:22.994756+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to keep all of COSMOS's machinery but replace the local text crops with sentences sampled from other images' captions; if retrieval and segmentation gains persist, the local-to-global text correspondence is not the source of the improvement, and if they collapse, the assumption that captions decompose regionally is load-bearing.","supporting_citations":[{"cited_title":"Dreamlip: Language- image pre-training with long captions","cited_arxiv_id":null,"evidence_quote":"Supplies the long synthetic captions used for text cropping and serves as the strongest same-data baseline in most tables."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Establishes the contrastive alignment objective and the zero-shot evaluation protocol that COSMOS builds on and compares against."},{"cited_title":"Slip: Self-supervision meets language-image pre- training","cited_arxiv_id":null,"evidence_quote":"Represents the earlier self-supervised-plus-contrastive approach that improves only the image encoder, the limitation COSMOS targets."},{"cited_title":"Silc: Improving vision language pretraining with self-distillation","cited_arxiv_id":null,"evidence_quote":"Provides the prior self-distillation baseline whose loss scaling problem COSMOS avoids, and a comparison point on segmentation."},{"cited_title":"Sigmoid loss for language image pre-training","cited_arxiv_id":null,"evidence_quote":"Provides a strong contrastive baseline with a sigmoid objective that COSMOS outperforms under matched data."},{"cited_title":"Sclip: Rethinking self-attention for dense vision-language inference","cited_arxiv_id":null,"evidence_quote":"Defines the zero-shot semantic segmentation evaluation protocol and the enhancement module applied to COSMOS for dense prediction."},{"cited_title":"Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality","cited_arxiv_id":null,"evidence_quote":"Supplies the SugarCrepe compositional-understanding benchmark used to show the model does not treat captions as bag-of-words."},{"cited_title":"Eyes wide shut? exploring the visual shortcomings of multimodal llms","cited_arxiv_id":null,"evidence_quote":"Supplies the MMVP-VLM visual-perception benchmark where COSMOS matches far larger models."}],"review_version":1}