{"id":"dc9d4447-bec1-40da-be75-15a13765f349","arxiv_id":"2602.16918","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A 2-billion-parameter vision encoder trained on 15B+ image-text and billions of video-hashtag pairs reports SOTA ImageNet linear-probe, Kinetics, and retrieval numbers, but relies on proprietary data and has several verification gaps.","lead":"Meta AI trained a unified vision model on billions of Facebook and Instagram images and videos using a three-stage pipeline of masked autoencoding, hashtag classification, and CLIP-style contrastive learning. The paper reports state-of-the-art scores on ImageNet, Kinetics, and retrieval benchmarks while using fewer tokens than competing models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM2CLIP text encoder is fine-tuned on MS-COCO captions (§4.6.2), yet COCO retrieval is reported as zero-shot (Tables 8/12, abstract); this directly inflates the MSCOCO leg of the SOTA claim.","rationale":"I read the paper's aim as scaling a multi-stage recipe and claiming efficiency/accuracy SOTA. I looked for the least secure condition: benchmark numbers must measure generalization, not leakage. The paper itself supplies the leak in §4.6.2: LLM2CLIP is SimCSE-fine-tuned on MS-COCO captions, then MSCOCO retrieval is called zero-shot. This is not an outside-consensus disagreement or a stylistic concern; it is an internal inconsistency between the method description and evaluation label. It directly affects the MSCOCO component of the headline claim, though not necessarily ImageNet/Kinetics. The reader's weakest assumption was broader (data curation quality/memorization); my concern is narrower and more concrete, hence partial agreement. A conditional verdict remains appropriate: the paper can satisfy the condition by retraining/relabeling and publishing true zero-shot COCO numbers and one consistent retrieval protocol.","tokens_in":28287,"tokens_out":5803,"duration_ms":47432,"concrete_test":"Retrain the stage-3 CLIP model with the same LLaMA-1b text encoder aligned only on non-COCO caption data (e.g., ShareCaptioner captions for non-COCO images, or MSR-VTT/CC3M), keeping the vision-side recipe unchanged, and re-evaluate MSCOCO image-to-text and text-to-image retrieval. Compare to Tables 8/12: a drop of more than 1–2 r@1 points indicates the reported 'zero-shot' COCO gains come from COCO fine-tuning. Also specify the exact retrieval protocol (candidate set size, K, caption set) for Tables 8 and 12 and reconcile the 48.21 vs 73.83 discrepancy; the SOTA comparison must be redone under one protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim bundles SOTA on ImageNet, Kinetics, and MSCOCO. The MSCOCO leg is compromised by an internal evaluation leak. §4.6.2 states that the LLM2CLIP text encoder is fine-tuned with 'supervised SimCSE on MS-COCO captions and re-annotated captions generated by ShareCaptioner.' Yet Table 8 labels MSCOCO retrieval as zero-shot, Table 12 repeats MSCOCO 'zero-shot' metrics, and the abstract claims SOTA cross-modal retrieval on MSCOCO. Because the text encoder has seen COCO captions during its alignment, the retrieval scores measure fit to the evaluation set, not zero-shot generalization. The comparison in Table 8 is also not a controlled test: only the LLM variant was fine-tuned on COCO. Separately, Tables 8 and 12 report inconsistent COCO r@1 numbers for the same XRay model (48.21 vs 73.83), so the retrieval protocol is under-specified. This does not by itself falsify the ImageNet or Kinetics results, but it is the most concrete, internally verifiable crack in the SOTA claim. No code/checkpoints are released, so external verification is impossible; at minimum, these results must be relabeled as 'COCO-fine-tuned' and true zero-shot numbers must be reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Xray-Visual, a unified image-video vision encoder trained on Meta's internal Facebook/Instagram data (claimed 15B image-text and 10B video-hashtag pairs, later stated as 10B and 5B respectively). Training proceeds in three stages: MAE self-supervision, semi-supervised hashtag classification, and CLIP-style contrastive learning with optional SLIP, denoising, and LLM (LLaMA-1b/LLM2CLIP) text encoders. Efficiency is achieved through EViT token pruning, giving 288 active tokens at 336px resolution. The paper claims SOTA on ImageNet (89.3% linear probe, image-only model), Kinetics (78.1%), HMDB51, and MS-COCO retrieval, plus strong OOD/robustness and internal retrieval metrics. Ablations support several design choices (resampling, filtering, batch sampling, EViT keep rate, denoising loss, registers).","tokens_in":28668,"tokens_out":5062,"duration_ms":44139,"significance":"If the results hold, the paper would demonstrate that a three-stage industry-scale recipe can produce a single image-video encoder that is both more accurate and more efficient than prior public models. The strengths are the very large scale, systematic ablations of data curation and training stages, and the explicit reporting of token counts and resolutions. However, the central SOTA claim is undermined by (i) an evaluation leak: the LLM2CLIP text encoder is fine-tuned on MS-COCO captions and the same MS-COCO benchmark is reported as 'zero-shot'; (ii) an internal contradiction on HMDB51, where the submitted model is below PE-G; and (iii) inconsistent data-scale numbers between the abstract and the body. These issues are fixable but currently overstate the contribution. No code, checkpoints, or data are released, limiting independent verification.","major_comments":[{"comment":"The MS-COCO leg of the SOTA claim is compromised by an evaluation leak. §4.6.2 states that the LLM2CLIP text encoder is fine-tuned with 'supervised SimCSE on MS-COCO captions and re-annotated captions generated by ShareCaptioner,' yet Table 8 labels the COCO retrieval numbers as 'zero-shot performance' and Table 12 reports 'MSCOCO Zero-Shot performance.' Because the text encoder has been trained on COCO captions, the COCO retrieval scores do not measure zero-shot generalization. The comparison in Table 8 is also uncontrolled: only the LLM-based variant was fine-tuned on COCO. Moreover, the two tables give inconsistent COCO r@1 values for the same XRay model family (Table 8: 48.21; Table 12, ViT-H: 73.83), so the retrieval protocol is underspecified. These results must be relabeled as COCO-fine-tuned (or true zero-shot scores reported), and the inconsistent numbers reconciled.","section":"§4.6.2, Tables 8/12, Abstract"},{"comment":"The abstract claims SOTA on 'HMDB51 for video understanding,' but Table 10 lists X-Ray Visual at 74.69% HMDB51 top-1, below PE-G at 77.50%. Similarly, Table 9 shows the unified X-Ray Visual model at 88.10% ImageNet, below PE-G's 89.22%; SOTA ImageNet is achieved only by the image-only X-Ray Image EViT-2b (89.30%). Thus the claims in the abstract and Section 5.1.1 are inaccurate unless explicitly restricted to the model variant that actually achieves SOTA. Please correct the overclaims and clearly separate image-only vs. unified model results.","section":"Abstract and Tables 9/10"},{"comment":"The data-scale claims are internally inconsistent. The abstract says 15B image-text pairs and 10B video-hashtag pairs, but §2.1 says the final ViSE dataset contains 'approximately 10 billion high-quality image-text pairs' and §2.2 says the video-hashtag collection is '∼5 billion' pairs (also reflected in URU-Video-Noun-5B). The paper should use one consistent set of numbers, with a clear explanation of what counts as a pair and what the final training corpus sizes are.","section":"Section 2.1/2.2 vs Abstract"},{"comment":"The efficiency claim is presented at a model level that is not clearly connected to the reported benchmarks. Fig 1 and the text claim '89.3% accuracy with 336px and 288 tokens vs baselines at 448px/1024 tokens,' but Tables 9/10 show the unified X-Ray Visual model (the one evaluated on Kinetics) achieving 88.1% ImageNet. It is unclear whether the unified video model also runs at 336px/288 tokens and whether the Kinetics 78.1% number was obtained under those settings. Please state explicitly, for each reported benchmark, the model variant, resolution, and active token count used. Without this, the '4× efficiency while maintaining SOTA' claim is not verifiable.","section":"§5.1.1 and Fig 1"}],"minor_comments":[{"comment":"The text says 'we still didn't see improved performance with scaling text encoder,' but Table 7 shows zero-shot accuracy increasing from 71.24 to 72.13 across the configurations. Please rephrase to 'limited improvement' or explain the criterion.","section":"§4.6.1 / Table 7"},{"comment":"The sentence about '+1.05% on MS-COCO Top-1 and +0.25% on ImageNet Top-1' does not specify the comparison baseline. Clarify which row in Table 12 is the reference.","section":"§5.1.2"},{"comment":"There are several labeling inconsistencies: Table 8 is in Section 4.6.2 but referenced as a zero-shot table; Table 12 appears under Section 5.2 yet is referenced in §5.1.2; Table 11 appears with a different column header format. Please renumber and cross-check all table references.","section":"Tables throughout"},{"comment":"Typos and name inconsistencies: 'avaliable', 'embdedding', 'near deuplicate', 'SiGLIP' vs 'SigLIP', 'XRay' vs 'X-Ray' vs 'XRV'. Also 'SiGLIP-L' and 'SiGLIP-L/16' appear in Table 14; unify notation.","section":"General"},{"comment":"The text states XRay 'achieves 78.1% Top-1 on Kinetics' but Table 10 lists 'X-Ray Visual EViT-2b' 78.10; confirm whether this is Kinetics-400 or Kinetics-700. The caption is ambiguous, and earlier ablation Table 6 uses K700.","section":"§5.1.1 and Table 10"},{"comment":"Internal metrics are defined well, but the comparison to 'state-of-the-art vision encoders' in Table 16/17 would benefit from noting which exact public checkpoints were used (e.g., PE-G, DINOv2 with text) and whether they are used in the same feature-extraction protocol.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper contains several promising ideas and a rare large-scale study, but the current version overstates its results. The MS-COCO evaluation leak is a serious correctness issue that necessarily changes the abstract/title-level claims. The HMDB51 contradiction and data-scale inconsistencies are also not merely cosmetic. I would encourage the authors to revise thoroughly, release at least some checkpoints or detailed evaluation code, and be explicit about which model variant is being reported for each benchmark. If these issues are addressed, the paper could become a valuable industrial-scale empirical study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news here is scale, not novelty: a three-stage MAE-to-hashtag-to-CLIP recipe at 26B images and videos, with an LLM text encoder, reaches 89.3% ImageNet linear probe and 78.1% Kinetics with 288 tokens at 336px. For anyone building production vision encoders, that is a meaningful data point, and the ablations on semantic resampling, filtering thresholds, batch sampling, and denoising loss give the recipe more empirical support than most papers at this scale. The authors also deserve credit for reporting where academic benchmarks and their internal metrics diverge; that is honest and useful.\n\nThe soft spots are real. The MSCOCO zero-shot retrieval results are compromised. Section 4.6.2 says the LLaMA text encoder is fine-tuned with supervised SimCSE on MS-COCO captions, yet Tables 8 and 12 label COCO retrieval as zero-shot. That is an internal evaluation leak on one leg of the SOTA claim. Worse, the same Xray model gets different COCO img-text r@1 numbers across tables: 48.21 in Table 8 and 73.83 in Table 12. That inconsistency makes the retrieval protocol under-specified. The leak does not falsify the ImageNet or Kinetics results; those are external benchmarks, and the paper does not claim to have trained on them. But it does mean the MSCOCO headline should be relabeled or removed.\n\nBeyond that, the paper has the usual industry-report limitations: no code or checkpoints, no error bars, and some dataset scale inconsistencies (5B vs 10B vs 15B in different places). The data curation pipeline is described in detail but is proprietary, so external verification is impossible. The hypothesis that social-media data at this scale trains more robust real-world embeddings is plausible and is supported by the internal retrieval numbers, but those internal numbers are not independently checkable.\n\nBottom line: the central ImageNet/Kinetics argument holds up as far as can be told from the paper, and the efficiency claim at quarter tokens is interesting, but the retrieval claim needs fixing before the SOTA framing is credible. This paper deserves a serious referee, not a desk reject, because if the scale recipe is real it matters to the field. I would send it out with a request for major revision: report true zero-shot COCO numbers, resolve the table inconsistency, and ideally release at least a small model or some checkpoints.","headline":"A credible industrial-scale vision recipe with a genuine evaluation leak on the MSCOCO leg of the SOTA claim; the ImageNet/Kinetics numbers deserve referee time, but the zero-shot retrieval tables cannot be trusted as reported.","tokens_in":29318,"tokens_out":2200,"would_cite":false,"duration_ms":22536,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-stage training recipe on billions of social media images and videos produces a single vision encoder that beats previous benchmarks while using a quarter of the tokens.","keywords":["vision transformer","self-supervised learning","CLIP","masked autoencoding","video understanding","image-text retrieval","data curation","token efficiency"],"falsifier":"Train the same architecture on a public large-scale corpus (e.g., DataComp or LAION) using the identical three-stage recipe and compare ImageNet linear-probe accuracy: if the gap to 89.3% is large, the private data is the source of the claim; if the gap is small, the architecture and recipe are the source. Also, measure the overlap between the 75k canonical hashtags and the Kinetics/ImageNet class lists; if hashtags are a superset of the benchmark classes, the reported video accuracy could be inflated by label leakage.","tokens_in":28136,"feed_emoji":"🖼️","tokens_out":2718,"duration_ms":27741,"temperature":0.7,"pith_summary":"This paper claims that a unified image-video vision encoder trained on an unprecedented scale of private social-media data can surpass existing state-of-the-art models on ImageNet, Kinetics, and MSCOCO while consuming only a quarter of the visual tokens at lower resolution. The recipe combines three stages: masked autoencoding (MAE), semi-supervised hashtag classification, and CLIP-style contrastive learning with a large language model as text encoder. The authors argue that careful data curation (WordNet-based balancing, similarity filtering, hashtag canonicalization, and LLM-refined synthetic captions) is the main driver of both accuracy and out-of-distribution robustness. If true, this suggests that industry-scale data and a staged training curriculum can substitute for architectural complexity, and that joint image-video training can yield a single model that is competitive for both modalities.","feed_headline":"A three-stage recipe tops vision benchmarks with a quarter of the tokens","feed_subtitle":"Trained on 25 billion social-media pairs, the unified image-video encoder reports SOTA on ImageNet, Kinetics, and COCO at 336px.","key_machinery":"The central object is a three-stage training pipeline: (1) MAE self-supervised reconstruction on 10B images and videos; (2) semi-supervised classification over ~22k-31k canonicalized hashtags; (3) CLIP-style contrastive learning with image and video captions, with an optional SLIP self-supervised loss and a denoising loss. The architecture is a ViT with 3D tokenization, register tokens, and EViT token pruning (dropping inattentive tokens) to achieve high resolution at low compute. The text encoder is LLaMA-1b fine-tuned via LoRA and bidirectional attention (LLM2Vec style) rather than a standard CLIP text tower.","core_discovery":"Xray-Visual is a ViT-based architecture that processes images and videos jointly using 3D tokenization and EViT token pruning, trained on over 15 billion image-text and 10 billion video-hashtag pairs from Facebook and Instagram. The authors report 89.3% ImageNet linear-probe accuracy and 78.1% Kinetics top-1 accuracy with a 2B-parameter model at 336 resolution using only 288 tokens, a 4x efficiency gain over methods like Perception Encoder and SigLIP that use 448 resolution and 1024 tokens. They also report state-of-the-art zero-shot retrieval on MSCOCO and MSR-VTT, and observe that using a decoder-only language model (LLaMA-1b) as the text encoder yields consistent gains on internal, produc","pith_inferences":["The paper implies that benchmark leadership is partly an artifact of data scale and curation rather than architecture novelty; if that is true, public datasets will need to become much larger before other labs can replicate these numbers.","The strongest evidence for the LLM-as-text-encoder claim comes from internal metrics, which are not reproducible externally; the academic-benchmark improvements are smaller, so the claim's generality is hard to verify without releasing the model or data.","One can test the load-bearing role of the curation pipeline directly by training the same architecture on a public corpus (e.g., a filtered subset of LAION) and comparing the gap to the reported 89.3%; a large gap would confirm the data, a small gap would suggest the architecture is the main driver.","The observed 0.3% ImageNet gain from the denoising loss and the 7.3% gain from WordNet rebalancing suggest that future work may find even larger gains from better long-tail handling rather than from scaling model size alone."],"forward_implications":["If the reported numbers are reproduced, the recipe shows that 336px input with 288 tokens can outperform 448px/1024-token baselines, a direct computational advantage for deployment.","A single joint image-video encoder can serve both modalities at near-state-of-the-art accuracy, eliminating the need for separate image and video models.","LLM-based text encoders, when properly aligned, improve cross-modal retrieval and real-world robustness even when they do not lift academic benchmark scores.","The systematic data-curation pipeline (balancing, filtering, canonicalization, synthetic caption refinement) constitutes a transferable methodology for any large-scale web data, not just social media.","Token pruning at inference produces negligible accuracy loss while increasing video inference speed by 58%, making high-resolution video understanding more practical."],"fun_headline_variants":["4x fewer tokens, same SOTA: Xray-Visual scales vision on 25B pairs","Image and video in one model: 288 tokens beat 1024 at 336px","Three-stage training + token pruning = vision benchmark leader","25B social pairs yield SOTA on ImageNet, Kinetics, and COCO","EViT cuts tokens 4x; model still dominates vision benchmarks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire edifice rests on the assumption that the private data-curation pipeline—WordNet rebalancing, MetaCLIP-style filtering at threshold 0.25, hashtag canonicalization, and LLM-refined synthetic captions—produces supervision that is clean and diverse enough that the downstream benchmark numbers reflect genuine visual understanding rather than memorization, leakage, or artifacts of the filter itself.","fun_headline_variants_meta":{"raw":{"variants":["4x fewer tokens, same SOTA: Xray-Visual scales vision on 25B pairs","Image and video in one model: 288 tokens beat 1024 at 336px","Three-stage training + token pruning = vision benchmark leader","25B social pairs yield SOTA on ImageNet, Kinetics, and COCO","EViT cuts tokens 4x; model still dominates vision benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000458,"raw_usage":{"total_tokens":2156,"prompt_tokens":793,"completion_tokens":1363,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":1257}},"tokens_in":537,"tokens_out":1363,"duration_ms":11267,"temperature":1.0,"reasoning_tokens":1257,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T22:22:47.095604+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same architecture on a public large-scale corpus (e.g., DataComp or LAION) using the identical three-stage recipe and compare ImageNet linear-probe accuracy: if the gap to 89.3% is large, the private data is the source of the claim; if the gap is small, the architecture and recipe are the source. Also, measure the overlap between the 75k canonical hashtags and the Kinetics/ImageNet class lists; if hashtags are a superset of the benchmark classes, the reported video accuracy could be inflated by label leakage.","supporting_citations":[],"review_version":1}