Pith. sign in

REVIEW 4 major objections 6 minor 33 references

A three-stage training recipe on billions of social media images and videos produces a single vision encoder that beats previous benchmarks while using a quarter of the tokens.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 22:22 UTC pith:B6L76PTH

load-bearing objection A credible industrial-scale vision recipe with a genuine evaluation leak on the MSCOCO leg of the SOTA claim; the ImageNet/Kinetics numbers deserve referee time, but the zero-shot retrieval tables cannot be trusted as reported. the 4 major comments →

arxiv 2602.16918 v2 pith:B6L76PTH submitted 2026-02-18 cs.CV cs.AI

Xray-Visual Models: Scaling Vision models on Industry Scale Data

Shlok Mishra , Tsung-Yu Lin , Linda Wang , Hongli Xu , Yimin Liu , Michael Hsu , Chaitanya Ahuja , Hao Yuan
show 19 more authors
Jianpeng Cheng Hong-You Chen Haoyuan Xu Chao Li Sreya Dutta Roy Abhijeet Awasthi Jihye Moon Don Husa Michael Ge Sumedha Singla Arkabandhu Chowdhury Phong Dingh Satya Narayan Shukla Yonghuan Yang David Jacobs Qi Guo Jun Xiao Xiangjun Fan Aashu Singh
This is my paper
classification cs.CV cs.AI
keywords vision transformerself-supervised learningCLIPmasked autoencodingvideo understandingimage-text retrievaldata curationtoken efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that a unified image-video vision encoder trained on an unprecedented scale of private social-media data can surpass existing state-of-the-art models on ImageNet, Kinetics, and MSCOCO while consuming only a quarter of the visual tokens at lower resolution. The recipe combines three stages: masked autoencoding (MAE), semi-supervised hashtag classification, and CLIP-style contrastive learning with a large language model as text encoder. The authors argue that careful data curation (WordNet-based balancing, similarity filtering, hashtag canonicalization, and LLM-refined synthetic captions) is the main driver of both accuracy and out-of-distribution robustness. If true, this suggests that industry-scale data and a staged training curriculum can substitute for architectural complexity, and that joint image-video training can yield a single model that is competitive for both modalities.

Core claim

Xray-Visual is a ViT-based architecture that processes images and videos jointly using 3D tokenization and EViT token pruning, trained on over 15 billion image-text and 10 billion video-hashtag pairs from Facebook and Instagram. The authors report 89.3% ImageNet linear-probe accuracy and 78.1% Kinetics top-1 accuracy with a 2B-parameter model at 336 resolution using only 288 tokens, a 4x efficiency gain over methods like Perception Encoder and SigLIP that use 448 resolution and 1024 tokens. They also report state-of-the-art zero-shot retrieval on MSCOCO and MSR-VTT, and observe that using a decoder-only language model (LLaMA-1b) as the text encoder yields consistent gains on internal, produc

What carries the argument

The central object is a three-stage training pipeline: (1) MAE self-supervised reconstruction on 10B images and videos; (2) semi-supervised classification over ~22k-31k canonicalized hashtags; (3) CLIP-style contrastive learning with image and video captions, with an optional SLIP self-supervised loss and a denoising loss. The architecture is a ViT with 3D tokenization, register tokens, and EViT token pruning (dropping inattentive tokens) to achieve high resolution at low compute. The text encoder is LLaMA-1b fine-tuned via LoRA and bidirectional attention (LLM2Vec style) rather than a standard CLIP text tower.

Load-bearing premise

The entire edifice rests on the assumption that the private data-curation pipeline—WordNet rebalancing, MetaCLIP-style filtering at threshold 0.25, hashtag canonicalization, and LLM-refined synthetic captions—produces supervision that is clean and diverse enough that the downstream benchmark numbers reflect genuine visual understanding rather than memorization, leakage, or artifacts of the filter itself.

What would settle it

Train the same architecture on a public large-scale corpus (e.g., DataComp or LAION) using the identical three-stage recipe and compare ImageNet linear-probe accuracy: if the gap to 89.3% is large, the private data is the source of the claim; if the gap is small, the architecture and recipe are the source. Also, measure the overlap between the 75k canonical hashtags and the Kinetics/ImageNet class lists; if hashtags are a superset of the benchmark classes, the reported video accuracy could be inflated by label leakage.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the reported numbers are reproduced, the recipe shows that 336px input with 288 tokens can outperform 448px/1024-token baselines, a direct computational advantage for deployment.
  • A single joint image-video encoder can serve both modalities at near-state-of-the-art accuracy, eliminating the need for separate image and video models.
  • LLM-based text encoders, when properly aligned, improve cross-modal retrieval and real-world robustness even when they do not lift academic benchmark scores.
  • The systematic data-curation pipeline (balancing, filtering, canonicalization, synthetic caption refinement) constitutes a transferable methodology for any large-scale web data, not just social media.
  • Token pruning at inference produces negligible accuracy loss while increasing video inference speed by 58%, making high-resolution video understanding more practical.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper implies that benchmark leadership is partly an artifact of data scale and curation rather than architecture novelty; if that is true, public datasets will need to become much larger before other labs can replicate these numbers.
  • The strongest evidence for the LLM-as-text-encoder claim comes from internal metrics, which are not reproducible externally; the academic-benchmark improvements are smaller, so the claim's generality is hard to verify without releasing the model or data.
  • One can test the load-bearing role of the curation pipeline directly by training the same architecture on a public corpus (e.g., a filtered subset of LAION) and comparing the gap to the reported 89.3%; a large gap would confirm the data, a small gap would suggest the architecture is the main driver.
  • The observed 0.3% ImageNet gain from the denoising loss and the 7.3% gain from WordNet rebalancing suggest that future work may find even larger gains from better long-tail handling rather than from scaling model size alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Xray-Visual, a unified image-video vision encoder trained on Meta's internal Facebook/Instagram data (claimed 15B image-text and 10B video-hashtag pairs, later stated as 10B and 5B respectively). Training proceeds in three stages: MAE self-supervision, semi-supervised hashtag classification, and CLIP-style contrastive learning with optional SLIP, denoising, and LLM (LLaMA-1b/LLM2CLIP) text encoders. Efficiency is achieved through EViT token pruning, giving 288 active tokens at 336px resolution. The paper claims SOTA on ImageNet (89.3% linear probe, image-only model), Kinetics (78.1%), HMDB51, and MS-COCO retrieval, plus strong OOD/robustness and internal retrieval metrics. Ablations support several design choices (resampling, filtering, batch sampling, EViT keep rate, denoising loss, registers).

Significance. If the results hold, the paper would demonstrate that a three-stage industry-scale recipe can produce a single image-video encoder that is both more accurate and more efficient than prior public models. The strengths are the very large scale, systematic ablations of data curation and training stages, and the explicit reporting of token counts and resolutions. However, the central SOTA claim is undermined by (i) an evaluation leak: the LLM2CLIP text encoder is fine-tuned on MS-COCO captions and the same MS-COCO benchmark is reported as 'zero-shot'; (ii) an internal contradiction on HMDB51, where the submitted model is below PE-G; and (iii) inconsistent data-scale numbers between the abstract and the body. These issues are fixable but currently overstate the contribution. No code, checkpoints, or data are released, limiting independent verification.

major comments (4)
  1. [§4.6.2, Tables 8/12, Abstract] The MS-COCO leg of the SOTA claim is compromised by an evaluation leak. §4.6.2 states that the LLM2CLIP text encoder is fine-tuned with 'supervised SimCSE on MS-COCO captions and re-annotated captions generated by ShareCaptioner,' yet Table 8 labels the COCO retrieval numbers as 'zero-shot performance' and Table 12 reports 'MSCOCO Zero-Shot performance.' Because the text encoder has been trained on COCO captions, the COCO retrieval scores do not measure zero-shot generalization. The comparison in Table 8 is also uncontrolled: only the LLM-based variant was fine-tuned on COCO. Moreover, the two tables give inconsistent COCO r@1 values for the same XRay model family (Table 8: 48.21; Table 12, ViT-H: 73.83), so the retrieval protocol is underspecified. These results must be relabeled as COCO-fine-tuned (or true zero-shot scores reported), and the inconsistent numbers reconciled.
  2. [Abstract and Tables 9/10] The abstract claims SOTA on 'HMDB51 for video understanding,' but Table 10 lists X-Ray Visual at 74.69% HMDB51 top-1, below PE-G at 77.50%. Similarly, Table 9 shows the unified X-Ray Visual model at 88.10% ImageNet, below PE-G's 89.22%; SOTA ImageNet is achieved only by the image-only X-Ray Image EViT-2b (89.30%). Thus the claims in the abstract and Section 5.1.1 are inaccurate unless explicitly restricted to the model variant that actually achieves SOTA. Please correct the overclaims and clearly separate image-only vs. unified model results.
  3. [Section 2.1/2.2 vs Abstract] The data-scale claims are internally inconsistent. The abstract says 15B image-text pairs and 10B video-hashtag pairs, but §2.1 says the final ViSE dataset contains 'approximately 10 billion high-quality image-text pairs' and §2.2 says the video-hashtag collection is '∼5 billion' pairs (also reflected in URU-Video-Noun-5B). The paper should use one consistent set of numbers, with a clear explanation of what counts as a pair and what the final training corpus sizes are.
  4. [§5.1.1 and Fig 1] The efficiency claim is presented at a model level that is not clearly connected to the reported benchmarks. Fig 1 and the text claim '89.3% accuracy with 336px and 288 tokens vs baselines at 448px/1024 tokens,' but Tables 9/10 show the unified X-Ray Visual model (the one evaluated on Kinetics) achieving 88.1% ImageNet. It is unclear whether the unified video model also runs at 336px/288 tokens and whether the Kinetics 78.1% number was obtained under those settings. Please state explicitly, for each reported benchmark, the model variant, resolution, and active token count used. Without this, the '4× efficiency while maintaining SOTA' claim is not verifiable.
minor comments (6)
  1. [§4.6.1 / Table 7] The text says 'we still didn't see improved performance with scaling text encoder,' but Table 7 shows zero-shot accuracy increasing from 71.24 to 72.13 across the configurations. Please rephrase to 'limited improvement' or explain the criterion.
  2. [§5.1.2] The sentence about '+1.05% on MS-COCO Top-1 and +0.25% on ImageNet Top-1' does not specify the comparison baseline. Clarify which row in Table 12 is the reference.
  3. [Tables throughout] There are several labeling inconsistencies: Table 8 is in Section 4.6.2 but referenced as a zero-shot table; Table 12 appears under Section 5.2 yet is referenced in §5.1.2; Table 11 appears with a different column header format. Please renumber and cross-check all table references.
  4. [General] Typos and name inconsistencies: 'avaliable', 'embdedding', 'near deuplicate', 'SiGLIP' vs 'SigLIP', 'XRay' vs 'X-Ray' vs 'XRV'. Also 'SiGLIP-L' and 'SiGLIP-L/16' appear in Table 14; unify notation.
  5. [§5.1.1 and Table 10] The text states XRay 'achieves 78.1% Top-1 on Kinetics' but Table 10 lists 'X-Ray Visual EViT-2b' 78.10; confirm whether this is Kinetics-400 or Kinetics-700. The caption is ambiguous, and earlier ablation Table 6 uses K700.
  6. [§5.3] Internal metrics are defined well, but the comparison to 'state-of-the-art vision encoders' in Table 16/17 would benefit from noting which exact public checkpoints were used (e.g., PE-G, DINOv2 with text) and whether they are used in the same feature-extraction protocol.

Circularity Check

1 steps flagged

COCO retrieval is reported as zero-shot after the LLM text encoder was fine-tuned on MS-COCO captions, compromising the MSCOCO leg of the SOTA claim; ImageNet/Kinetics remain external.

specific steps
  1. fitted input called prediction [Section 4.6.2, Tables 8 and 12, Abstract]
    "Subsequently, we perform caption contrastive fine-tuning using supervised SimCSE Gao et al. (2021) on MS-COCO Chen et al. (2015) captions and re-annotated captions generated by ShareCaptioner. ... Table 8: Using LLM as text encoder zero-shot performance. ... Table 12: Model and MSCOCO/ImageNet Zero-Shot performance metrics."

    The LLM2CLIP text encoder is fine-tuned on MS-COCO captions (§4.6.2), and the same dataset is then used to report MSCOCO image-text and text-image retrieval as 'zero-shot' in Tables 8 and 12, with the abstract claiming SOTA cross-modal retrieval on MSCOCO. The retrieval therefore is not zero-shot: the text encoder has been optimized on the COCO caption distribution, so the scores reflect fit to the evaluation set rather than generalization. The Table 8 comparison is also uncontrolled because only the LLM variant was fine-tuned on COCO. In addition, Table 8 lists XRay COCO Img-Text r@1 as 48.21 while Table 12 lists 73.83 for the same XRay Visual model, indicating the retrieval protocol is under-specified. This does not invalidate the ImageNet or Kinetics results, but it makes the MSCOCO leg

full rationale

The ImageNet and Kinetics claims are measured against external benchmarks and do not reduce to fitted parameters on those eval sets, so the core image/video derivation is not circular. The pipeline itself is a standard empirical recipe (MAE → hashtag classification → CLIP), and no load-bearing uniqueness theorem or ansatz is imported via self-citation. The one concrete circular step is the MSCOCO evaluation: §4.6.2 fine-tunes the LLM2CLIP text encoder with supervised SimCSE on MS-COCO captions, and Tables 8/12 plus the abstract then report MSCOCO retrieval as zero-shot SOTA. This is a fitted-input-called-prediction leak specific to the retrieval leg. The inconsistent COCO r@1 values across Tables 8 and 12 (48.21 vs 73.83 for the same XRay model) reinforce that the protocol is under-specified. Self-citations such as Mishra et al. (2022) are incidental and not load-bearing. Overall score 6: partial circularity, with the central image/video classification results retaining independent empirical content.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 3 invented entities

This is an empirical systems paper, so the ledger is dominated by fitted hyperparameters and unverifiable proprietary resources. The most consequential free parameters are the data-filtering threshold, EViT operating point, and loss weights, all tuned on the very benchmarks used to claim SOTA. The axioms are domain assumptions about social-media supervision and the staged curriculum. No new theoretical entities are introduced, but the ViSE/URU datasets and internal MMLLM act as invented resources that cannot be independently checked.

free parameters (7)
  • MetaCLIP similarity filtering threshold = 0.25
    Chosen in Table 2 to maximize ImageNet linear-probe accuracy; central to ViSE curation.
  • EViT keep rate / resolution = 0.7 @288 (ViT-B), 0.5 @336 (large models)
    Selected via throughput-accuracy ablations in Table 4; the final SOTA numbers are reported at this operating point.
  • Denoising loss weight lambda_den = 1
    Explicitly tuned empirically in Section 4.2.5; gives 0.3% ImageNet gain.
  • Siamese loss weighting (synthetic : hashtag) = 1:1
    Tuned across 'different loss weighting schemes' in Section 4.7.1.
  • MAE masking ratios = 75% images, 90% videos
    Free choices borrowed from prior work, but they shape the first-stage representations.
  • Label smoothing in contrastive loss = not reported
    Mentioned as beneficial in Table 2, but the actual smoothing value is not disclosed.
  • Batch modality sampling probabilities = P_I:P_V = N_I/B_I : N_V/B_V
    The formula is natural, but the decision to use probabilistic sampling over strict alternation was tuned (Table 5).
axioms (6)
  • domain assumption Raw social-media captions and hashtags carry semantic signal aligned with visual content
    The entire data pipeline assumes that noisy user captions/hashtags can serve as supervision (Section 2.1-2.2).
  • domain assumption WordNet synset-based resampling improves long-tail coverage without introducing harmful bias
    Used in Section 2.1 to justify semantic balancing; only evaluated via ImageNet linear probe, not isolated.
  • domain assumption MetaCLIP similarity scores are a valid measure of image-text alignment for filtering
    The noise-suppression filter in Section 2.1 uses a pre-trained MetaCLIP model; its scores are assumed to be trustworthy proxies.
  • domain assumption MAE-to-hashtag-to-CLIP staged training is an effective curriculum
    The whole Section 4.2 rests on this; no comparison against a monolithic single-stage training is provided.
  • ad hoc to paper LLM2CLIP text encoder fine-tuned on MS-COCO does not compromise zero-shot COCO retrieval evaluation
    Section 4.6.2 fine-tunes the LLM on MS-COCO captions; Table 8 then reports MS-COCO retrieval as zero-shot. This assumption is not stated and is likely false.
  • domain assumption Internal MMLLM-generated synthetic captions are diverse and faithful enough to train from
    The synthetic-caption pipeline in Section 2.3 depends on an undisclosed internal model, and the paper's handling of caption repetition is heuristic.
invented entities (3)
  • ViSE image dataset no independent evidence
    purpose: Approximately 10-15B image-text pairs from Facebook and Instagram used for the CLIP stage.
    Not released; counts appear inconsistent (over 15B in the abstract, about 10B in Section 2.1). No external verification.
  • URU video dataset no independent evidence
    purpose: Billions of video-hashtag pairs from Instagram, split into noun and action variants.
    Not released; abstract claims 10B video-hashtag pairs while Section 2.2 says about 5B. No external verification.
  • Internal MMLLM no independent evidence
    purpose: Generates synthetic captions for images and videos.
    No architecture, training data, or release is described; results depend on it for caption quality.

pith-pipeline@v1.3.0-alltime-deepseek · 27967 in / 15604 out tokens · 139553 ms · 2026-08-02T22:22:47.095604+00:00 · methodology

0 comments
read the original abstract

We present Xray-Visual, a unified vision model architecture for large-scale image and video understanding trained on industry-scale social media data. Our model leverages over 15 billion curated image-text pairs and 10 billion video-hashtag pairs from Facebook and Instagram, employing robust data curation pipelines that incorporate balancing and noise suppression strategies to maximize semantic diversity while minimizing label noise. We introduce a three-stage training pipeline that combines self-supervised MAE, semi-supervised hashtag classification, and CLIP-style contrastive learning to jointly optimize image and video modalities. Our architecture builds on a Vision Transformer backbone enhanced with efficient token reorganization (EViT) for improved computational efficiency. Extensive experiments demonstrate that Xray-Visual achieves state-of-the-art performance across diverse benchmarks, including ImageNet for image classification, Kinetics and HMDB51 for video understanding, and MSCOCO for cross-modal retrieval. The model exhibits strong robustness to domain shift and adversarial perturbations. We further demonstrate that integrating large language models as text encoders (LLM2CLIP) significantly enhances retrieval performance and generalization capabilities, particularly in real-world environments. Xray-Visual establishes new benchmarks for scalable, multimodal vision models, while maintaining superior accuracy and computational efficiency.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 28 linked inside Pith

  1. [7]

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. Symbolic discovery of optimization algorithms.ArXiv, abs/2302.06675, 2023.https://api.semanticscholar.org/CorpusID:256846990. Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, ...

    Pith/arXiv arXiv 2023

  2. [8]

    Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325,

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325,

    Pith/arXiv arXiv

  3. [9]

    Deconstructing denoising diffusion models for self-supervised learning.arXiv preprint arXiv:2401.14404,

    Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning.arXiv preprint arXiv:2401.14404,

    Pith/arXiv arXiv

  4. [10]

    Metaclip 2: A worldwide scaling recipe.arXiv preprint arXiv:2507.22062,

    23 Yung-Sung Chuang, Yang Li, Dong Wang, Ching-Feng Yeh, Kehan Lyu, Ramya Raghavendra, James Glass, Lifei Huang, Jason Weston, Luke Zettlemoyer, Xinlei Chen, Zhuang Liu, Saining Xie, Wen tau Yih, Shang-Wen Li, and Hu Xu. Metaclip 2: A worldwide scaling recipe.arXiv preprint arXiv:2507.22062,

    Pith/arXiv arXiv

  5. [11]

    Vision transformers need registers.ArXiv, abs/2309.16588, 2023.https://api.semanticscholar.org/CorpusID:263134283

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers.ArXiv, abs/2309.16588, 2023.https://api.semanticscholar.org/CorpusID:263134283. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InCVPR,

    Pith/arXiv arXiv 2023

  6. [12]

    Improving clip training with language rewrites.ArXiv, abs/2305.20088, 2023a.https://api.semanticscholar.org/CorpusID:258987272

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites.ArXiv, abs/2305.20088, 2023a.https://api.semanticscholar.org/CorpusID:258987272. Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites, 2023b.https://arxiv.org/abs/2305....

    Pith/arXiv arXiv 2022

  7. [13]

    Simcse: Simple contrastive learning of sentence embeddings.ArXiv, abs/2104.08821, 2021.https://api.semanticscholar.org/CorpusID:233296292

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings.ArXiv, abs/2104.08821, 2021.https://api.semanticscholar.org/CorpusID:233296292. Songwei Ge, Shlok Kumar Mishra, Haohan Wang, Chun-Liang Li, and David Jacobs. Robust contrastive learning using negative samples with diminished semantics. InNEURIPS, volume a...

    Pith/arXiv arXiv 2021

  8. [14]

    Deepti Ghadiyaram, Matt Feiszli, Du Tran, Xueting Yan, Heng Wang, and Dhruv Kumar Mahajan. Large-scale weakly-supervised pre-training for video action recognition.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12038–12047, 2019.https://api.semanticscholar.org/CorpusID:143423501. Aaron Grattafiori, Abhimanyu Dubey, Abhina...

    Pith/arXiv arXiv 2019

  9. [16]

    Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low- rank adaptation of large language models.ArXiv, abs/2106.09685, 2021.https://api.semanticscholar.org/CorpusID: 235458009. Weiquan Huang, Aoqi Wu, Yifan Yang, Xufang Luo, Yuqing Yang, Liang Hu, Qi Dai, Xiyang Dai, Dongdong Chen, Chong Luo, and L...

    Pith/arXiv arXiv 2021

  10. [17]

    The kinetics human action video dataset, 2017.https://arxiv.org/abs/1705.06950

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017.https://arxiv.org/abs/1705.06950. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson,...

    Pith/arXiv arXiv 2017

  11. [19]

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking.2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23390–23400, 2022.https://api.semanticscholar.org/CorpusID:254125280. Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. E...

    2023

  12. [20]

    Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten

    https: //api.semanticscholar.org/CorpusID:251647803. Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining, 2018.https://arxiv.org/abs/ 1805.00932. Shlok Kumar Mishra, Joshua Robinson, Huiwen Chang, David Jacobs, Aaron Sarn...

    Pith/arXiv arXiv 2018

  13. [21]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Q. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russ Howes, Po-Yao (Bernie) Huang, Shang-Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Huijiao Xu, Hervé Jégou, Julien Ma...

    Pith/arXiv arXiv

  14. [22]

    Laion-5b: An open large-scale dataset for training next generation image-text models.ArXiv, abs/2210.08402,

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models....

    Pith/arXiv arXiv

  15. [23]

    https://api.semanticscholar.org/CorpusID: 252917726. Noam M. Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost.ArXiv, abs/1804.04235, 2018.https://api.semanticscholar.org/CorpusID:4786918. Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, S...

    Pith/arXiv arXiv 2018

  16. [24]

    Mannat Singh, Laura Gustafson, Aaron B

    https://arxiv.org/abs/2508.10104. Mannat Singh, Laura Gustafson, Aaron B. Adcock, Vinicius de Freitas Reis, Buğra Gedik, Raj Prateek Kosaraju, Dhruv Kumar Mahajan, Ross B. Girshick, Piotr Doll’ar, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models.2022 IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    Pith/arXiv arXiv 2022

  17. [25]

    Roformer: Enhanced transformer with rotary position embedding.ArXiv, abs/2104.09864, 2021.https://api.semanticscholar.org/CorpusID:233307138

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.ArXiv, abs/2104.09864, 2021.https://api.semanticscholar.org/CorpusID:233307138. Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-clip: A clip model focusing on wherever you wa...

    Pith/arXiv arXiv 2021

  18. [26]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al

    https://arxiv.org/abs/2409.02727. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295,

    Pith/arXiv arXiv

  19. [27]

    Llama: Open and efficient foundation language models.ArXiv, abs/2302.13971, 2023a

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur’elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models.ArXiv, abs/2302.13971, 2023a. https://api.semanticscholar.org/CorpusID:...

    Pith/arXiv arXiv 2015

  20. [28]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786,

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786,

    Pith/arXiv arXiv

  21. [29]

    Representation learning with contrastive predictive coding.preprint arXiv:1807.03748,

    Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding.preprint arXiv:1807.03748,

    Pith/arXiv arXiv

  22. [31]

    Demystifying clip data.arXiv preprint arXiv:2309.16671,

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data.arXiv preprint arXiv:2309.16671,

    Pith/arXiv arXiv

  23. [32]

    Msr-vtt: A large video description dataset for bridging video and language.2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5288–5296,

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language.2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5288–5296,

    2016

  24. [2015]

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick

    https://arxiv.org/abs/1512.03385. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. InCVPR, pages 9729–9738,

    Pith/arXiv arXiv

  25. [2016]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer

    https://api.semanticscholar.org/CorpusID:206594535. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11941–11952, 2023a. https://api. semanticscholar.org/CorpusID:257767223. Xiaohua Zhai, Basil Mustafa, Alexander Kolesni...

    Pith/arXiv arXiv 2023

  26. [2017]

    Hong-You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang, Marcin Eichner, Keen You, Meng Cao, Bowen Zhang, Yinfei Yang, and Zhe Gan

    https: //api.semanticscholar.org/CorpusID:206596127. Hong-You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang, Marcin Eichner, Keen You, Meng Cao, Bowen Zhang, Yinfei Yang, and Zhe Gan. Contrastive localized language-image pre-training. InForty-second International Conference on Machine Learning, 2025.https://openreview.net/forum?id=sGQEOXlezg. Ting Chen, ...

    2025

  27. [2018]

    Locca: Visual pretraining with location-aware captioners.arXiv preprint arXiv:2403.19596,

    Bo Wan, Michael Tschannen, Yongqin Xian, Filip Pavetic, Ibrahim Alabdulmohsin, Xiao Wang, André Susano Pinto, Andreas Steiner, Lucas Beyer, and Xiaohua Zhai. Locca: Visual pretraining with location-aware captioners.arXiv preprint arXiv:2403.19596,

    Pith/arXiv arXiv

  28. [2020]

    Language models are few-shot learners.arXiv preprint arXiv:2005.14165,

    Tom B Brown. Language models are few-shot learners.arXiv preprint arXiv:2005.14165,

    Pith/arXiv arXiv 2005

  29. [2021]

    Quo vadis, action recognition? a new model and the kinetics dataset

    João Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724–4733,

    2017

  30. [2022]

    Tenenbaum, and Boris Katz

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Joshua B. Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. InNeural Information Processing Systems, 2019.https://api.semanticscholar.org/CorpusID:202777185. Ioana Bica, Anastasija Ilić, Mat...

    Pith/arXiv arXiv 2019

  31. [2023]

    Poggio, and Thomas Serre

    Hilde Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso A. Poggio, and Thomas Serre. Hmdb: A large video database for human motion recognition.2011 International Conference on Computer Vision, pages 2556–2563,

    2011

  32. [2024]

    Token merging: Your vit but faster.ArXiv, abs/2210.09461, 2022.https://api.semanticscholar.org/CorpusID:252968113

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster.ArXiv, abs/2210.09461, 2022.https://api.semanticscholar.org/CorpusID:252968113. Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed...

    Pith/arXiv arXiv 2022

  33. [2025]

    Hangbo Bao, Li Dong, and Furu Wei

    https://arxiv.org/abs/2506.09985. Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. InICLR,

    Pith/arXiv arXiv