Pith. sign in

REVIEW 3 cited by

PolyViT: Co-training Vision Transformers on Images, Videos and Audio

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2111.12993 v1 pith:JEEA5SAC submitted 2021-11-25 cs.CV cs.LG

Valerii Likhosherstov , Anurag Arnab , Krzysztof Choromanski , Mario Lucic , Yi Tay , Adrian Weller , Mostafa Dehghani This is my paper · ORCID
classification cs.CVcs.LG
keywords co-trainingdatasetsmodelmultiplepolyvitaudiomodalitiessingle
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Can we train a single transformer model capable of processing multiple modalities and datasets, whilst sharing almost all of its learnable parameters? We present PolyViT, a model trained on image, audio and video which answers this question. By co-training different tasks on a single modality, we are able to improve the accuracy of each individual task and achieve state-of-the-art results on 5 standard video- and audio-classification datasets. Co-training PolyViT on multiple modalities and tasks leads to a model that is even more parameter-efficient, and learns representations that generalize across multiple domains. Moreover, we show that co-training is simple and practical to implement, as we do not need to tune hyperparameters for each combination of datasets, but can simply adapt those from standard, single-task training.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Audio-Visual Continual Test-Time Adaptation without Forgetting

    cs.LG 2026-02 conditional novelty 6.0 of 10

    By adapting only the fusion layer and retrieving past good parameter states via raw input statistics, AV-CTTA outperforms existing audio-visual continual test-time adaptation methods and forgets far less.

  2. Any-to-3D Generation via Hybrid Diffusion Supervision

    cs.CV 2024-11 reject novelty 6.0 of 10

    XBind generates 3D objects from text, image, or audio prompts using ImageBind aligned embeddings and hybrid 2D/3D diffusion supervision.

  3. Sensitive Image Classification by Vision Transformers

    cs.CV 2024-12 conditional novelty 3.0 of 10

    Vision transformers, particularly LITv2, outperform ResNet baselines on a newly curated 3-class pornography classification dataset, but the evaluation is limited by dataset overlap and tuning issues.

Pith tools