Pith. sign in

REVIEW 15 cited by

VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.08345 v2 pith:6YZWALWP submitted 2023-04-17 cs.LG cs.CLcs.CVcs.MMeess.AS

Jing Liu , Sihan Chen , Xingjian He , Longteng Guo , Xinxin Zhu , Weining Wang , Jinhui Tang This is my paper · ORCID
classification cs.LGcs.CLcs.CVcs.MMeess.AS
keywords valormultimodalpretrainingaudiomodelvisionvision-audio-languagevision-language
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

In this paper, we propose a Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multi-modal understanding and generation. Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner. It contains three separate encoders for single modality representations, and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain VALOR model, including Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language and audio to the same common space, building vision-language, audio-language and audiovisual-language alignment simultaneously. MGC learns how to generate text tokens in conditions of vision, audio or their both. To promote vision-audio-language pretraining research, we construct a large-scale high-quality tri-modality dataset named VALOR-1M, which contains 1M audiable videos with human annotated audiovisual captions. Extensive experiments show that VALOR can learn strong multimodal correlations and be generalized to various downstream tasks (e.g., retrieval, captioning and question answering), with different input modalities (e.g., vision-language, audio-language and audiovisual-language). VALOR achieves new state-of-the-art performances on series of public cross-modality benchmarks. Code and data are available at project page https://casia-iva-group.github.io/projects/VALOR.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MOON: Multi-Objective OrthoNormalized Updates for Multitask Learning

    cs.LG 2026-08 conditional novelty 7.0 of 10

    MOON applies spectral-nuclear-norm geometry to multi-objective gradient manipulation and uses polar-factor updates, with O(T^-1/2) deterministic and O(T^-1/4) stochastic convergence to Pareto stationarity.

  2. A Benchmark for Omni-Modal Reasoning in Long Videos

    cs.CV 2025-12 reject novelty 7.0 of 10

    A new 45-minute-scale omni-modal video Q&A benchmark and a training-free retrieval-refine agent, whose reported agent score (66.64% in the abstract) is not supported by the paper's own main results (44.66%).

  3. Patch-level Sounding Object Tracking for Audio-Visual Question Answering

    cs.MM 2024-12 conditional novelty 7.0 of 10

    A new patch-level sounding object tracking method with motion-, sound-, and question-driven graph modules achieves 78.42% average accuracy on MUSIC-AVQA, competitive with large-scale pretraining approaches.

  4. Empowering Long-form Omni-modal Understanding with Robust Audio Perception

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Decoupled audio-visual caption and CoT-QA datasets plus two-stage fine-tuning measurably strengthen auditory perception and cross-modal reasoning in a 7B omni-modal LLM.

  5. EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs

    cs.CV 2025-12 conditional novelty 6.0 of 10

    EchoingPixels prunes audio-visual LLM input tokens jointly across modalities and re-tunes RoPE frequencies so that 5–20% of tokens retain roughly full-model performance.

  6. CAT-SG: A Large Dynamic Scene Graph Dataset for Fine-Grained Understanding of Cataract Surgery

    cs.CV 2025-06 conditional novelty 6.0 of 10

    CAT-SG is a new cataract surgery scene graph dataset with 1.811 million relation annotations, a two-class technique recognition task, and a query-based scene graph generation baseline.

  7. HuMoCon: Concept Discovery for Human Motion Understanding

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A framework that combines explicit video-motion feature alignment with velocity-aware masked autoencoding to improve LLM-based human motion and video question answering.

  8. Temporal Object Captioning for Street Scene Videos from LiDAR Tracks

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A rule-based pipeline converts LiDAR tracks into template captions of traffic dynamics, and training SwinBERT on them lowers the Visual Bias Measure across three datasets.

  9. Gramian Multimodal Representation Learning and Alignment

    cs.CV 2024-12 conditional novelty 6.0 of 10

    GRAM replaces cosine similarity with the Gramian volume of the parallelotope formed by multiple modality embeddings, and a volume-based contrastive loss improves multimodal retrieval and classification.

  10. AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering

    cs.CV 2025-10 unverdicted novelty 5.0 of 10

    AV-Master reports state-of-the-art accuracy on four audio-visual question answering benchmarks by combining sequential question-guided focus sampling with modality-preference activation.

  11. Open-set Cross Modal Generalization via Multimodal Unified Representation

    cs.CV 2025-07 conditional novelty 5.0 of 10

    The authors propose OSCMG, an open-set version of Cross Modal Generalization, and show their MICU method with masked contrastive learning and unified jigsaw puzzles outperforms prior methods.

  12. From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approach

    cs.CV 2025-07 conditional novelty 5.0 of 10

    The paper presents GEST, an event-graph representation of videos that is converted automatically into natural language and is also used as a teacher to pre-train end-to-end video captioning models.

  13. VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding

    cs.CV 2024-12 conditional novelty 5.0 of 10

    VidHalluc is a 5,002-video paired benchmark for action, temporal sequence, and scene transition hallucinations in video MLLMs, and DINO-HEAL is a training-free saliency reweighting method that improves hallucination s...

  14. Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback

    cs.AI 2024-12 conditional novelty 4.0 of 10

    The paper proposes learning from language feedback to synthesize multimodal preference pairs, but the evidence is weakened by an undefined improvement metric and small, unvalidated effect sizes.

  15. A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning

    cs.MM 2024-11 conditional novelty 3.0 of 10

    A review that categorizes deep audio-visual correlation learning methods by architectures, objective functions, datasets, and evaluation metrics, and points to missing standardized benchmarks.

Pith tools