Pith. sign in

REVIEW 24 cited by

Meta-Transformer: A Unified Framework for Multimodal Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.10802 v1 pith:QGYNBJY2 submitted 2023-07-20 cs.CV cs.AIcs.CLcs.LGcs.MM

Yiyuan Zhang , Kaixiong Gong , Kaipeng Zhang , Hongsheng Li , Yu Qiao , Wanli Ouyang , Xiangyu Yue This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LGcs.MM
keywords datameta-transformermultimodalunifiedmodalitiesencoderframeworklearning
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Multimodal learning aims to build models that can process and relate information from multiple modalities. Despite years of development in this field, it still remains challenging to design a unified network for processing various modalities ($\textit{e.g.}$ natural language, 2D images, 3D point clouds, audio, video, time series, tabular data) due to the inherent gaps among them. In this work, we propose a framework, named Meta-Transformer, that leverages a $\textbf{frozen}$ encoder to perform multimodal perception without any paired multimodal training data. In Meta-Transformer, the raw input data from various modalities are mapped into a shared token space, allowing a subsequent encoder with frozen parameters to extract high-level semantic features of the input data. Composed of three main components: a unified data tokenizer, a modality-shared encoder, and task-specific heads for downstream tasks, Meta-Transformer is the first framework to perform unified learning across 12 modalities with unpaired data. Experiments on different benchmarks reveal that Meta-Transformer can handle a wide range of tasks including fundamental perception (text, image, point cloud, audio, video), practical application (X-Ray, infrared, hyperspectral, and IMU), and data mining (graph, tabular, and time-series). Meta-Transformer indicates a promising future for developing unified multimodal intelligence with transformers. Code will be available at https://github.com/invictus717/MetaTransformer

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 24 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features

    cs.CV 2025-05 conditional novelty 7.0 of 10

    CSTrack proposes compact spatial and temporal feature modules for RGB-X tracking, reporting new state-of-the-art results on DepthTrack, VOT-RGBD2022, LasHeR, RGBT234, and VisEvent.

  2. Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Structured intermediate probing that separates modality-specific from modality-general signals lets privileged training modalities improve single-modality MLLM inference by large margins over naive multimodal training.

  3. UniPre3D: Unified Pre-training of 3D Point Cloud Models with Cross-Modal Gaussian Splatting

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A point cloud pre-training method that uses 3D Gaussian splatting rendering and cross-modal image features to work for both objects and scenes.

  4. VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection

    cs.CV 2025-05 conditional novelty 6.0 of 10

    VAEmo achieves state-of-the-art audio-visual emotion recognition by pre-training a unified encoder and aligning it with MLLM-generated emotion captions.

  5. TxP: Reciprocal Generation of Ground Pressure Dynamics and Activity Descriptions for Improving Human Activity Recognition

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A bidirectional text-pressure model with a learned codebook generates synthetic pressure data from activity descriptions and classifies real pressure sequences via LLM-generated text, gaining up to 12.4 macro-F1 point...

  6. An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Llambda adapts LVLMs to low-resolution human-behavior videos by generating pseudo-label-guided captions from unlabeled data and fine-tuning with LoRA, reporting Bert-Score gains over LVLM baselines.

  7. Representation Learning for Tabular Data: A Comprehensive Survey

    cs.LG 2025-04 conditional novelty 6.0 of 10

    A comprehensive survey that categorizes deep tabular representation learning into specialized, transferable, and general models, with a feature/sample/objective taxonomy for specialized methods.

  8. VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A web-context-derived dataset and a 'leaky modality mix' of captions with Q&A pairs improve vision-language model fine-tuning on several benchmarks.

  9. Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Data

    cs.LG 2024-12 conditional novelty 6.0 of 10

    A 22-dataset benchmark compares existing multimodal AutoML tricks, and an automatic ensemble of those tricks achieves the most robust performance.

  10. SimCMF: A Simple Cross-modal Fine-tuning Strategy from Vision Foundation Models to Any Imaging Modality

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A small convolutional adapter plus a frozen patch embedding lets SAM segment depth, thermal, polarization, HHA, and NIR images far better than training from scratch, with parameter-efficient fine-tuning matching full ...

  11. Toward Long-Tailed Online Anomaly Detection through Class-Agnostic Concepts

    cs.CV 2025-07 conditional novelty 5.0 of 10

    The paper proposes the long-tailed online anomaly detection (LTOAD) benchmark and a class-agnostic concept-based framework that outperforms class-aware baselines in most offline settings and in the online setting.

  12. SkySense V2: A Unified Foundation Model for Multi-modal Remote Sensing

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A unified multi-modal remote sensing foundation model with adaptive patch merging, modality prompt tokens, mixture of experts, and query-based semantic aggregation contrastive learning outperforms SkySense by 1.8 poin...

  13. OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A shared-backbone transformer with pairwise modality training reports top results across 25 datasets spanning 12 modalities.

  14. CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigation

    cs.CV 2025-05 conditional novelty 5.0 of 10

    CoNav lets a frozen 3D-text model pass spatial text hints to a lightly fine-tuned image-text navigation agent, improving path efficiency on several VLN benchmarks, though not all claimed state-of-the-art results hold.

  15. Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach

    cs.DC 2025-02 reject novelty 5.0 of 10

    MPSL applies parallel split learning with a single aggregated server backward pass to fine-tune multimodal transformers, matching federated learning accuracy while reducing client compute by about 250x.

  16. Visualizing the Invisible: A Generative AR System for Intuitive Multi-Modal Sensor Data Presentation

    cs.HC 2024-12 conditional novelty 5.0 of 10

    Vivar uses barycentric interpolation in a pre-trained CLIP embedding space to generate AR visualizations of multi-modal sensor data, with caching that speeds generation 11x.

  17. Grounding Intelligence in Movement

    cs.AI 2025-07 conditional novelty 4.0 of 10

    Movement should be treated as a first-class AI modeling modality, and a unified, biomechanically grounded movement foundation model built from aggregated data across species and sensors is the proposed path forward.

  18. Multimodal Representation Alignment for Cross-modal Information Retrieval

    cs.IR 2025-06 conditional novelty 4.0 of 10

    Across CLIP, BLIP, Meta-Transformer, and three combined unimodal models on IMDB, Flickr30K, and MS-COCO, cosine similarity gives the best cross-modal retrieval for contrastively trained models, while learned MLP align...

  19. NTIRE 2025 Challenge on Day and Night Raindrop Removal for Dual-Focused Images: Methods and Results

    cs.CV 2025-04 conditional novelty 4.0 of 10

    The NTIRE 2025 challenge report presents a new day/night dual-focus raindrop removal benchmark and documents 32 solutions, with the best composite score of 34.35 achieved by a Restormer-based method.

  20. MoE$^2$: Optimizing Collaborative Inference for Edge Large Language Models

    cs.NI 2025-01 reject novelty 4.0 of 10

    MoE2 trains a gating network once on all edge LLMs, then selects a resource-feasible subset and a per-prompt top-k of experts to fuse token probabilities under latency and energy budgets.

  21. PSA-VLM: Enhancing Vision-Language Model Safety through Progressive Concept-Bottleneck-Driven Alignment

    cs.CV 2024-11 conditional novelty 4.0 of 10

    PSA-VLM adds a safety classifier and prompt-rewriting gate to LLaVA-style vision-language models, reporting state-of-the-art average safety scores on RTVLM and large gains on politics, porn, and cyberbullying datasets.

  22. Vision Generalist Model: A Survey

    cs.CV 2025-06 conditional novelty 3.0 of 10

    A structured review of vision generalist models, classifying them into encoding-based and sequence-to-sequence frameworks and summarizing datasets, benchmarks, techniques, and open problems.

  23. A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospects

    cs.CV 2025-06 conditional novelty 2.0 of 10

    A comprehensive survey of video scene parsing that organizes methods, datasets, metrics, and benchmark results across VSS, VIS, VPS, VTS, and OVVS.

  24. A Comprehensive Survey of Large AI Models for Future Communications: Foundations, Applications and Challenges

    cs.IT 2025-05 conditional novelty 1.0 of 10

    A survey organizing the growing literature on large AI models for 6G communications, with a classification of model types, training and evaluation methods, and a list of challenges.

Pith tools