Pith. sign in

REVIEW 7 cited by

MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2107.07502 v2 pith:IUP4KIT4 submitted 2021-07-15 cs.LG cs.AIcs.CLcs.CVcs.MM

Paul Pu Liang , Yiwei Lyu , Xiang Fan , Zetian Wu , Yun Cheng , Jason Wu , Leslie Chen , Peter Wu
show 4 more authors
Michelle A. Lee Yuke Zhu Ruslan Salakhutdinov Louis-Philippe Morency
This is my paper · ORCID
classification cs.LGcs.AIcs.CLcs.CVcs.MM
keywords multibenchmultimodalresearchlearningmodalitiesrobustnessdatasetsareas
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. It is a challenging yet crucial area with numerous real-world applications in multimedia, affective computing, robotics, finance, human-computer interaction, and healthcare. Unfortunately, multimodal research has seen limited resources to study (1) generalization across domains and modalities, (2) complexity during training and inference, and (3) robustness to noisy and missing modalities. In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas. MultiBench provides an automated end-to-end machine learning pipeline that simplifies and standardizes data loading, experimental setup, and model evaluation. To enable holistic evaluation, MultiBench offers a comprehensive methodology to assess (1) generalization, (2) time and space complexity, and (3) modality robustness. MultiBench introduces impactful challenges for future research, including scalability to large-scale multimodal datasets and robustness to realistic imperfections. To accompany this benchmark, we also provide a standardized implementation of 20 core approaches in multimodal learning. Simply applying methods proposed in different research areas can improve the state-of-the-art performance on 9/15 datasets. Therefore, MultiBench presents a milestone in unifying disjoint efforts in multimodal research and paves the way towards a better understanding of the capabilities and limitations of multimodal models, all the while ensuring ease of use, accessibility, and reproducibility. MultiBench, our standardized code, and leaderboards are publicly available, will be regularly updated, and welcomes inputs from the community.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FiGuRO: Intrinsic Dimension Estimation for Multi-Modal Data

    cs.LG 2026-08 conditional novelty 6.0 of 10

    FiGuRO estimates the intrinsic dimensionality of shared and private subspaces in multi-modal data by adaptively growing or shrinking low-rank bottleneck layers guided by a reconstruction-fidelity budget.

  2. EPIC: Efficient Prompt Interaction for Text-Image Classification

    cs.CV 2025-07 conditional novelty 6.0 of 10

    EPIC uses similarity-gated cross-modal prompt interactions in frozen CLIP to improve text-image classification accuracy while training about 1% of parameters.

  3. CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 9,356-pair Chinese multimodal financial benchmark reveals that state-of-the-art multimodal LLMs, including GPT-4V, still score below 53% on objective and 39% on subjective financial chart tasks.

  4. Efficient Quantification of Multimodal Interaction at Sample Level

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A lightweight estimator quantifies sample-level multimodal interactions (redundancy, uniqueness, synergy) in continuous distributions and uses them for data partitioning, distillation, and ensembling.

  5. An Open-Source Software Toolkit & Benchmark Suite for the Evaluation and Adaptation of Multimodal Action Models

    cs.LG 2025-06 conditional novelty 4.0 of 10

    MultiNet provides an open-source benchmark, data SDK, evaluation harness, and adapted VLA models for assessing generalization across vision, language, and action tasks.

  6. Generalizing Large Language Model Usability Across Resource-Constrained

    cs.LG 2025-05 conditional novelty 4.0 of 10

    The dissertation shows that text-centric prompting, inference-time optimization, and correct-by-construction synthetic data can improve LLM robustness and Verilog code generation under resource constraints.

  7. AlzheimerRAG: Multimodal Retrieval Augmented Generation for Clinical Use Cases using PubMed articles

    cs.IR 2024-12 reject novelty 3.0 of 10

    AlzheimerRAG, a PubMed-based multimodal retrieval-augmented generation system, is reported, but its PubMedQA results are in-sample because PubMedQA was used for both fine-tuning and testing.

Pith tools