REVIEW 37 cited by
Aria: An Open Multimodal Native Mixture-of-Experts Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Information comes in diverse modalities. Multimodal native AI models are essential to integrate real-world information and deliver comprehensive understanding. While proprietary multimodal native models exist, their lack of openness imposes obstacles for adoptions, let alone adaptations. To fill this gap, we introduce Aria, an open multimodal native model with best-in-class performance across a wide range of multimodal, language, and coding tasks. Aria is a mixture-of-expert model with 3.9B and 3.5B activated parameters per visual token and text token, respectively. It outperforms Pixtral-12B and Llama3.2-11B, and is competitive against the best proprietary models on various multimodal tasks. We pre-train Aria from scratch following a 4-stage pipeline, which progressively equips the model with strong capabilities in language understanding, multimodal understanding, long context window, and instruction following. We open-source the model weights along with a codebase that facilitates easy adoptions and adaptations of Aria in real-world applications.
Forward citations
Cited by 37 Pith papers
-
Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models
Phase-aware expert merging based on routing role profiles preserves more MoE-VLM accuracy than global routing aggregation at matched compression ratios.
-
Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding
MF2 evaluates long-movie understanding by asking models to classify fact/fib claim pairs; the best model trails humans by 23.5 points in pairwise accuracy.
-
VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videos
A new dataset of 420 math video-question pairs with step-by-step reasoning annotations shows that current multimodal AI models, including the best proprietary system, answer fewer than half of the multi-binary questio...
-
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
Multimodal pretraining transfers asymmetrically: language boosts vision, understanding boosts generation, generation is mostly neutral, and early unified training prevents vision laziness.
-
InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoning
Large multimodal models do far worse when collision videos violate familiar physics, and their small gains come from text exemplars, not the videos.
-
MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models
MESH, a three-layer video hallucination benchmark, shows LVMs ace basic objects and coarse traits but slip badly on fine character details and multi-subject actions in longer clips.
-
HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
A hierarchical benchmark for multimodal models on human-centric visual understanding finds frontier models average under 60% and miss question-uncued visual evidence, with test-time scaling helping only marginally.
-
VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
VLM4D benchmarks spatiotemporal reasoning in VLMs and finds large gaps versus humans, with proposed methods showing partial improvement.
-
"Harmless to You, Hurtful to Me!": Investigating the Detection of Toxic Languages Grounded in the Perspective of Youth
The authors construct the first Chinese youth-toxicity dataset, show that youth and adult perceptions of toxic language diverge, and report that adding contextual meta information improves detection accuracy.
-
VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Models
A perturbation computed from early attention and value matrices can make LLaVA, Instruct-BLIP, and BLIP2-T5 fail to detect objects inside a specified image region while keeping the rest of the image usable.
-
AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.
-
AVC-DPO: Aligned Video Captioning via Direct Preference Optimization
Using preference pairs synthesized from the model's own prompt-varied outputs, DPO fine-tuning improves Qwen2.5-VL-7B's video captioning on the VDC benchmark from 43.9 to 51.1 average VDCSCORE.
-
CAVALRY-V: A Large-Scale Generator Framework for Adversarial Attacks on Video MLLMs
CAVALRY-V trains a two-stage generator with a semantic-visual loss to produce transferable adversarial video perturbations that reduce video and image MLLM benchmark scores.
-
IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering
A benchmark that scores vision-language models by reconstructing the 3D scene behind an image as executable Blender code finds the models fail mainly on spatial precision, not tool usage.
-
VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
A vision-expert-filtered, chain-of-thought-guided, iteratively fine-tuned reward model boosts a compact 7B model's ability to judge vision-language responses, especially detecting hallucinations.
-
SeqPE: Transformer with Sequential Position Encoding
SeqPE encodes each position as a symbolic digit sequence through a small Transformer, and with contrastive plus distillation losses it reports improved extrapolation in language, QA, and image classification.
-
VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos
VRBench is a benchmark of 960 long narrative videos with 8,243 human-written multi-step questions, plus a two-level evaluation of answer accuracy and reasoning quality for 31 large models.
-
Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning
Training a multimodal LLM on GPT-4o-generated chain-of-thought referring traces, then optimizing with GRPO, improves referring accuracy and abstention on HumanRef.
-
VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation
VCapsBench is a video caption quality benchmark with 109,796 QA pairs across 21 fine-grained dimensions on 5,677 videos, evaluating caption accuracy, inconsistency, and coverage.
-
TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments
TransBench is a new benchmark of 1,459 screenshots and 22,000 grounding instructions for measuring how well GUI agents transfer across app versions, platforms, and applications.
-
IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMs
Multimodal LLMs achieve at most 28.9% accuracy on IV-Bench, a new benchmark requiring both an external image and a video to answer each question.
-
Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark
Video-MMLU: a 1,065-video lecture benchmark where most AI video models score 10-50%, but text-only models answer 40% of quiz questions without video.
-
Temporal Preference Optimization for Long-Form Video Understanding
TPO trains video-LMMs to prefer answers generated from complete, relevant frames over answers from incomplete or irrelevant frames, improving temporal grounding on LongVideoBench, MLVU, and Video-MME.
-
PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension
PunchBench, a 54,000-question benchmark spanning cartoons, posts, comments, and memes, shows that multimodal LLMs lag humans in punchline comprehension, and its Simple-to-Complex Chain-of-Question prompt yields small ...
-
Apollo: An Exploration of Video Understanding in Large Multimodal Models
Design decisions for video-LMMs can be made on 2-4B models and datasets and transfer to larger models, yielding efficient Apollo models, though some SOTA claims are contradicted by the paper's own table.
-
VISTA: Enhancing Long-Duration and High-Resolution Video Understanding by Video Spatiotemporal Augmentation
VISTA creates synthetic long and high-resolution video instruction data by combining existing clips, and finetuning video LMMs on it improves their accuracy on long-video and high-resolution benchmarks.
-
Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models
A monolithic multimodal LLM that cuts pre-training data by 58% and first-token latency by up to 69% while matching or beating its predecessor on 15 benchmarks.
-
TimeExpert: An Expert-Guided Video LLM for Video Temporal Grounding
TimeExpert applies dynamic mixture-of-experts routing to video temporal grounding, reporting small state-of-the-art gains over TRACE on dense video captioning, moment retrieval, and highlight detection.
-
Enhancing Sports Strategy with Video Analytics and Data Mining: Assessing the effectiveness of Multimodal LLMs in tennis video analysis
VideoLLaMA2's tennis sequence edit score jumps from 39.7 to 76.0 when text coordinates from detection models are included in the prompt, and a separately fine-tuned CLIP encoder raises single-event accuracy from 0.41 to 0.56.
-
CyberV: Cybernetics for Test-time Scaling in Video Understanding
A training-free test-time feedback loop, using attention drift to select key frames, improves video MLLM accuracy, with the largest gains on knowledge-heavy VideoMMMU.
-
SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities
SMAR, a KL-based regularizer on per-modality expert routing, retains 86.6% of a Mixtral 8x7B's language score during visual instruction tuning with only 2.5% pure-text data.
-
FlexSelect: Flexible Token Selection for Efficient Long Video Understanding
FlexSelect selects a small fraction of query-relevant visual tokens using attention from an intermediate layer, improving long-video accuracy and inference speed across multiple VideoLLMs.
-
EVEv2: Improved Baselines for Encoder-Free Vision-Language Models
An encoder-free vision-language model using separate attention, normalization, and feed-forward weights for image versus text tokens outperforms earlier encoder-free models and narrows the gap to encoder-based VLMs wi...
-
ChatRex: Taming Multimodal LLM for Joint Perception and Understanding
ChatRex couples a universal proposal network with an LLM that retrieves box indices, reaching 48.2 mAP on COCO and strong referring and region-level results.
-
Street-Level Geolocalization Using Multimodal Large Language Models and Retrieval-Augmented Generation
Using SigLIP retrieval to feed a Qwen2-VL or InternVL2 model with similar and dissimilar coordinates yields reported street-level accuracies of 23.2%, 17.1%, and 24.3% on IM2GPS, IM2GPS3k, and YFCC4k.
-
DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understanding
DynTok dynamically merges similar adjacent visual tokens into groups, reducing video token counts to 44.4% with comparable or better video understanding accuracy.
-
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection
A 2B multimodal agent trained with two-stage supervised fine-tuning and synthesized hierarchical/reflection reasoning achieves competitive results on ScreenSpot and AndroidWorld.
Discussion (0). Continue with ORCID to comment.