HeRA aligns least-aligned attention heads in MLLMs using an MKNN-based contrastive objective to preserve cross-modal topological structure, yielding gains on vision-centric tasks and reduced hallucinations across 18 benchmarks.
hub Canonical reference
Beyond language modeling: An exploration of multimodal pretraining
Canonical reference. 100% of citing Pith papers cite this work as background.
15
Pith papers citing it
Background
100% of classified citations
hub tools
citation-role summary
background 5
citation-polarity summary
years
2026 15roles
background 5polarities
background 5representative citing papers
Modeling sparse visual state updates instead of full images cuts generated visual tokens ~55.6% and improves interleaved multimodal reasoning over full-image ULMMs.
A VLM planner that adaptively inserts latent visual thoughts of future states into its reasoning trace beats language-only and prior VLM planners on long-horizon kitchen tasks, especially under tight free space.
Audex unifies audio understanding and generation on a strong text MoE backbone with multi-stage SFT plus text-only Cascade RL, matching open SOTA audio scores while mostly retaining text capability.
Neuro-JEPA, a JEPA-and-MoE transformer pretrained on 1.55M brain MRI scans, outperformed prior neuroimaging foundation models and was the only one to beat a simple CNN baseline on average.
HYDRA-X presents the first unified multimodal model using a single ViT for holistic image-video tokenization, with ablations on attention and compression plus a latent-level editing improvement.
Tuna-2 shows that direct pixel embeddings can replace vision encoders in unified multimodal models, achieving competitive generation and stronger understanding at scale.
Discrete action tokenization in VLA models creates an information bottleneck that prevents vision encoder scaling from improving performance, unlike continuous policies, as validated on the LIBERO benchmark.
Rosetta proposes a composable multimodal pretraining method with MAOP to prevent catastrophic forgetting when expanding modalities beyond standard MoE and MoT approaches.
VGGT-Ω improves feed-forward reconstruction accuracy and efficiency by architectural simplifications, register-based attention, and training on much larger supervised and unlabeled video data.
SenseNova-U1 presents native unified multimodal models that match top understanding VLMs while delivering strong performance in image generation, infographics, and interleaved tasks via the NEO-unify architecture.
Semantic latent spaces from pretrained encoders outperform reconstruction-based spaces for robotic world models on planning and downstream policy performance.
Eywa enables language-based agentic AI systems to collaborate with specialized scientific foundation models for improved performance on structured data tasks.
A roadmap that defines architectural nativity for multimodal models and categorizes them into Multi-to-Text, Multi-to-Target, and Multi-to-Multi types while outlining an industrial pipeline toward unified transformer-based native multimodal modeling.
citing papers explorer
-
Mind the Heads: Topological Representation Alignment for Multimodal LLMs
HeRA aligns least-aligned attention heads in MLLMs using an MKNN-based contrastive objective to preserve cross-modal topological structure, yielding gains on vision-centric tasks and reduced hallucinations across 18 benchmarks.
-
DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Models
Modeling sparse visual state updates instead of full images cuts generated visual tokens ~55.6% and improves interleaved multimodal reasoning over full-image ULMMs.
-
APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts
A VLM planner that adaptively inserts latent visual thoughts of future states into its reasoning trace beats language-only and prior VLM planners on long-horizon kitchen tasks, especially under tight free space.
-
Unified Audio Intelligence Without Regressing on Text Intelligence
Audex unifies audio understanding and generation on a strong text MoE backbone with multi-stage SFT plus text-only Cascade RL, matching open SOTA audio scores while mostly retaining text capability.
-
Learning Sparse Latent Predictive Foundation Model for Multimodal Neuroimaging
Neuro-JEPA, a JEPA-and-MoE transformer pretrained on 1.55M brain MRI scans, outperformed prior neuroimaging foundation models and was the only one to beat a simple CNN baseline on average.
-
HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers
HYDRA-X presents the first unified multimodal model using a single ViT for holistic image-video tokenization, with ablations on attention and compression plus a latent-level editing improvement.
-
Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation
Tuna-2 shows that direct pixel embeddings can replace vision encoders in unified multimodal models, achieving competitive generation and stronger understanding at scale.
-
The Compression Gap: Why Discrete Tokenization Limits Vision-Language-Action Model Scaling
Discrete action tokenization in VLA models creates an information bottleneck that prevents vision encoder scaling from improving performance, unlike continuous policies, as validated on the LIBERO benchmark.
-
Rosetta: Composable Native Multimodal Pretraining
Rosetta proposes a composable multimodal pretraining method with MAOP to prevent catastrophic forgetting when expanding modalities beyond standard MoE and MoT approaches.
-
VGGT-$\Omega$
VGGT-Ω improves feed-forward reconstruction accuracy and efficiency by architectural simplifications, register-based attention, and training on much larger supervised and unlabeled video data.
-
SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture
SenseNova-U1 presents native unified multimodal models that match top understanding VLMs while delivering strong performance in image generation, infographics, and interleaved tasks via the NEO-unify architecture.
-
Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models
Semantic latent spaces from pretrained encoders outperform reconstruction-based spaces for robotic world models on planning and downstream policy performance.
-
Heterogeneous Scientific Foundation Model Collaboration
Eywa enables language-based agentic AI systems to collaborate with specialized scientific foundation models for improved performance on structured data tasks.
-
Toward Native Multimodal Modeling: A Roadmap
A roadmap that defines architectural nativity for multimodal models and categorizes them into Multi-to-Text, Multi-to-Target, and Multi-to-Multi types while outlining an industrial pipeline toward unified transformer-based native multimodal modeling.
- Latent Denoising Improves Visual Alignment in Large Multimodal Models