REVIEW 5 cited by
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We present Any-Modality Augmented Language Model (AnyMAL), a unified model that reasons over diverse input modality signals (i.e. text, image, video, audio, IMU motion sensor), and generates textual responses. AnyMAL inherits the powerful text-based reasoning abilities of the state-of-the-art LLMs including LLaMA-2 (70B), and converts modality-specific signals to the joint textual space through a pre-trained aligner module. To further strengthen the multimodal LLM's capabilities, we fine-tune the model with a multimodal instruction set manually collected to cover diverse topics and tasks beyond simple QAs. We conduct comprehensive empirical analysis comprising both human and automatic evaluations, and demonstrate state-of-the-art performance on various multimodal tasks.
Forward citations
Cited by 5 Pith papers
-
SensorQA: A Question Answering Benchmark for Daily-Life Monitoring
SensorQA is a new crowdsourced benchmark showing that current AI models answer only about 28% of daily-life sensor-data questions correctly.
-
SensorChat: Answering Qualitative and Quantitative Questions during Long-Term Multimodal Sensor Interactions
A three-stage pipeline with LLM decomposition, pretrained embedding retrieval, and LLM assembly outperforms prior sensor QA systems on long-duration, high-frequency data, with caveats on evaluation leakage.
-
QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization
QLPO resamples GRPO training groups to favor short correct and long incorrect responses, cutting reasoning length substantially while keeping accuracy roughly unchanged.
-
CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models
CoF improves multimodal LLM benchmark scores by having the model locate an answer region, then reweighting attention toward that region during inference.
-
A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future
A historical review that organizes multimodal explainability methods into four chronological eras and three explainability types, extending coverage to generative LLMs.
Discussion (0). Continue with ORCID to comment.