Home activity benchmark shows AI question-answering gaps
HOME-KGQA tests multimodal KGQA on daily household tasks, where LLM methods lag behind their encyclopedic results.
cs.MM
Multimedia
Roughly includes material in ACM Subject Class H.5.1.
sort pith recommended most recent
Story-driven game embeds design rules for ADHD transition support
Literature on ADHD and LD challenges is turned into concrete game features that target academic, social, and organizational hurdles during t
First complete virtual reading of Herculaneum scroll
X-ray microtomography and refined algorithms recover full text from unopened ancient papyrus without damage.
· “Complete virtual unwrapping and reading of a rolled Herculaneum papyrus”
Paired counterfactuals expose omni models at 32-41% binding accuracy
3600-item benchmark from 256 hours of video shows calibration without retraining lifts open models by up to 10 points and aids other tasks.
Automated matching creates region labels for better person search
ROGLE mines pseudo region-sentence pairs from existing data to add local alignment, raising accuracy on detailed natural-language queries.
Semantic codebook creates style-matched co-speech gestures
By organizing motion codes according to gesture semantics and applying reference prompts, the system produces motions faithful to both word,
CPR restores periodic structure from locally private time series
Multi-scale probing and phase-matched aggregation cut reconstruction error versus standard LDP methods at tight privacy budgets.
· “Period-conscious Time-series Reconstruction under Local Differential Privacy”
Dual-path diffusion sharpens lip sync and head poses
Separating expression and pose features while highlighting intense motion frames produces more accurate talking-head video from audio.
· “KSDiff: Keyframe-Augmented Speech-Aware Dual-Path Diffusion for Facial Animation”
300 generated tasks make visual reasoning trainable and verifiable
Suite training transfers to seven external benchmarks; rule-based scorers beat VLM judges and power multi-task RL.
· “VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning”
Top-5 memory beats top-200 in 134 ms for voice agents
Dual-brain retrieval hides inside the silence gap, adding facts and empathy without slowing the conversation down.
· “VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction”
One neural field answers any lead time and grid for temperature forecasts
Regional 2-m temperature forecasting as query-conditioned field evaluation beats fixed-grid baselines in skill.
· “Learning Continuous Regional Temperature Fields with Lead-Time and Resolution Queries”
Flow-matching radar retrieval cuts sampling FLOPs 6.7x
A spectral-local-wavelet backbone lifts CSI at 35 dBZ from 0.094 to 0.146.
Task-vector injection cuts furniture-collision errors in frozen MLLMs
DART-I routes spatial and color priors into hidden states, beating LoRA fine-tuning without retraining the backbone.
Head avatars shrink to 2.83 MB
Semantic-aware density control puts Gaussian detail where eyes go, not evenly, cutting storage to a fraction.
· “SACHA: Semantic-Aware Compression for 3D Gaussian Head Avatars”
Top AI models score under 30% on cultural video moments
Naming helps some models spot a concept; spotting it rarely helps them locate the moment in time.
· “Cultural Moment Benchmark: Evaluating Video Cultural Reasoning and Grounding in Southeast Asia”
Best egocentric-video QA system hits 58%
All four teams beat chance on 185 questions over 600+ hours of egocentric video; nine questions stumped everyone.
Modality alignment fails at one depth
Per-layer shared-private experts keep text, vision, and audio cues at their natural semantic depth.
· “Adaptive Hierarchical Representation Alliance for Multimodal Learning”
This paper builds a fashion knowledge graph from textbooks and combines it with a…
A textbook-grounded fashion knowledge graph plus pruning and grounding retrieval steps improves fashion QA accuracy over non-RAG and KG-RAG…
0.7057 final: speech-to-mask pipeline takes second on MeViS-Audio
Delayed commitment, trajectory ranking, and verified empty recovery let absent queries stay empty.
Two-stage art generator scores 1.00 on every emotion-art axis
Per-field reference retrieval plus targeted repair turns vague captions into aligned art, ranking 2nd in the challenge.
· “ReART: Reference-Guided Retrieval and Refinement for Emotion-Aware Art Generation”
Benchmark shows multimodal AI falls for situational illusions
Even top models stay below 63% accuracy on scenes whose look contradicts their physical state.
· “Beyond What Meets the Eye: Unveiling Situational Illusions for Multimodal Large Language Models”
Three-agent workflow beats standalone VLMs on a new image-text task
Q2IT lifts Gemini's joint accuracy from 0.507 to 0.722 on the ITJoint benchmark.
· “Query-Driven Multimodal Information Extraction from Long Documents”
Byte-level priors lift restoration
On corrupted MJPEG and H.264 videos, injecting byte-derived action priors raises PSNR by 2.51 dB, CIDEr by 0.20, and PCK by 0.18.
Training-free AI pipeline wins micro-action benchmark by 10 points
Splitting recognition from explanation in frozen models lifts fine-grained reasoning by 28 points.
Masking image details during training curbs AI hallucinations
A dual loss forces vision-language models to prefer intact images, cutting hallucination rates on three benchmarks.
· “PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment”
Saxophone robot makes music-emotion experiments reproducible
Humanoid performers let researchers isolate acoustic, visual, and social cues in music-evoked emotion.
· “Humanoid Musical Robots as Experimental Interfaces for Music-Evoked Emotion”
Metadata-routed adapters lift ScienceQA accuracy to 93.1 percent
With the backbone frozen, routing small adapters by subject, grade, and visual cues adds 2.6 points of accuracy.
Multi-view reasoning can be pretrained once and reused
A frozen model trained only on synthetic episodes matches task-specific rivals; light adapters top most benchmarks.
· “Pretraining Reusable Inference Across Views with Synthetic Task Priors”
MeGraS turns media files into queryable graph nodes
Images, video, and audio become first-class nodes, so SPARQL can search and segment them directly.
· “MediaGraph: A Content-Aware Data Model and Query Framework for Multimodal Knowledge Graphs”
Audio beats text 57% to 27% on eight speech stances
A new prosody-only benchmark shows tone, not words, carries confident, sarcastic, and impatient attitudes.
· “SpeechSense: A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis”
Model-written folk dialogues lift music-AI scores 15 points
Model-generated training data plus latent audio alignment lifts Qwen2.5-Omni by 14.9 points on folk music.
Look beyond audio to decode Pop music culture
Music-computing survey finds single-modality studies and biased sample corpora; it maps three data-driven fixes.
DynaForcing restores motion to frozen AI avatars at 45 FPS
Anchored rollouts plus motion rewards break the freeze that plagues distilled streaming avatars.
Attack steers face-recognition models to leak faces at 92% accuracy
White-box attack steers an ODE flow toward target identities, beating published baselines on six face models.
· “Steering the Flow: Inverting Face Recognition Models via Gradient-Guided Flow Matching”
Task-aware codec hits 91.4% accuracy with a single trained model
One universal image codec steers bits to task-relevant regions at runtime, matching task-specific codecs within 1.9 points.
· “UniTAC: Universal Task-Aware Compression via Weighted Distortion Measures”
Foveated MLLMs match human outcomes
Identical retinal input still yields non-human scanpaths, so outcome metrics miss the process gap.
· “Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humans”
No-training video pipeline lifts face identity by a third
Draft the motion, face-swap a few anchors, interpolate the rest — no fine-tuning needed.
· “KeyID: Decoupled Drafting and Keyframe Editing for Identity-Preserving Video Generation”
One fine-tuned video model gives any 3D character lip-synced speech
AnyTalk lifts 2D talking-head video to blendshape animation, no per-character training data required.
· “AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Model”
Reading videos clip by clip lifts hate detection by up to 7.4%
CLARA aligns speech, audio, and visuals inside short clips to catch hate that builds across a video.
· “CLARA: Clip-Level Multimodal Alignment with VLM-Derived Rationales for Hateful Video Detection”
Internalized visual thinking beats image-drawing reasoning 5x faster
Training on future-frame embeddings matches explicit visual chain-of-thought while cutting inference latency by over 5x.
· “Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning”
7B manual-trained planner beats GPT-5 tenfold on appliances
Synthetic data from appliance manuals lets a small open model outplan zero-shot giants on real robots.
· “Scaling Manual-Grounded Appliance Manipulation with Data Synthesis and Unified Planning”
Path overlays lift navigation-instruction quality past prior models
Map-free path, turn, and goal overlays drive frozen-follower success to 63.3 percent.
Fake-news detector makes LLMs argue both sides before ruling
RoE-FND mines past reasoning mistakes into reusable guidelines and outperforms trained detectors on five benchmarks.
Four compression ratios from one neural codec
A single set of weights switches between rates 2, 4, 8, and 16, cutting storage needs by 61%.
· “Flexible Deep Joint Source-Channel Coding: A Vibrotactile Example”
One offline pass makes repeated multimodal delivery nearly 4x faster
Prepared once and stored at the edge, each request sends only the parameters it needs, cutting rate and latency.
· “ParaJSCC: A Parameterized Framework for Reusable Multimodal Joint Source-Channel Coding”
Editing a JSON sketch beats text-only geometry reasoning
Tool-augmented meta-editing reaches 56.1% on a hard exam benchmark, beating a 48.7% proprietary baseline.
Tailored image semantics beat user-agnostic links by 45.8 percent
A 210-bit codebook stream keeps source meaning and matches receiver history, even at low SNR.
· “Personalized Digital Semantic Communication for Image Transmission with Vision-Language Models”
AlignFace beats LPIPS and DreamSim on human face-similarity picks
A metric that splits similarity into 20 explainable facial attributes matches human choices and per-attribute reasoning
· “AlignFace: Human-Aligned Face Similarity Metric with Interpretable Concept Relations”
Six hours of improv: players agree on stability
Six hours of duo improvisation, annotated from both sides, show perception lags intention.
· “H2H Music Improv: A Communication Model and Audio-Visual Dataset for Music Improvisation”
Best AI model scores 76 percent on long Japanese video benchmark
NARU pairs narrative tracking with cultural subtext like “reading the air,” where even leading models struggle most.
Document AI leaks name and ID when images go blank
New framework DRUF drops correlated-field leakage from 64% to near zero while preserving extraction quality.
· “Beyond Visual Evidence: Revealing and Mitigating Relational Privacy Leakage in Document MLLMs”
Alignment drift slows ASR speculation
Draft attention drifts up to 21 frames; verification attention recenters it to 2, restoring speed.
· “Alignment Drift in Single-Model Speculative Decoding for ASR: Mechanism, Correction, and Cost”
Coverage beats repulsion when search must narrow results
When narrowing to one place or hour, MASCOT keeps R@10 at 0.94 where the DPP baseline drops to 0.49.
· “MASCOT: Model-Aware Submodular Coverage for Composite-Attribute Text-to-Image Retrieval”
Closed-loop video search beats random trial-and-error
A two-stage optimizer tunes prompt, seed, and guidance scale, winning up to 69% of human preference tests.
· “Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence”
5x smaller learned image codecs
Per-layer bit allocation keeps compression quality while killing cross-platform decode failures.
· “HAMP-LIC: Hessian-Aware Mixed-Precision Post-Training Quantization for Learned Image Compression”
One neural codec holds point clouds together through 30% packet loss
A single loss-rate-aware model scatters the damage, restores missing latent features, and beats standard codecs on ShapeNet and…
· “ResPCC: A Loss-Resilient Neural Point Cloud Codec over Lossy Networks”
Agent loop repairs all 11 VSR pipeline faults
Where static checks fix only 2 of 11, real inference feedback drives 100% recovery on the CMLR system.
Three added training losses lift novel-class mask AP by 6.5 points
Same pseudo-masks, new losses: open-vocabulary segmentation improves through captions and CLIP-verified synonyms.
Dialogue timing helps AI film music match the scene
Speech-driven adapter beats baselines on a new 246-hour public-domain film benchmark.
· “Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections”
Keeping audio time structure beats 26-point gap over pooled vectors
Frozen Whisper frames with a linear projector reach 97.3% on music AVQA, beating larger omni-modal fine-tuning.
· “Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QA”
One piano model both performs and critiques
Frozen embeddings from score-conditioned JEPA training beat separate baselines on quality, ranking, technique, and mistake tasks.
· “MAJEPPA: Morphing and Assessing in a Unified Piano Performance Space”
Object memory beats text memory for egocentric video Q&A
A relative 4D scene graph lifts object-question accuracy 6.7 points and when-questions 12.5 points on EgoLifeQA.
Benchmark measures AI memory across a year of phone life
Multi-hop recall, temporal reasoning, preference drift, and abstention all get measured on synthesized year-scale trajectories.
Four relay rules cut live-video frame tail from 385 ms to 114 ms
A thin rule layer between Media over QUIC and multipath QUIC is the only setup to meet the 150 ms interactive-latency budget.
· “Media-over-Multipath-QUIC for Realtime Video Applications”
Understanding art needs four perspectives at once
74,234 paintings each get narrative, formal, emotional, and historical captions — none works alone.
· “MMArt: A Multi-Perspective Multimodal Dataset for Visual Art Understanding”
KLT-then-RAHT coder beats prior plenoptic point cloud codecs
Moves plenoptic light-field attributes into G-PCC, reporting 1–3 dB gains over earlier schemes.
Vibration feedback boosts realism in dynamic XR scenes
A 29-actuator glove translating motion and edges into touch raised perceived realism for video; static textures saw no gain.
Make image and text predict alone: shortcut learning gone
New objective stops models leaning on text cues and improves diagnosis on glaucoma and chest X-ray benchmarks.
Masking whole modality streams triples recommender retention
When 95% of items lose a modality, the model keeps 61% of its accuracy; without the trick, only 22%.
· “Sequential Modality Dropout for Robust Multi-Modal Sequential Recommendation”
Phase-aware video streaming cuts teleop task time by 36%
Giving the biggest slice of a 3 Mbps uplink to the camera the operator needs lifts success from 48% to 71%.
· “TAMS: Task-Aware Multi-View Adaptive Streaming for Wireless Telerobotic Manipulation”
Zero-shot drone retrieval beats fine-tuned rivals at 39.47 mR
Region alignment and semantic perturbations sharpen fine-grained attribute matching at zero added inference cost.
One model now translates sign language to and from text
A shared tokenizer splits semantic and motion detail, letting one language model translate and produce signs from pose data.
Layout planning before audio improves long-range music structure
It writes a time-aligned structural plan before audio and beats a matched no-layout baseline on boundary scores.
· “MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation”
Adaptive speed and video summaries cut learning time
Reports 1.3x adaptive audio playback, 53% shorter lecture viewing, and pronunciation gains.
Deferred audio pruning keeps 97% accuracy at 78% less prefill
A-PACK thins video first, then prunes low-relevance audio and video inside the LLM, lifting decoding throughput 2.21x
· “Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs”
Token video survives fading without retransmission
Shuffling structural and motion tokens across packets lets receivers rebuild lost frames from survivors.
· “Loss-Resilient Wireless Video Token Communication over Block Fading Channels”
Latent world model tops video upscaling benchmarks
Causal sparse attention and a user smoothness dial keep quality high at 18.6 FPS on 768×1408 clips.
· “MotionCraft: Latent World Modeling with Sparse Attention for Visual Upscaling”
Relay coordination keeps 100-user volumetric streams under 40 ms
A relay that shares base content and selectively adds enhancement cuts link load 27% versus clustering.
· “MD2G-Cast: Relay-Coordinated Multicast for Scalable Volumetric Streaming over MoQ”
Object-based editing cuts audio drama creation from 17 to 4 hours
One change to a character or scene ripples through every linked sound asset automatically.
· “Dramarrator: Object-Based Audio Editing for Audio Drama Production from Books”
Ask yes or no, not timestamps: frozen VLMs find events
Swapping timestamp regression for a binary-question scan lifts R@0.5 by 28–50 points with zero training.
· “Your VLM Already Knows When: Training-Free Temporal Grounding by Asking Yes or No”
