{"id":"423c531a-ee62-4359-9dc1-20ce4ed42ede","arxiv_id":"2605.26244","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"LongAV-Compass is a new benchmark and evaluation framework for minute-scale audio-visual generation across T2AV, I2AV, and V2AV with multi-dimensional assessment.","lead":"The paper introduces LongAV-Compass, a benchmark with 284 test cases for minute-long audio-visual generation from text, image, or video inputs, along with a unified evaluation framework using MLLM and perceptual metrics. Researchers in AI multimedia generation should read it to understand how to diagnose coherence and alignment failures that short-clip tests miss.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"Reader correctly flags the accuracy of the metrics+cases for long-horizon measurement as the pivotal assumption. No evidence from the abstract or claim description shows this assumption fails, so the contribution as a testbed stands; full text would be needed only for reproducibility details, not for soundness of the argument itself.","tokens_in":1808,"tokens_out":247,"duration_ms":16549,"concrete_test":"Compute Pearson/Spearman correlation between the MLLM-assisted scores and the human-alignment validation scores across the reported dimensions; if mean correlation < 0.65 on identity consistency or narrative coherence, the diagnostic reliability is limited.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the 284-case benchmark with MLLM assessment plus DINO-v2/ArcFace/CLIP/ImageBind metrics, validated on 11 models, supplies a usable diagnostic for minute-scale degradation. The construction (taxonomy-guided cases spanning T2AV/I2AV/V2AV, 20+ fine-grained dimensions, human alignment) follows standard benchmark practice and contains no evident internal contradiction or unsupported leap from the described components.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces LongAV-Compass, a benchmark for minute-scale audio-visual generation containing 284 curated test cases spanning T2AV, I2AV, and V2AV modalities. Cases are organized by application scenario and generation complexity using a taxonomy-guided approach. The unified evaluation framework integrates MLLM-assisted assessment with perceptual and multimodal metrics (DINO-v2, ArcFace, CLIP, ImageBind) to score more than 20 fine-grained dimensions covering within-segment quality, cross-segment consistency, global narrative coherence, semantic alignment, and audio-visual synchronization. Experiments on 11 representative models are paired with human-alignment validation to position the benchmark as a diagnostic testbed for limitations in sustaining coherent, semantically aligned, and temporally consistent minute-scale generation across input modalities.","tokens_in":1875,"tokens_out":541,"duration_ms":27003,"significance":"If the curation, metrics, and human validation hold, the work addresses a genuine gap in existing short-clip benchmarks by enabling analysis of temporal degradation in identity consistency, narrative coherence, and audio-visual alignment. The multi-modality coverage and fine-grained dimensions could serve as a practical reference for diagnosing model weaknesses in long-form generation, with the human-alignment step providing a useful check on automated metrics.","major_comments":[{"comment":"Abstract and Experiments section: The central claim that LongAV-Compass supplies a usable diagnostic testbed rests on experiments with 11 models and human validation, yet the manuscript provides no quantitative results, per-model scores, degradation trends, or error analysis. Without these, the diagnostic utility remains asserted rather than demonstrated.","section":"Abstract / Experiments"},{"comment":"Benchmark Construction section: The 284 test cases are described as taxonomy-guided and spanning 20+ dimensions, but the manuscript supplies no explicit selection criteria, inter-annotator agreement statistics, or ablation showing that the chosen cases reliably surface degradation in identity consistency and narrative coherence over minute-scale horizons.","section":"Benchmark Construction"}],"minor_comments":[{"comment":"A table or figure summarizing the distribution of the 284 cases across modalities (T2AV/I2AV/V2AV) and complexity levels would improve readability of the benchmark composition.","section":"Benchmark Construction"},{"comment":"The description of how MLLM-assisted scores are aggregated with DINO-v2/ArcFace/CLIP/ImageBind metrics would benefit from an explicit formula or pseudocode in the Evaluation Framework section.","section":"Evaluation Framework"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and will revise the manuscript to strengthen the empirical demonstration and construction transparency.","responses":[{"response":"We agree that the current manuscript version does not present the quantitative results, per-model scores, degradation trends, or error analysis. In the revised version we will add these elements from the experiments on the 11 models together with the human-alignment validation results, thereby demonstrating rather than asserting the benchmark's diagnostic value.","revision_made":"yes","referee_comment":"[Abstract / Experiments] Abstract and Experiments section: The central claim that LongAV-Compass supplies a usable diagnostic testbed rests on experiments with 11 models and human validation, yet the manuscript provides no quantitative results, per-model scores, degradation trends, or error analysis. Without these, the diagnostic utility remains asserted rather than demonstrated."},{"response":"We acknowledge the absence of these details. The revised Benchmark Construction section will include explicit selection criteria, inter-annotator agreement statistics, and an ablation study confirming that the 284 cases surface the targeted degradations in identity consistency and narrative coherence.","revision_made":"yes","referee_comment":"[Benchmark Construction] Benchmark Construction section: The 284 test cases are described as taxonomy-guided and spanning 20+ dimensions, but the manuscript supplies no explicit selection criteria, inter-annotator agreement statistics, or ablation showing that the chosen cases reliably surface degradation in identity consistency and narrative coherence over minute-scale horizons."}],"tokens_in":1463,"tokens_out":336,"duration_ms":24755,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that this paper introduces LongAV-Compass, a benchmark with 284 cases for evaluating minute-long audio-visual outputs conditioned on text, images, or video.\n\nIt is new in moving past the usual 5-10 second clips and in trying to cover T2AV, I2AV, and V2AV in one framework. The taxonomy for organizing cases by scenario and complexity, plus the list of more than 20 dimensions on consistency and alignment, is a reasonable step beyond existing short-form tests. Running the setup on 11 models and adding human alignment checks shows they are thinking about practical use.\n\nThe soft spots are the missing quantitative results and curation details. The abstract claims the metrics plus MLLM assessment give a usable diagnostic for degradation in identity, narrative, and sync, but without numbers or error analysis it is hard to tell how well DINO-v2, ArcFace, CLIP, and ImageBind actually track those issues over long horizons. The assumption that 284 cases are enough to expose real limitations feels light without more validation data.\n\nThis is for researchers working on long-form audio-visual generation who need a testbed that goes beyond short clips. Readers in multimedia AI or generative modeling would get the most from the unified setup and the fine-grained dimensions.\n\nIt deserves a serious referee because benchmarks in this area can shape what counts as progress. I would send it for peer review but ask for the full experimental numbers and any checks on how well the metrics match human judgments of long-term coherence.","headline":"LongAV-Compass is a benchmark for minute-scale audio-visual generation across three modalities that addresses a clear gap but rests on thin evidence for its diagnostic power.","tokens_in":2479,"tokens_out":391,"would_cite":false,"duration_ms":27540,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"LongAV-Compass supplies 284 test cases and a multi-metric framework to diagnose coherence and alignment failures in minute-scale audio-visual generation across text, image, and video conditioning.","keywords":[],"falsifier":"Running the eleven models on the 284 cases and finding that high automated scores on all metrics correspond to low human ratings for overall coherence and alignment would falsify the benchmark's measurement validity.","tokens_in":2695,"feed_emoji":"🎬","tokens_out":693,"duration_ms":24642,"temperature":0.7,"pith_summary":"Existing benchmarks for audio-visual generation cover only short 5-10 second clips and rarely handle unified evaluation across text, image, and video inputs. LongAV-Compass fills the gap with a collection of 284 curated minute-long test cases organized by scenario and complexity for T2AV, I2AV, and V2AV tasks. The benchmark applies a unified evaluation framework that combines MLLM-assisted scoring with perceptual metrics including DINO-v2, ArcFace, CLIP, and ImageBind to track more than 20 dimensions such as within-segment quality, cross-segment consistency, narrative coherence, semantic alignment, and audio-visual synchronization. Experiments across 11 representative models plus human-alignment checks establish the benchmark as a diagnostic tool for identifying where current systems lose temporal consistency and semantic fidelity over extended durations.","feed_headline":"Benchmark tests minute-scale audio-visual generation across three modalities","feed_subtitle":"284 cases and combined MLLM plus perceptual metrics reveal where models lose coherence and alignment over full minutes.","key_machinery":"LongAV-Compass benchmark, whose taxonomy-guided test construction and unified evaluation framework integrate MLLM-assisted assessment with perceptual and multimodal metrics (DINO-v2, ArcFace, CLIP, ImageBind) to score over twenty dimensions of quality, consistency, and alignment.","core_discovery":"LongAV-Compass is a benchmark of 284 curated test cases spanning text-to-audio-video, image-to-audio-video, and video-to-audio-video generation, built through taxonomy-guided construction and evaluated via an integrated framework of MLLM-assisted assessment together with complementary metrics such as DINO-v2, ArcFace, CLIP, and ImageBind; the framework measures fine-grained dimensions of within-segment quality, cross-segment consistency, global narrative coherence, semantic alignment, and audio-visual synchronization, and experiments on eleven models with human validation demonstrate its utility in revealing limitations of current systems for sustaining coherent minute-scale outputs.","pith_inferences":["Model developers could target the identified failure modes by retraining on subsets of the benchmark that isolate particular consistency dimensions.","The benchmark construction method could be extended to longer sequences or new modalities to test whether the observed degradation patterns continue or saturate.","Integration into training loops might allow direct optimization against the fine-grained scores rather than short-clip proxies.","The diagnostic results could guide selection of conditioning modality for specific application scenarios that require sustained narrative.","keywords:["],"forward_implications":["Current models show measurable drops in identity consistency and narrative coherence as generation length reaches one minute.","A single framework can now compare performance across three conditioning modalities on the same set of long-form criteria.","The multi-metric approach isolates specific failure modes such as cross-segment drift and synchronization loss.","Human validation confirms that the automated scores track human judgments on the targeted dimensions of quality and alignment."],"fun_headline_variants":["LongAV-Compass unifies evaluation of minute-scale audio-visual outputs","284 cases test long-form audio-visual coherence across modalities","Benchmark measures alignment degradation in minute-scale audio-video","Framework evaluates narrative flow in T2AV I2AV and V2AV generation"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The 284 curated test cases together with the chosen MLLM-assisted metrics and perceptual measures accurately capture degradation in identity consistency, narrative coherence, and audio-visual alignment over minute-scale durations.","fun_headline_variants_meta":{"raw":{"variants":["LongAV-Compass unifies evaluation of minute-scale audio-visual outputs","284 cases test long-form audio-visual coherence across modalities","Benchmark measures alignment degradation in minute-scale audio-video","Framework evaluates narrative flow in T2AV I2AV and V2AV generation"]},"model":"grok-4.3","cost_usd":0.004897,"raw_usage":{"total_tokens":2442,"prompt_tokens":751,"num_sources_used":0,"completion_tokens":69,"cost_in_usd_ticks":48974500,"prompt_tokens_details":{"text_tokens":751,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1622,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":751,"tokens_out":69,"duration_ms":15136,"temperature":1.0,"reasoning_tokens":1622,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T22:50:16.322759+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the eleven models on the 284 cases and finding that high automated scores on all metrics correspond to low human ratings for overall coherence and alignment would falsify the benchmark's measurement validity.","supporting_citations":[],"review_version":1}