{"id":"d9aaf103-c873-4918-b942-b914c1dc3349","arxiv_id":"2607.00726","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"AV-SyncBench is a new benchmark dataset of 3,269 videos that separates temporal and semantic audio-visual synchronization assessment across voice, music, and sound scenarios.","lead":"This paper introduces AV-SyncBench, a benchmark that decouples temporal offset detection from semantic matching in audio-visual synchronization evaluation. A smart generalist might read it to see how future multimodal models could be tested more precisely on real-world video data without mixed signals.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Data construction via auto-filtering + manual verification may introduce selection bias coupling temporal and semantic scores","rationale":"The reader's weakest_assumption directly identifies the load-bearing precondition for the 'fully separate' claim. Because the abstract is the only text supplied and the data-construction step is the sole described mechanism for achieving decoupling, the concern is both precise and central; no stronger internal inconsistency is visible from the given material.","tokens_in":1647,"tokens_out":317,"duration_ms":17790,"concrete_test":"Randomly sample 150 videos; obtain independent annotations for (a) presence/visibility of on-screen sound source and (b) separate 1-5 difficulty ratings for temporal offset and semantic match; compute Pearson correlation between the two difficulty ratings. If r > 0.25, the decoupling is compromised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that AV-SyncBench enables fully independent assessment of temporal offset detection versus semantic matching. This rests on the dataset of 3,269 in-the-wild videos (Voice/Music/Sound, 10 scenarios, 5 tasks) where automatic filtering plus manual verification reliably identifies on-screen sound sources. If the verification step preferentially retains videos in which temporal alignment and semantic content are naturally correlated (e.g., clear on-screen speech with visible lip motion versus off-screen ambient sound), then difficulty on the temporal tasks will covary with difficulty on the semantic tasks, undermining the decoupling. The abstract provides no quantitative check on this independence.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes AV-SyncBench as the first benchmark to decouple temporal offset detection from semantic matching in audio-visual synchronization evaluation. It constructs a dataset of 3,269 in-the-wild videos (Voice/Music/Sound categories, 10 scenarios, 5 tasks) via automatic filtering plus manual verification to ensure on-screen sound sources, then evaluates five representative models on alignment quality and downstream tasks, with code and data released.","tokens_in":1742,"tokens_out":447,"duration_ms":13521,"significance":"If the claimed decoupling holds without selection bias, the benchmark would enable independent diagnosis of temporal versus semantic failures in multimodal feature extractors, addressing a documented limitation of prior coupled protocols. The scale (38,390 samples) and public release would support reproducible progress in audio-visual understanding and generation.","major_comments":[{"comment":"Abstract and data construction description: the central claim of 'fully separate temporal and semantic evaluation' rests on the assertion that automatic filtering plus manual verification reliably identifies on-screen sources without coupling the two dimensions, yet no quantitative check (e.g., correlation between temporal-task and semantic-task difficulty, or error rates from the verification step) is reported; this is load-bearing for the decoupling guarantee.","section":"Abstract / Data construction"},{"comment":"Evaluation of five models: without reported inter-annotator agreement or false-positive rates from manual verification, it is unclear whether retained videos preferentially preserve natural correlations between visible lip motion and speech timing, which would undermine independent scoring on the five tasks.","section":"Evaluation section"}],"minor_comments":[{"comment":"Clarify the exact definitions and sample counts for each of the five challenge tasks and how they map to the temporal versus semantic axes.","section":"Benchmark description"},{"comment":"The abstract states '38,390 samples' but does not specify whether this counts clips, pairs, or annotations; add a table breaking down the dataset statistics by category and task.","section":"Dataset statistics"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address the two major comments point by point below.","responses":[{"response":"The primary mechanism for decoupling is the design of the five tasks themselves: temporal tasks evaluate offset detection via controlled shifts independent of content semantics, while semantic tasks evaluate matching via content mismatches without temporal offsets. Automatic filtering combined with manual verification ensures on-screen sources as a prerequisite for both task types to be valid, but this step does not couple the evaluation dimensions. We agree that explicit quantitative validation would strengthen the claim. In revision we will add correlation analysis between temporal-task and semantic-task difficulties and include available statistics on the verification step.","revision_made":"partial","referee_comment":"[Abstract / Data construction] Abstract and data construction description: the central claim of 'fully separate temporal and semantic evaluation' rests on the assertion that automatic filtering plus manual verification reliably identifies on-screen sources without coupling the two dimensions, yet no quantitative check (e.g., correlation between temporal-task and semantic-task difficulty, or error rates from the verification step) is reported; this is load-bearing for the decoupling guarantee."},{"response":"The verification targets on-screen sound sources across Voice, Music, and Sound categories to support evaluation validity; the task construction (offset shifts for temporal tasks, content mismatches for semantic tasks) enables independent scoring regardless of residual natural correlations in the source videos. We did not originally report inter-annotator agreement or false-positive rates. We will revise the data construction section to provide a more detailed description of the verification protocol and, to the extent the original process permits, include agreement metrics or estimated error rates.","revision_made":"partial","referee_comment":"[Evaluation section] Evaluation of five models: without reported inter-annotator agreement or false-positive rates from manual verification, it is unclear whether retained videos preferentially preserve natural correlations between visible lip motion and speech timing, which would undermine independent scoring on the five tasks."}],"tokens_in":1271,"tokens_out":426,"duration_ms":25579,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that this paper releases a benchmark of 3,269 in-the-wild videos meant to let people score audio-visual models on temporal alignment separately from semantic content matching. They filter automatically then verify manually, cover voice/music/sound across 10 scenarios and 5 tasks, and run five existing models through it.\n\nWhat is new is the explicit attempt to decouple the two axes inside a single collection rather than running separate protocols. The dataset size and the public code are concrete steps forward for anyone who needs controlled test cases.\n\nThe soft spot is exactly the one the stress-test flagged. Automatic filtering plus manual verification does not automatically guarantee that temporal difficulty and semantic difficulty stay independent. If the retained videos tend to have clear on-screen sources where lip motion and sound content line up naturally, then the two scores will still covary. The abstract gives no numbers on inter-annotator agreement or on any post-hoc check that the axes are actually orthogonal in practice.\n\nThis is a benchmark paper, not a modeling advance. Readers who build or tune multimodal feature extractors will find the split useful if the independence claim holds up. Readers looking for new theory or large-scale training gains will not.\n\nIt is worth sending to referees. The idea is straightforward and the release is real; the main job for review is to pressure-test whether the manual step actually delivered the promised separation.","headline":"AV-SyncBench tries to split temporal offset from semantic matching in one dataset, but the construction method leaves that split unverified.","tokens_in":2235,"tokens_out":356,"would_cite":false,"duration_ms":14961,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"AV-SyncBench evaluates audio-visual synchronization by separating temporal offset detection from semantic content matching.","keywords":["audio-visual synchronization","benchmark","temporal evaluation","semantic evaluation","feature extraction","multimodal alignment","in-the-wild videos"],"falsifier":"A result showing that model rankings on the new temporal-only and semantic-only tasks match the rankings produced by any existing coupled audio-visual synchronization benchmark.","tokens_in":2540,"feed_emoji":"🎬","tokens_out":653,"duration_ms":14205,"temperature":0.7,"pith_summary":"The paper introduces a benchmark that isolates two distinct aspects of audio-visual alignment so each can be measured without the other influencing the score. Existing protocols mix timing checks with meaning checks and build data in ways that keep those dimensions entangled. By starting from in-the-wild videos, applying automatic filters, and adding manual checks to confirm visible sound sources, the benchmark supplies independent test sets for timing accuracy and for semantic correspondence. Five representative models are run on the resulting collection of 3,269 videos and 38,390 samples to show how the separation changes measured feature quality. The construction covers voice, music, and general sound across ten scenarios and five task variants.","feed_headline":"Benchmark splits audio-visual sync into separate timing and meaning scores","feed_subtitle":"AV-SyncBench lets researchers measure offset accuracy apart from content matching on 38k verified samples.","key_machinery":"AV-SyncBench, a dataset and evaluation protocol that maintains separate tracks for temporal alignment and semantic matching while ensuring on-screen sound sources are verified.","core_discovery":"The central claim is that audio-visual feature extraction models can be assessed for temporal consistency and semantic consistency through completely separate evaluation tracks. AV-SyncBench achieves this separation by constructing its data from in-the-wild videos that are automatically filtered and manually verified to contain identifiable on-screen sound sources, then organizing the material into distinct temporal-offset and semantic-matching tasks across Voice, Music, and Sound categories.","pith_inferences":["Designers of new models could optimize the temporal track without regard to semantic performance and vice versa.","Similar decoupling might be applied to other paired modalities such as video-text or audio-text to reveal independent failure modes.","If the separated scores prove stable, existing single-score leaderboards for audio-visual tasks could be replaced by paired leaderboards."],"forward_implications":["Feature extraction models can now receive separate scores for timing accuracy and for content correspondence.","Downstream multimodal tasks can be linked to one dimension or the other rather than to a single entangled metric.","Evaluations become possible across voice, music, and ambient sound without one category dominating the combined score.","The benchmark supplies 38,390 samples that can be used to isolate whether alignment failures stem from offset or from mismatch."],"fun_headline_variants":["AV-SyncBench separates temporal and semantic audio-visual sync checks","Benchmark enables independent timing and semantic AV alignment tests","AV-SyncBench tests temporal offsets apart from semantic content matching","Decoupled AV sync benchmark assesses timing and meaning independently"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Automatic filtering plus manual verification of in-the-wild videos produces a dataset where on-screen sound sources are reliably identified without introducing selection bias that affects the decoupled scores.","fun_headline_variants_meta":{"raw":{"variants":["AV-SyncBench separates temporal and semantic audio-visual sync checks","Benchmark enables independent timing and semantic AV alignment tests","AV-SyncBench tests temporal offsets apart from semantic content matching","Decoupled AV sync benchmark assesses timing and meaning independently"]},"model":"grok-4.3","cost_usd":0.004679,"raw_usage":{"total_tokens":2279,"prompt_tokens":600,"num_sources_used":0,"completion_tokens":64,"cost_in_usd_ticks":46787000,"prompt_tokens_details":{"text_tokens":600,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1615,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":600,"tokens_out":64,"duration_ms":12098,"temperature":1.0,"reasoning_tokens":1615,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-02T14:40:39.624452+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A result showing that model rankings on the new temporal-only and semantic-only tasks match the rankings produced by any existing coupled audio-visual synchronization benchmark.","supporting_citations":[],"review_version":1}