{"id":"17cfd29b-fa6f-4dca-b9ea-fb0e01d024ef","arxiv_id":"2411.11278","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An ImageBind-based fine-tuned model outperforms a training-free zero-shot baseline on the new OV-AVEBench, which spans 67 event classes with 21 unseen at test time.","lead":"This paper defines a new task and dataset for open-vocabulary audio-visual event localization, where models must name both seen and unseen event classes in video. It evaluates two baseline systems and shows that fine-tuning on the new benchmark improves accuracy, especially for unseen events.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning vs. training-free baselines confound model training with prediction-fusion strategy; the claimed advantage may stem from fusion, not fine-tuning.","rationale":"The reader's weakest assumption concerned label quality in the OV-AVEBench dataset, which is a real risk for a benchmark paper. However, the strongest claim singled out by the reader is the quantitative comparison between the fine-tuning and training-free baselines. That comparison is internally confounded: the two methods differ both in training/temporal modeling and in the fusion strategy used to combine audio and visual evidence. Table 4 demonstrates the fusion rule alone can swing average performance by 18.8 points, larger than the reported 11.2-point gap between the baselines. Since the training-free hard-consistency rule is not the same soft geometric mean used by the fine-tuning model, the observed improvement cannot be unambiguously attributed to fine-tuning. This is a concrete, checkable threat to the paper's central claim, more direct than the label-quality concern. I therefore focus the stress-test there while acknowledging the label-quality issue is also legitimate. The verdict remains CONDITIONAL: the paper's dataset and task formulation are valuable, but the headline baseline comparison needs the isolating experiment described above before the claim can be taken as established.","tokens_in":20822,"tokens_out":4124,"duration_ms":39960,"concrete_test":"Re-run the training-free baseline (Sec. 3.1) on OV-AVEBench test data, but replace the hard audio-visual consistency with the soft geometric-mean fusion of Eq. 2, using identical class texts (seen + unseen + 'other') and frozen ImageBind features. Compare its Avg. to the reported 46.6 (hard consistency) and 57.8 (fine-tuned). If training-free + Eq. 2 reaches or approaches 57.8, the improvement attributed to fine-tuning is confounded by fusion; if it remains near 46.6, the temporal fine-tuning is the effective factor. Also run the fine-tuned model with hard consistency to complete the 2x2 design.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central comparison (Table 2) contrasts a training-free baseline that fuses audio and visual predictions via hard top-1 consistency (Sec. 3.1) with a fine-tuning baseline that uses the geometric-mean fusion of Eq. 2 (Sec. 3.2). These differ in two variables: (i) temporal layers and fine-tuning, and (ii) the fusion rule. Table 4 shows that within the fine-tuning model, fusion choice alone swings Avg. from 39.0 (Prob-avg) to 57.8 (Sqrt), an 18.8-point effect larger than the 11.2-point training-free-vs-fine-tuning gap reported in Sec. 4.2. The hard-consistency rule used by the training-free baseline resembles a strict AND and is likely closer to Prob-avg than to Sqrt. Consequently, if the Sqrt fusion — not temporal fine-tuning — drives most of the gain, the headline claim that 'fine-tuning significantly outperforms the training-free version' is not cleanly supported by the current experimental design. An ablation isolating the fusion rule from the temporal layers is necessary to attribute the improvement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) problem, in which a model must localize audio-visual events in a video and classify them into explicit categories, including categories whose labels do not appear in the training data. To support this task, the authors build OV-AVEBench, a dataset of 24,800 YouTube videos from VGGSound, with manual segment-level labels for 67 event classes (46 seen, 21 unseen during training), and propose three metrics: accuracy, segment-level F1, and event-level F1. They evaluate two baselines: a training-free approach that uses ImageBind embeddings and checks agreement between the top audio-text and visual-text classes per segment, and a fine-tuning approach that adds lightweight temporal transformer layers and fuses audio and visual probabilities with a geometric mean. In their experiments the fine-tuning baseline achieves 57.8 average metric on the total test set versus 46.6 for the training-free baseline, and ablations examine the 'other' text, fusion rule, temporal layers, and training-data ratios.","tokens_in":21010,"tokens_out":5786,"duration_ms":56948,"significance":"If the benchmark and results hold up, OV-AVEBench is a potentially useful resource: it is roughly six times larger than the existing AVE dataset, spans 67 classes, provides segment-level labels, and defines an open-vocabulary evaluation protocol with multiple metrics. The paper also demonstrates a simple zero-shot baseline with ImageBind and that adding a small amount of fine-tuning on seen classes can improve unseen-class localization at the event level. The ablation study on the fusion rule (Table 4) is informative for the community, even though it complicates the headline comparison. The code release supports reproducibility, and the dataset, if reliable, could serve as a standardized testbed for future open-vocabulary audio-visual event localization research.","major_comments":[{"comment":"The central comparison in Table 2 between the training-free baseline (46.6 Avg.) and the fine-tuning baseline (57.8 Avg.) varies two factors at once: (i) the use of fine-tuned temporal layers and (ii) the rule for combining audio and visual predictions. The training-free model (Sec. 3.1) takes the top-1 class per modality and requires agreement, while the fine-tuning model (Sec. 3.2) uses the geometric-mean fusion of Eq. (2). Table 4 shows that replacing Sqrt with Prob-avg within the fine-tuned model drops Avg. from 57.8 to 39.0, an effect of 18.8 points, which is larger than the 11.2-point gap in Table 2. Since the hard-agreement rule used in the training-free baseline is not directly evaluated with the fine-tuned features, the improvement attributed to 'fine-tuning' could in principle be driven by the fusion rule rather than by learning temporal relations. Please add an ablation that decouples these factors, for example applying the geometric-mean fusion to the training-free model, or applying the hard-consistency rule to the features with temporal layers, and discuss the resulting attribution.","section":"§3.1, §3.2, Table 2, Table 4"},{"comment":"The OV-AVEBench benchmark rests on manual segment-level annotations, but the paper does not report any quantification of label reliability. Section 2.2 describes a two-round review with discussion, yet it gives no inter-annotator agreement (e.g., Cohen's kappa or a per-class agreement rate) and no estimate of label noise. Given that all model evaluations are computed against these labels, the benchmark's usefulness depends on their accuracy. I request that the authors provide agreement statistics on a held-out subset, describe the adjudication process in more detail, and, if possible, report the stability of the main results under a small amount of simulated label noise.","section":"§2.2"}],"minor_comments":[{"comment":"In Section 4.3, the in-text reference 'Table A8' should be 'Table 3' for the main-text ablation on the 'other' class, since Table A8 is in the supplementary material.","section":"§4.3"},{"comment":"There are typos: 'downloadeded' in Section 2.1, and the legend in Figure 2(a) appears to repeat 'musical instruments'.","section":"§2.1"},{"comment":"The sentence 'the training-free baseline model performs slightly better on the unseen test data' is ambiguous; if it means that the training-free model performs better on unseen than on seen data, it should be rewritten to avoid the apparent contradiction with Table 2, where the seen Avg. is 45.5 and the unseen Avg. is 47.0.","section":"§4.2"},{"comment":"The paper uses 'significantly outperforms' without reporting variance across runs or significance tests; please clarify whether 'significantly' means statistically tested or merely numerically higher, and report mean and standard deviation if multiple runs were conducted.","section":"§4.2"},{"comment":"The 'Best epoch' column in Table 6 is not explained; please clarify whether the best epoch is selected on the validation set and report the corresponding validation metrics.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"This is a solid benchmark contribution: the dataset, metrics, and ImageBind-based baselines are useful for the community. The main experimental claim about fine-tuning vs. training-free, however, is not cleanly supported because the comparison confounds fusion strategy with fine-tuning. The additional ablation and label-quality analysis requested in the major comments should be feasible within the scope of the manuscript, so I see this as a major revision rather than a rejection. I would also encourage the authors to make the dataset release details (e.g., annotation statistics after adjudication) more transparent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a dataset-and-benchmark paper that defines a sensible new task, builds a substantially larger annotated testbed than AVE, and provides simple baselines. The dataset and task are worth referee time. But the paper's headline claim — that fine-tuning beats the training-free baseline — is not cleanly supported, because the two baselines differ in fusion strategy as well as in training. That needs fixing before the paper can be relied on.\n\nWhat is genuinely new: the OV-AVEL formulation, with explicit category naming for unseen classes, and OV-AVEBench (24,800 videos, 67 classes, 46/21 seen/unseen split, segment-level labels, three metrics). That's a real resource. The baselines are straightforward ImageBind applications, and the paper is appropriately modest about them. The section-level ablations on the 'other' prompt, fusion rule, and temporal layers are useful.\n\nThe soft spots are the usual benchmark-paper ones but one is more serious. The comparison in Table 2 contrasts a training-free baseline using hard top-1 consistency with a fine-tuning baseline using sqrt geometric fusion (Eq. 2). Table 4 shows that within the fine-tuning model, fusion alone swings Avg. from 39.0 (Prob-avg) to 57.8 (Sqrt) — an 18.8-point effect, larger than the 11.2-point gap the paper claims shows fine-tuning's benefit. The training-free hard-consistency rule is likely closer to the low-performing Prob-avg than to Sqrt. So the reported advantage of fine-tuning is confounded with the fusion rule. The paper needs a training-free baseline using Sqrt fusion (and ideally a fine-tuning baseline using top-1 or Prob-avg) to attribute the gain. The temporal-vs-linear ablation (Table 5) shows temporal layers matter, but it's inside the fine-tuning setup with Sqrt fixed.\n\nAlso: no inter-annotator agreement is reported for the crowdsourced labels, and no variance/error bars across runs. And the paper doesn't check whether ImageBind's pretraining overlaps with VGGSound/YouTube videos used here — relevant for zero-shot claims. These are not fatal, but for a benchmark they should be addressed.\n\nWho this is for: anyone working on audio-visual event localization or zero-shot multimodal recognition. The benchmark has the potential to become a standard testbed. It deserves a serious referee, and I'd engage with it, but the baseline comparison and annotation-quality reporting need work before the central claims are taken at face value.\n\nRecommendation: send to peer review, but expect the authors to disentangle fusion from fine-tuning and to report label reliability.","headline":"New benchmark and task worth engaging, but the headline fine-tuning advantage is confounded by the fusion rule.","tokens_in":21543,"tokens_out":2831,"would_cite":true,"duration_ms":25284,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces open-vocabulary audio-visual event localization, with a 24,800-video benchmark and baselines showing that fine-tuning beats zero-shot on both seen and unseen event classes.","keywords":["open-vocabulary audio-visual event localization","OV-AVEBench dataset","temporal event localization","audio-visual correspondence","zero-shot multimodal embedding","fine-tuning baseline","segment-level annotation","benchmark metrics"],"falsifier":"Re-annotate a random sample of test segments with two independent expert annotators and measure per-segment label agreement; if agreement is far below the 46.6-to-57.8 difference between the baselines, the central comparison could be an artifact of noisy ground truth.","tokens_in":20628,"feed_emoji":"🎬","tokens_out":5496,"duration_ms":48213,"temperature":0.7,"pith_summary":"This paper introduces a new task, Open-Vocabulary Audio-Visual Event Localization (OV-AVEL): given a 10-second video, a model must mark which one-second segments contain an event that is both audible and visible, and must name the event category even when that category never appeared in training. To support the task, the authors build OV-AVEBench, a dataset of 24,800 videos covering 67 real-life event classes, with 46 classes seen in training and 21 held out as unseen, all manually annotated segment by segment. They also define three evaluation metrics: accuracy, segment-level F1, and event-level F1. The paper argues that a simple fine-tuning baseline, which adds lightweight temporal layers on top of a frozen joint audio-visual-text embedding model, substantially outperforms a training-free zero-shot baseline on both seen and unseen test data, reaching a 57.8 average metric against 46.6. If the result holds, it establishes that open-vocabulary temporal event localization is feasible and gives the community a benchmark to build on.","feed_headline":"Open-vocabulary test lets models name unseen video events","feed_subtitle":"A 24,800-video benchmark splits 67 event classes into seen and unseen, and fine-tuning lifts the average score to 57.8.","key_machinery":"The load-bearing object is OV-AVEBench, with its fixed split of 46 seen and 21 unseen classes, its manual segment-level labels, and its three metrics. The method-side mechanism is a joint multimodal embedding: a pretrained model that maps audio, image, and text into one shared space. The training-free baseline computes per-segment cosine similarities between audio features and all candidate class texts, and between visual features and the same texts, then labels a segment as an event only if both modalities pick the same category. The fine-tuning baseline inserts a stack of Transformer layers (one layer works best) after the frozen audio and image encoders to model temporal context, fuses modalities through the geometric mean $\\sqrt{S'_{ae}\\odot S'_{ve}}$ of audio-text and visual-text probabilities, and adds a special \"other\" text to absorb background and out-of-list events. The \"other\" text alone is worth a 10.8-point average gain, and the geometric-mean fusion outperforms simple averaging or feature averaging.","core_discovery":"The central claim is that OV-AVEL is a tractable problem and that OV-AVEBench is a usable testbed for it. The authors report that a training-free baseline, which picks the highest audio-text and visual-text cosine similarity from a joint embedding space and requires the two modalities to agree, reaches 46.6 average on the full test set. Adding one learnable transformer temporal layer per modality on frozen audio and image encoders, then fine-tuning on 13,182 training videos whose classes are all seen, raises the average to 57.8, with gains on both seen (62.9 versus 45.5) and unseen (55.8 versus 47.0) test data. The paper interprets the gains on unseen classes as evidence that temporal-relation learning transfers across categories, and the event-level F1 jump from 34.0 to 49.5 as better temporal boundary localization. The benchmark also shows that closed-set models, which perform well on seen classes, collapse on unseen classes, motivating the open-vocabulary formulation.","pith_inferences":["Beyond the paper, the same recipe of frozen joint embeddings plus a small trainable temporal head could transfer to longer videos or to weakly labeled web data, where per-segment manual labels are unavailable.","Also beyond the paper, the finding that 25% of training data nearly matches 100% suggests the bottleneck may be the quality of the joint embedding space and the temporal prior, not the size of the seen-class label set; varying the number of seen classes would be a sharper test.","Because the paper reports no inter-annotator agreement statistics, an editorial next step is to add a small expert-relabeled subset to OV-AVEBench and measure how much of the 11.2-point fine-tuning gain survives label noise."],"forward_implications":["If OV-AVEBench is adopted, closed-set AVEL models can be re-evaluated for their unseen-class behavior rather than only their overall accuracy.","Open-vocabulary localization can be bootstrapped without retraining the frozen multimodal encoder; adding temporal layers and fine-tuning on seen data is enough to improve unseen-class recognition.","Event-level F1 becomes a meaningful third metric, penalizing poor temporal boundaries that segment accuracy alone misses.","The benchmark's roughly 3:7 seen-to-unseen test ratio forces models to generalize rather than memorize, and closed-set methods land in the 39-42 average range on total data.","Fine-tuning benefits unseen classes mainly by improving temporal boundary localization, since the largest gain appears in event-level F1."],"supporting_citations":[{"why":"Supplies the joint audio-visual-text embedding space that both baselines rely on.","marker":"[11]"},{"why":"Defines the original closed-set AVEL task and dataset that this work extends and compares against.","marker":"[39]"},{"why":"VGGSound is the source of the videos and category tags from which OV-AVEBench is built.","marker":"[4]"},{"why":"Provides the zero-shot text-based classification paradigm that the training-free baseline imitates.","marker":"[37]"},{"why":"Supplies the F1-style evaluation metrics adapted here to segment-level and event-level scoring.","marker":"[40]"},{"why":"Prior open-set AVEL work that this paper contrasts with, since it only predicts an unknown class rather than explicit unseen categories.","marker":"[49]"}],"fun_headline_variants":["Unseen events get names in new AV benchmark","AVEL goes open-vocabulary: models name unseen events","First open-vocabulary benchmark for audio-visual events","Open-vocabulary AVEL: localize and name unseen events","New dataset lets AVEL models predict unseen event classes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The crowd-sourced segment labels in OV-AVEBench are accurate enough to ground every comparison, but the paper reports no agreement or quality statistics for those labels.","fun_headline_variants_meta":{"raw":{"variants":["Unseen events get names in new AV benchmark","AVEL goes open-vocabulary: models name unseen events","First open-vocabulary benchmark for audio-visual events","Open-vocabulary AVEL: localize and name unseen events","New dataset lets AVEL models predict unseen event classes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000733,"raw_usage":{"total_tokens":3341,"prompt_tokens":1068,"completion_tokens":2273,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":2192}},"tokens_in":684,"tokens_out":2273,"duration_ms":18125,"temperature":1.0,"reasoning_tokens":2192,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:42:20.351426+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of test segments with two independent expert annotators and measure per-segment label agreement; if agreement is far below the 46.6-to-57.8 difference between the baselines, the central comparison could be an artifact of noisy ground truth.","supporting_citations":[{"cited_title":"Imagebind: One embedding space to bind them all","cited_arxiv_id":null,"evidence_quote":"Supplies the joint audio-visual-text embedding space that both baselines rely on."},{"cited_title":"Audio-visual event localization in unconstrained videos","cited_arxiv_id":null,"evidence_quote":"Defines the original closed-set AVEL task and dataset that this work extends and compares against."},{"cited_title":"VGGSound: A large-scale audio-visual dataset","cited_arxiv_id":null,"evidence_quote":"VGGSound is the source of the videos and category tags from which OV-AVEBench is built."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Provides the zero-shot text-based classification paradigm that the training-free baseline imitates."},{"cited_title":"Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing","cited_arxiv_id":null,"evidence_quote":"Supplies the F1-style evaluation metrics adapted here to segment-level and event-level scoring."},{"cited_title":"Ope- nA VE: Moving towards open set audio-visual event localiza- tion","cited_arxiv_id":null,"evidence_quote":"Prior open-set AVEL work that this paper contrasts with, since it only predicts an unknown class rather than explicit unseen categories."}],"review_version":1}