{"id":"f6200d17-96f2-4417-8bd1-63ecf3bc7413","arxiv_id":"2505.09615","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"UWAV generates temporally coherent, uncertainty-weighted segment-level pseudo-labels with a transformer pre-trained on a larger dataset, then uses them with mixup and class rebalancing to train an audio-visual video parser, improving state-of-the-art results on the LLP and AVE benchmarks.","lead":"UWAV is a method for recognizing which events (such as a guitar, crowd, or lawn mower) appear in each second of a video and whether they are visible, audible, or both, trained with only cheap video-level labels. It adds temporal context, uncertainty-weighted pseudo-labels, mixup regularization, and class rebalancing to outperform previous audio-visual video parsing methods on standard benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Joint-label-only pre-training leaves unimodal pseudo-labels unconstrained; audio pseudo-labels are already less accurate than VALOR, so the 'better pseudo-labels' mechanism is not established for the audio stream.","rationale":"The reader's weakest assumption pinpoints the joint-label-only pre-training and cross-vocabulary generalization. My stress-test agrees and makes the concern more concrete: the Eq. 9 loss does not constrain the modality split, and Table 3 already shows the audio pseudo-labels are worse than VALOR's. This does not refute the empirical SOTA claim, because the final model could still benefit from uncertainty weighting, mixup, and rebalancing even if the pseudo-label story is only partially true. However, it does mean the central causal claim—'temporally coherent pseudo-labels improve training'—is not uniformly supported, and the unspecified threshold selection leaves a reproducibility gap. The reader's CONDITIONAL verdict is the right one: accept only if the authors supply the missing threshold details and demonstrate that the pseudo-label advantage holds per modality. No verdict change is needed.","tokens_in":17707,"tokens_out":4429,"duration_ms":51525,"concrete_test":"On the LLP validation/test set, compute per-modality and per-event-type pseudo-label accuracy/F1 separately for A-only, V-only, and AV segments, comparing (i) UWAV's pre-trained generator, (ii) a per-segment CLIP/CLAP zero-shot baseline without the temporal transformer, and (iii) a variant pre-trained with separate audio and visual supervision (or with an auxiliary unimodal loss) if such labels can be obtained. If UWAV's audio pseudo-labels are not more accurate than the per-segment CLAP baseline on A-only and AV segments, the claimed 'more accurate pseudo-labels' mechanism fails for audio and the SOTA gain must be attributed to the other training components. Additionally, report the exact procedure for setting θv and θa, and re-run the main LLP experiment with thresholds fixed without using validation segment labels to rule out threshold tuning as the source of improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The pseudo-label generator is pre-trained only with the BCE loss in Eq. 9 on the product p_v ⊙ p_a against joint audio-visual labels. This objective is invariant to how probability mass is split between the two modalities: for a joint-positive segment, increasing p_v while decreasing p_a can keep the product roughly unchanged. Consequently, the separate visual and audio probabilities are only weakly constrained, and this weakness is compounded when the generator is applied to target classes whose text embeddings were not seen during pre-training. The central difficulty of AVVP is precisely unimodal (A-only, V-only) and temporally misaligned events, and those are exactly the pseudo-labels that the pre-training objective never supervises. Table 3 gives direct evidence: UWAV's audio segment-level pseudo-label F-score is 78.4, below VALOR's 80.5, so the paper's statement that UWAV generates 'more accurate segment-level pseudo-labels' is already false for the audio stream. The final LLP SOTA may still be real, but the causal story that temporally coherent pseudo-labels drive the improvement is not supported for audio; gains could come from uncertainty weighting, mixup, class rebalancing, or from the class-wise thresholds θv, θa in Eq. 10, which are described as 'pre-defined' but never specified. If those thresholds are tuned on validation segment labels, the pseudo-label pipeline is not purely weakly supervised.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"UWAV is a two-stage weakly-supervised audio-visual video parsing (AVVP) method. In Stage 1, per-modality transformer pseudo-label generators are pre-trained on the supervised UnAV dataset using a BCE loss on the product of visual and audio probabilities, then transferred to the LLP and AVE datasets by replacing the class text embeddings with CLIP/CLAP encodings of the target classes and thresholding with class-wise constants. In Stage 2, a HAN inference module is trained with binary and soft pseudo-labels, an uncertainty-weighted feature mixup loss, a class-balanced reweighting of the soft-label loss, and the standard video-level BCE loss. The paper reports state-of-the-art results on LLP (e.g., segment-level Type@AV 65.9, event-level Type@AV 60.9) and on AVE (80.6% accuracy), plus ablations, hyperparameter sensitivity, scalability, and compute-time analyses.","tokens_in":18003,"tokens_out":7393,"duration_ms":72029,"significance":"If the results are reproducible, the paper is a meaningful empirical contribution. It combines several sensible ideas for weakly-supervised AVVP -- temporally coherent transformer-based pseudo-labeling, confidence-weighted soft targets, mixup regularization, and class-frequency rebalancing -- and the full system improves on strong recent baselines on two datasets. The supplementary is thorough, including code, runtime, sensitivity to alpha and W, scalability to fewer training videos/classes, and full-metric ablations; these strengthen the manuscript. The main caveats are that the pseudo-label pretraining objective only supervises the joint audio-visual product, the class-wise thresholds are undisclosed, and Eq. (17) appears to reweight at the video level rather than the segment level, so the mechanism behind the reported gains needs clarification.","major_comments":[{"comment":"The pre-training objective L_temp = BCE(p̂^av, y^av) supervises only the element-wise product p̂^v ⊙ p̂^a. This objective is invariant to how probability mass is split across modalities, so the unimodal visual and audio probabilities used to generate pseudo-labels in Eq. (10) are only weakly constrained; the AVVP cases that matter most -- audio-only, visual-only, and temporally misaligned events -- are exactly the ones that this objective never labels directly. The concern is not hypothetical: Table 3 shows the audio pseudo-label F-score is 78.4, below VALOR's 80.5, while the text claims 'more accurate segment-level pseudo-labels' (Section 5.2). Please add direct evidence that the unimodal streams are reliable (e.g., an auxiliary unimodal supervision term, or per-modality pseudo-label diagnostics), or restrict the claim to the overall Type@AV accuracy and explain why the audio inference stream improves despite worse audio pseudo-labels.","section":"§4.1, Eq. (9)"},{"comment":"The class-wise thresholds θv and θa are called 'pre-defined' but their values and selection procedure are never given, in the main text or the supplementary. This is a free parameter of the pseudo-label pipeline and directly controls the binary labels in Eq. (10). Please report all θ values and state whether they were chosen using segment-level validation labels. If they are tuned on ground-truth segments, the procedure is not purely weakly supervised and should be described as such; if not, give the criterion used.","section":"§4.1, Eq. (10)"},{"comment":"The reweighting in L_w-soft multiplies the BCE terms by the video-level label vector y and (1-y), not by segment-level pseudo-labels. For a class present somewhere in a video, Eq. (17) up-weights the loss of every segment regardless of that segment's pseudo-label; for a class absent from the video, every segment is down-weighted. This contradicts the prose, which says the loss from positive/negative events in the pseudo-labels is rebalanced. Please either correct the formula so the mask is per-segment (e.g., using ŷ or p̂ at the segment level) or justify the video-level mask explicitly; as written, the ablation results in Table 4 cannot be attributed to the intended class-balanced mechanism.","section":"§4.4, Eq. (17)"},{"comment":"No error bars or multiple-seed results are reported. Several decisive margins are small -- the AVE gain over VALOR is 0.2% (80.6 vs 80.4), and the LLP gains over PPL are 1.1 and 1.0 for the two Type@AV metrics. Without variance estimates, the state-of-the-art claim is not substantiated. Please report mean and standard deviation over at least three runs for the main tables.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"The caption says 'Accuracy' but the entries are F-scores; use 'F-score' or report accuracy separately.","section":"Table 3"},{"comment":"The text refers to Table A7b for less training data and Table A7a for fewer classes, but the table labels are swapped; please correct.","section":"Supplementary Section 13"},{"comment":"The term 'uncertainty-weighted' might better be described as 'confidence-weighted', since Eq. (12) uses confidence as the soft target; consider clarifying that lower-confidence targets produce smaller gradients, which is the mechanism.","section":"§4.2"},{"comment":"Please define the dimensions of λ and clarify whether the same λ is shared across modalities or sampled independently per modality for the mixup.","section":"Eqs. (14)-(15)"},{"comment":"The claim of outperforming VALOR/CoLeaf/LEAP 'by up to 3%' should specify the exact baseline and metric, since the margins differ across rows of Table 1.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"This is a competent empirical paper with strong reproducibility materials. The main issues are fixable with disclosure of thresholds, correction or justification of Eq. (17), and error-bar reporting; I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nUWAV is a solid, honestly reported SOTA entry for weakly-supervised audio-visual video parsing, but it needs a bit more disclosure before I'd fully trust the mechanism. The paper builds a plausible recipe: pre-train a temporal transformer on the large UniAV dataset to generate segment-level pseudo-labels from CLIP/CLAP features, then train the standard HAN inference model with uncertainty-weighted soft pseudo-labels, feature mixup, and class reweighting. The experiments are honestly reported, with a limitations section, compute times, and sensitivity curves. The main LLP numbers are genuinely state-of-the-art: segment-level Type@AV 65.9 vs 64.8 for the previous best, and visual-event F1 jumps by 3 points. The AVE gain is small (0.2) but consistent.\n\nThe evaluation is not circular: test labels are held out, pseudo-labels come from a model pre-trained on a different dataset, and hyperparameters are chosen on validation. Code is linked. That's real evidence.\n\nThe soft spots are addressable but real. Most important: Table 3 shows the audio pseudo-labels are less accurate than VALOR's (78.4 vs 80.5), so the claim that temporally coherent pseudo-labels are 'more accurate' doesn't hold for audio. The overall Type@AV improves only because visual and audio-visual labels improve a lot. That doesn't kill the final result, but it means the paper's central mechanism story is only half-supported. The pre-training objective on the product of audio and visual probabilities plausibly leaves unimodal streams underconstrained, which would explain the audio degradation. The authors should ablate their generator per modality and discuss this.\n\nSecond, the class-wise thresholds θv and θa in Eq. 10 are never specified. 'Pre-defined' isn't enough. Were they tuned on validation segment labels? If so, the pipeline isn't purely weakly supervised. They need to disclose this.\n\nThird, no error bars or multiple seeds. Given the AVE gain is 0.2%, one seed is not enough.\n\nFourth, Eq. 17 masks by video-level labels rather than segment-level pseudo-labels; it's not clearly wrong, but the text should explain why.\n\nThis paper is for researchers in weakly-supervised audio-visual parsing, a niche but active area. The method is practical, the writing is clear, and the results are reproducible in principle. It deserves peer review; with thresholds disclosed and a bit more honesty about audio, I'd accept it.\n\nRecommendation: send to peer review.","headline":"UWAV is a solid, honestly reported SOTA entry for weakly-supervised AV video parsing, but under-disclosed thresholds and a mechanism claim that doesn't hold for the audio stream keep it from being fully convincing.","tokens_in":18594,"tokens_out":3528,"would_cite":true,"duration_ms":31662,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Temporal-aware, uncertainty-weighted pseudo-labels push weakly-supervised audio-visual video parsing past prior state of the art.","keywords":["audio-visual video parsing","weakly supervised learning","pseudo-label generation","temporal transformers","uncertainty weighting","feature mixup","video-level labels","event localization"],"falsifier":"On the LLP test set, compare thresholded pseudo-labels from UWAV's frozen generators against per-segment ground truth, separated into audio-only, visual-only, and audio-visual segments; if the generators' accuracy on unimodal segments is no better than VALOR's segment-wise CLIP/CLAP pseudo-labels, then the claimed benefit from temporal coherence and cross-modal pretraining is not actually carrying the result, and downstream gains would have to come from the training-side components instead.","tokens_in":17486,"feed_emoji":"🎬","tokens_out":5889,"duration_ms":52910,"temperature":0.7,"pith_summary":"This paper tackles weakly supervised audio-visual video parsing (AVVP), where training data supply only video-level event labels and a model must localize events per one-second segment in audio, video, or both. UWAV argues that prior pseudo-label generators are limited because they label each segment in isolation, ignoring inter-segment context, and because they bias toward the majority \"event absent\" class. To fix this, UWAV pretrains transformer-based pseudo-label generators on a large supervised audio-visual event dataset, using CLIP/CLAP text embeddings so the generators can later be applied to new event vocabularies. On the target dataset, it trains the standard HAN inference module with temporally coherent pseudo-labels weighted by the generator's confidence, a class-rebalancing loss, and feature mixup. The paper reports state-of-the-art F-scores on LLP and top accuracy on AVE, with the largest relative gains on visual-only events.","feed_headline":"Temporal-aware pseudo-labels lift weakly supervised video parsing","feed_subtitle":"A frozen temporally pretrained generator plus confidence-weighted training beats state-of-the-art on LLP and AVE.","key_machinery":"The load-bearing object is the temporally coherent pseudo-label generator: two transformer encoders (visual and audio) pretrained on UniAV and then frozen. A segment's center frame goes through CLIP's image encoder; the audio waveform goes through CLAP's audio encoder; each transformer attends across segments via self-attention, and the resulting features are matched to class text embeddings (\"A photo of ...\" / \"This is the sound of ...\"). The pretraining loss is BCE on the product of the audio and visual probabilities, which is what makes the two streams share burden while training on audio-visual-only labels. The uncertainty mechanism is a soft target $\\hat p_t = \\mathrm{Sigmoid}(\\hat z_t - \\theta) \\odot y$: distance from a per-class threshold encodes confidence, and video-level labels mask out classes absent from the video. The mixup regularization interpolates segment features and the same soft targets, giving the model extra training points whose labels are weighted by generator confidence.","core_discovery":"The central claim is that the bottleneck in weakly supervised AVVP is pseudo-label quality, and that quality improves by generating pseudo-labels with a temporal model rather than frame- or segment-wise foundation model scoring. UWAV pretrains a pair of small transformers, one for video and one for audio, on UniAV, where supervision is applied to the elementwise product of audio and visual probabilities because labels are audio-visual; this forces each stream to learn from the other during pretraining. At inference on the target dataset, the frozen transformers encode all segments jointly, so predicted labels for a segment reflect neighboring context; these are combined with video-level ground truth and converted into soft targets by a sigmoid of the margin to a class-wise threshold. Training the HAN inference module with those uncertainty-weighted targets, class-frequency reweighting, and uncertainty-weighted feature mixup yields the reported state-of-the-art numbers, especially for visual events. In the paper's own characterization, pseudo-labels improve by up to 6% Type@AV F-score over prior pseudo-label generators, and the downstream gains follow from that.","pith_inferences":["A direct test of the transfer claim would be to keep the generator frozen and evaluate it on datasets whose event names are rarer in CLIP/CLAP training; if text-embedding quality limits pseudo-labels, performance should fall even with identically trained temporal transformers.","The product-supervision pretraining (Eq. 9) means each stream is supervised only through both-stream agreement; one would expect the separate visual and audio pseudo-labels to be less reliable for events that are strictly unimodal, which is exactly the case LLP's A/V metrics measure.","The same uncertainty-weighted pseudo-label recipe could transfer to other weakly supervised temporal localization tasks, such as video-level action detection, where segment-level pseudo-labels are generated by a frozen temporal model and reweighted by confidence.","A stronger claim implicit in the results is that temporal coherence matters more than foundation-model raw scoring; a clean comparison would hold the backbone fixed and sweep transformer depth, predicting Type@AV rises with context length."],"forward_implications":["On LLP, the method reports segment-level Type@AV 65.9 and event-level Type@AV 60.9, beating the closest prior PPL by about one point in each, with the visual-event F-score more than three points higher.","On AVE, the method reaches 80.6% segment accuracy against 80.4% for VALOR, so the recipe transfers to a smaller dataset with a different event vocabulary.","Pseudo-labels generated by the temporal transformers score up to 6% higher Type@AV F-score than VALOR's segment-wise labels, which is the paper's explanation for the downstream gains.","The ablation shows uncertainty weighting contributes about 2 points, class reweighting about 1 point, and feature mixup about 0.8 point of segment-level Type@AV F-score.","Because the generator is frozen and text-conditioned, the same pretrained module can be pointed at new event sets by swapping text embeddings, which the AVE experiment demonstrates."],"supporting_citations":[{"why":"UniAV dataset that supplies large-scale supervised audio-visual event labels for pretraining the temporal pseudo-label generators.","marker":"[14]"},{"why":"Defines the AVVP task, the HAN inference model, LLP dataset, and evaluation protocol that all comparisons build on.","marker":"[38]"},{"why":"The AVE dataset and its segment accuracy metric used to test generalizability.","marker":"[37]"},{"why":"VALOR, the closest CLIP/CLAP segment-wise pseudo-label baseline that UWAV improves on and ablates against.","marker":"[20]"},{"why":"PPL, the prototype-based pseudo-labeling competitor that provides the nearest prior state-of-the-art numbers.","marker":"[29]"},{"why":"CLIP supplies the image and text encoders whose text-aligned visual features are fed to the visual transformer.","marker":"[31]"},{"why":"CLAP supplies the audio and text encoders producing text-aligned audio features for the audio transformer.","marker":"[43]"},{"why":"LSLD, another language-perspective pseudo-label baseline with CLIP/CLAP whose approach is contrasted with temporal coherence.","marker":"[10]"}],"fun_headline_variants":["Uncertainty-weighted pseudo-labels refine weak AV parsing","Uncertainty-weighting sharpens pseudo-labels for AV parsing","Confidence-weighted pseudo-labels boost weak AV video parsing","Temporal and uncertainty-aware pseudo-labels for weak video parsing","Uncertainty-weighted mixup and temporal pseudo-labels for weak AV parsing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on the assumption that a transformer trained on the large UniAV dataset, with labels that only say whether an event is present in both audio and video, learns a text-aligned feature space that remains accurate enough for the smaller LLP and AVE datasets and for event names it never saw during pretraining.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty-weighted pseudo-labels refine weak AV parsing","Uncertainty-weighting sharpens pseudo-labels for AV parsing","Confidence-weighted pseudo-labels boost weak AV video parsing","Temporal and uncertainty-aware pseudo-labels for weak video parsing","Uncertainty-weighted mixup and temporal pseudo-labels for weak AV parsing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002347,"raw_usage":{"total_tokens":9072,"prompt_tokens":1001,"completion_tokens":8071,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":7983}},"tokens_in":617,"tokens_out":8071,"duration_ms":54242,"temperature":1.0,"reasoning_tokens":7983,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:27:14.853438+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the LLP test set, compare thresholded pseudo-labels from UWAV's frozen generators against per-segment ground truth, separated into audio-only, visual-only, and audio-visual segments; if the generators' accuracy on unimodal segments is no better than VALOR's segment-wise CLIP/CLAP pseudo-labels, then the claimed benefit from temporal coherence and cross-modal pretraining is not actually carrying the result, and downstream gains would have to come from the training-side components instead.","supporting_citations":[{"cited_title":"Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline","cited_arxiv_id":null,"evidence_quote":"UniAV dataset that supplies large-scale supervised audio-visual event labels for pretraining the temporal pseudo-label generators."},{"cited_title":"Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing","cited_arxiv_id":null,"evidence_quote":"Defines the AVVP task, the HAN inference model, LLP dataset, and evaluation protocol that all comparisons build on."},{"cited_title":"Audio-visual event localization in unconstrained videos","cited_arxiv_id":null,"evidence_quote":"The AVE dataset and its segment accuracy metric used to test generalizability."},{"cited_title":"Modality-independent teachers meet weakly-supervised audio-visual event parser","cited_arxiv_id":null,"evidence_quote":"VALOR, the closest CLIP/CLAP segment-wise pseudo-label baseline that UWAV improves on and ablates against."},{"cited_title":"Weakly-supervised audio-visual video parsing with prototype-based pseudo-labeling","cited_arxiv_id":null,"evidence_quote":"PPL, the prototype-based pseudo-labeling competitor that provides the nearest prior state-of-the-art numbers."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"CLIP supplies the image and text encoders whose text-aligned visual features are fed to the visual transformer."},{"cited_title":"Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation","cited_arxiv_id":null,"evidence_quote":"CLAP supplies the audio and text encoders producing text-aligned audio features for the audio transformer."},{"cited_title":"Revisit weakly-supervised audio-visual video parsing from the lan- guage perspective","cited_arxiv_id":null,"evidence_quote":"LSLD, another language-perspective pseudo-label baseline with CLIP/CLAP whose approach is contrasted with temporal coherence."}],"review_version":1}