{"id":"fc234597-1069-483f-a4de-b6dc42feab35","arxiv_id":"2607.06892","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":5,"one_line_summary":"UBG-Net improves noisy audio-visual speech recognition by using data-level uncertainty to guide model-level Bayesian fusion and a hierarchical voting strategy for decoding.","lead":"This paper introduces a method for audio-visual speech recognition that uses two types of uncertainty (data noise and model ambiguity) to better fuse audio and visual signals in noisy environments. A smart generalist might read it to understand how probabilistic AI models can be made more robust for real-world applications like transcribing speech in crowded rooms.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Improvements over baseline are small (0.2% absolute avg on LRS2) with no reported variance despite stochastic MC inference; statistical significance of the superiority claim is unverified.","rationale":"The reader correctly identified a valid concern about the uncertainty coupling mechanism, but I find the more immediately load-bearing issue to be the statistical fragility of the empirical claims. The improvements are small enough (0.2% absolute on LRS2) that without variance estimates — which the authors already have from their 10 runs — the claim of 'overall superiority' is not adequately substantiated. This is not a fatal flaw: the data to resolve it likely already exists in the authors' logs. The mechanism concern (does learned variance track signal noise?) is also valid and would be worth investigating via correlation analysis, but the ablation results do provide indirect evidence that the components contribute. The verdict remains CONDITIONAL: the paper presents a reasonable and novel approach, but the empirical case needs to be strengthened with variance reporting. If the 10-run variance shows the improvements are statistically significant, the CONDITIONAL could move toward ACCEPT. If not, the headline claim needs to be scaled back. I partially agree with the reader: the mechanism verification gap is real, but the statistical significance gap is more directly threatening to the central claim as stated.","tokens_in":9315,"tokens_out":2846,"duration_ms":82922,"concrete_test":"Report the per-run WER values from the 10 independent runs already conducted (footnote 3) for both UBG-Net and the baseline on at least the LRS2 average and AVCocktail Gold settings. Compute standard deviations and run a paired significance test (e.g., Wilcoxon signed-rank). If the 95% confidence interval of the improvement on LRS2 average includes zero, the 'superiority' claim on that benchmark is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is 'overall superiority of UBG-Net compared to SOTA baselines.' The reported improvements are small in absolute terms: on LRS2 (Table 1), the average WER improves from 4.1% to 3.9% (0.2% absolute), and several individual conditions show no improvement or slight regression (e.g., 2 interfering speakers at 5 dB: 3.2→3.2; clean: 2.1→2.2). On AVCocktail (Table 2), Gold segmentation improves by 0.7% absolute (22.6→21.9). The ablation differences are even smaller: removing epistemic uncertainty changes Gold WER by 0.4% (21.9→22.3), removing aleatoric by 0.2% (21.9→22.1). Critically, the paper acknowledges in footnote 3 that 'all reported results are averaged over 10 independent runs to ensure statistical reliability' due to the stochastic nature of MC sampling inference — yet no standard deviations, confidence intervals, or significance tests are reported anywhere. For improvements of this magnitude under a stochastic inference procedure, the absence of variance estimates makes it impossible to determine whether the observed gains exceed run-to-run noise. This directly undermines the load-bearing claim of superiority. The reader's concern about whether learned variance correlates with actual signal noise is valid for the mechanism, but the more immediate issue is that the empirical evidence for the headline claim itself may be insufficiently supported without variance reporting.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The manuscript proposes UBG-Net, an uncertainty-aware Bayesian gating framework for robust audio-visual speech recognition (AVSR). The core methodological contribution is the Modality Uncertainty-aware Bayesian Fusion (MUBF) mechanism, which models aleatoric uncertainty (data noise) via modality-dependent MLP encoders and injects the resulting distribution parameters as a context vector into a Bayesian gating network that models epistemic uncertainty (model parameter uncertainty). A secondary contribution is the Distribution Uncertainty-aware Hierarchical Voting (DUHV) strategy, which selects the final transcript from Monte Carlo (MC) samples by majority voting, using inference scores as a tie-breaker. Experiments are conducted on the AVCocktail and noise-augmented LRS2 datasets, with ablation studies isolating the contributions of the aleatoric and epistemic components and the voting strategy.","tokens_in":9578,"tokens_out":1370,"duration_ms":165251,"significance":"The paper addresses a practically important problem: robust AVSR under signal corruption and distribution shifts. The conceptual framing—using aleatoric uncertainty as a contextual cue for epistemic uncertainty rather than treating them independently—is reasonable and not standard in the AVSR literature. The method is built on established tools (variational inference, reparameterization, AV-HuBERT backbone), and the ablation design (Table 2) is structured to isolate individual components. The inclusion of Whisper and Qwen3-Omni as reference points contextualizes task difficulty. The work is a reasonable engineering contribution to robust multimodal speech recognition.","major_comments":[{"comment":"Section 4.1 and Tables 1–2: The manuscript states in footnote 3 that 'all reported results are averaged over 10 independent runs to ensure statistical reliability,' yet no standard deviations, confidence intervals, or significance tests are reported anywhere. The improvements over the baseline are small in absolute terms: on LRS2 (Table 1), the average WER improves from 4.1% to 3.9% (0.2% absolute), and several individual conditions show no improvement or slight regression (e.g., 2 interfering speakers at 5 dB: 3.2→3.2; clean: 2.1→2.2). On AVCocktail (Table 2), Gold segmentation improves by 0.7% absolute (22.6→21.9). The ablation differences are even smaller: removing epistemic uncertainty changes Gold WER by 0.4% (21.9→22.3), removing aleatoric by 0.2% (21.9→22.1). Given that the inference procedure is explicitly stochastic (MC sampling with K=5), the absence of variance estimates makes","section":null},{"comment":"Section 2.1.1, Eq. (1): The central claim that aleatoric uncertainty serves as a 'signal-quality indicator' is not empirically validated. The learned variance (σ_t^m)^2 is produced by a simple MLP encoder and constrained by a KL divergence toward a standard Gaussian prior (Eq. 7, L_KL-Feat). There is no analysis showing that the predicted variance correlates with actual signal corruption (e.g., SNR, noise level, or silent face segments). Without such evidence, it remains unclear whether the gating network in Eq. (4) is modulating features based on genuine signal quality or on a spurious learned signal. A scatter plot or correlation analysis between predicted variance and known noise levels would substantially strengthen the load-bearing claim that aleatoric uncertainty guides epistemic modeling.","section":null},{"comment":"Section 2.1.2, Eq. (5): The fusion formula H_t^final = H_t^deep + g_t ⊙ Concat(h_t^v, h_t^a) adds the gated multimodal features to the deep representation H_t^deep from the pre-trained backbone. However, the dimensionality and origin of H_t^deep relative to the concatenated features h_t^v and h_t^a are not clearly specified. If H_t^deep is the output of the AV-HuBERT encoder (which already fuses audio and visual streams), the relationship between this already-fused representation and the separately computed h_t^v, h_t^a needs clarification. This affects the interpretability of the gating mechanism: is the BGN re-weighting features that the backbone has already fused?","section":null}],"minor_comments":[{"comment":"Table 1: The 'Avg' column header is rendered as 'A vg' with a space. Also, the SNR column header '∞' (clean condition) is placed under the SNR grouping but the column appears to be the last data column; alignment is slightly confusing.","section":null},{"comment":"Section 3.2: The loss coefficients β_1 and β_2 are warmed up to target values of 1e-7 and 1e-4, respectively. The value of β_1 = 1e-7 is extremely small; it would help to briefly justify why such a small weight is appropriate for the KL term on the Bayesian weights.","section":null},{"comment":"Figure 1: The figure is dense and the text labels are small. The distinction between the training path and inference path could be made clearer, perhaps by using color coding or separating the two paths into sub-figures.","section":null},{"comment":"Section 4.2, Figure 2: The x-axis labels '0', '1', '≥2' are described in the caption but the figure itself would benefit from clearer tick labels (e.g., 'Det', 'K=1', 'K=2', etc.) for readability.","section":null},{"comment":"Reference [8] (Nguyen et al., Interspeech 2025) and [30] (Qwen3-Omni, arXiv 2025) are cited; the Qwen3-Omni reference lists the arXiv ID as 2509.17765, which appears to be a future-dated preprint. Please verify the citation accuracy.","section":null}],"recommendation":"major_revision","confidential_remarks":"The core methodological idea is reasonable and the ablation structure is appropriate, but the empirical evidence for the headline claim of 'overall superiority' is thin given the small absolute improvements and the complete absence of variance estimates despite an explicitly stochastic inference procedure. The mechanism validation gap (no evidence that learned variance tracks actual noise) is also a concern. If the authors can provide variance estimates showing statistical significance and a correlation analysis for the aleatoric uncertainty, the paper could meet the bar for acceptance. The citation of a 2025 Interspeech paper and a future-dated arXiv preprint is worth noting but likely reflects the review timeline."},"author_rebuttal":null,"desk_editor":{"model":"glm-5.2","letter":"Quick read on UBG-Net (arXiv:2607.06892): the core idea is genuinely new and the engineering is competent, but the empirical case for superiority is thinner than it looks because the gains are small and no variance is reported despite stochastic inference. Worth a serious referee, but the authors need to address the statistics gap before this is publishable. Here's the breakdown. What's actually new: instead of modeling aleatoric and epistemic uncertainty in isolation (the standard approach), they feed the predicted mean and variance from per-modality encoders as a context vector into a Bayesian gating network. So the signal-level uncertainty estimate directly conditions the model-level uncertainty estimate. That's a clean architectural contribution. The DUHV decoding strategy — majority vote over MC samples with confidence-based tie-breaking — is a reasonable solution to the sequence alignment problem that arises when different MC samples produce different-length hypotheses. Both ideas are well-motivated and the ablations in Table 2 do show each component contributing. The 7.9% relative WER reduction on AVCocktail fixed-chunk is the strongest result, and the framing of MUBF as a soft voice activity detector in noisy segments is a nice insight. Now the soft spots. The stress-test concern about missing variance is the real issue. The paper explicitly states results are averaged over 10 runs due to stochastic MC inference, yet reports no standard deviations or confidence intervals anywhere. On LRS2, the average improvement is 0.2% absolute (4.1 to 3.9), and several conditions show no change or slight regression (clean: 2.1 to 2.2). The ablation differences are even smaller — 0.2 to 0.4% absolute. Without variance estimates, we cannot tell if these gains exceed run-to-run noise. This is the load-bearing problem. The reader's concern about whether learned variance actually correlates with signal noise is also valid but secondary — a correlation analysis between predicted variance and SNR would strengthen the mechanistic claim, though the ablations provide indirect support. The slight regression in clean conditions is acknowledged but not really explained. Minor: no public code repository mentioned. Who this is for: researchers in robust multimodal speech recognition and uncertainty estimation. The coupling idea is worth knowing about regardless of whether the final numbers hold up. Recommendation: send to peer review. The novelty is real and the method is clearly described. But the reviewers should demand variance estimates across runs, and ideally a direct analysis of whether predicted aleatoric uncertainty tracks actual noise levels. If the gains survive proper statistical testing, this is a solid contribution. If they don't, the architectural idea still has value but the empirical claims need scaling back.","headline":"Novel uncertainty coupling for AVSR, but gains are small and missing variance estimates under stochastic inference","tokens_in":10319,"tokens_out":611,"would_cite":true,"duration_ms":1911722,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Noise-Aware Bayesian Gating Lifts Audio-Visual Speech Recognition in Cocktail-Party Noise","keywords":[],"falsifier":"If the learned per-modality variance does not correlate with actual signal-to-noise quality under corruption, or if decoupling aleatoric and epistemic modeling yields no measurable difference from independent modeling, the premise that data noise guides model confidence collapses. The ablation table partially addresses this, showing isolated aleatoric or epistemic modeling underperforms their joint injection.","tokens_in":9613,"feed_emoji":"🔇","tokens_out":649,"duration_ms":122748,"temperature":0.7,"pith_summary":"Audio-visual speech recognition systems struggle when audio is corrupted by overlapping speakers or background noise, because standard fusion methods cannot tell clean signal from garbage. This paper proposes UBG-Net, which couples two kinds of uncertainty: aleatoric uncertainty (measuring inherent data noise per modality) and epistemic uncertainty (measuring model confidence). The key idea is that the learned variance of each modality's features is fed as a context vector into a Bayesian gating network, so the gate knows how noisy each stream is and can downweight corrupted features accordingly. At inference time, the Bayesian network produces multiple candidate transcripts via Monte Carlo sampling; a hierarchical voting scheme picks the most frequent candidate and breaks ties using beam-search confidence scores. On the AVCocktail and noise-augmented LRS2 benchmarks, the method reduces word error rates relative to the baseline, with the largest gains in heavily corrupted conditions.","feed_headline":"Bayesian Gating Uses Learned Noise to Shield Lip-Reading from Cocktail-Party Babble","feed_subtitle":"Feeding per-modality variance into a Bayesian gate lets AVSR downweight corrupted audio, cutting WER up to 7.9% in real multi-party noise.","key_machinery":"The central mechanism is the Modality Uncertainty-aware Bayesian Fusion (MUBF): an MLP predicts mean and variance for each modality's features (capturing aleatoric uncertainty), the variance is concatenated with the mean into a context vector, and this context drives a Bayesian linear gating network whose weights carry distributions (capturing epistemic uncertainty). The gate output modulates the fused multimodal features before they reach the decoder. At inference, Monte Carlo sampling of the Bayesian weights yields multiple transcripts, and DUHV applies majority voting with confidence-score tie-breaking to select the final output.","core_discovery":"The paper claims that explicitly injecting signal-level aleatoric uncertainty as a contextual cue into a Bayesian gating network to model epistemic uncertainty (the MUBF mechanism), combined with a distribution uncertainty-aware hierarchical voting strategy (DUHV), achieves superior robustness for audio-visual speech recognition compared to SOTA baselines on the AVCocktail and LRS2 datasets.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Bayesian Gating Fuses Per-Modality Noise Estimates for Robust AV Speech Recognition","Uncertainty-Aware Fusion Downweights Corrupted Audio in Audio-Visual Speech Recognition","Injecting Signal Uncertainty Into Bayesian Gating Improves AVSR in Real-World Noise","Distribution-Aware Voting Over Monte Carlo Samples Boosts Audio-Visual Speech Accuracy","Aleatoric-to-Epistemic Uncertainty Pipeline Hardens Audio-Visual Speech Recognition"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The central assumption is that the variance predicted by a simple MLP genuinely reflects actual signal corruption quality, so that feeding it into the Bayesian gating layers meaningfully couples data noise with model confidence. If the network learns a variance that does not track real noise, the gating mechanism modulates features on a spurious signal.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian Gating Fuses Per-Modality Noise Estimates for Robust AV Speech Recognition","Uncertainty-Aware Fusion Downweights Corrupted Audio in Audio-Visual Speech Recognition","Injecting Signal Uncertainty Into Bayesian Gating Improves AVSR in Real-World Noise","Distribution-Aware Voting Over Monte Carlo Samples Boosts Audio-Visual Speech Accuracy","Aleatoric-to-Epistemic Uncertainty Pipeline Hardens Audio-Visual Speech Recognition"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":569,"prompt_tokens":475,"completion_tokens":94,"prompt_tokens_details":null},"tokens_in":475,"tokens_out":94,"duration_ms":141781,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T23:25:54.175509+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the learned per-modality variance does not correlate with actual signal-to-noise quality under corruption, or if decoupling aleatoric and epistemic modeling yields no measurable difference from independent modeling, the premise that data noise guides model confidence collapses. The ablation table partially addresses this, showing isolated aleatoric or epistemic modeling underperforms their joint injection.","supporting_citations":[],"review_version":1}