{"id":"9769a6e3-a42e-47d5-99df-70bc53637d4d","arxiv_id":"2412.19005","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A preference optimization objective with input-side and output-side preference pairs improves audiovisual ASR word error rates on How2, VisSpeech, and Ego4D beyond prior state-of-the-art.","lead":"This paper introduces a preference optimization training method for audiovisual speech recognition, using pairs of good and deliberately corrupted audio-visual inputs and transcripts. It reports that the method beats prior state-of-the-art on real-world video speech datasets while training on far less audiovisual data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Head-to-head SOTA claim is confounded by backbone choice: the SFT-only OWSM-visual baseline already beats AVFormer on all three sets; BPO adds only small marginal gains on VisSpeech/Ego4D, so the reported superiority is mostly inherited from OWSM v3.1 rather than from bifocal preference…","rationale":"The reader's weakest_assumption (invalid DPO for input-side preference) is a valid theoretical objection: Eq. 4 applies Eq. 2 to pairs with different inputs, so Z(x) cancellation fails; also, the pair shares the same output, so it is not a standard preference between completions. This would weaken the 'preference optimization' framing and should be fixed, but it does not by itself falsify the empirical claim that BPO-AVASR outperforms previous SOTA. The more load-bearing issue is attribution: the SOTA comparison in Table 1 is not controlled for backbone. Table 2 provides an internal control, and that control shows OWSM-visual small already beats AVFormer on all three datasets before any preference optimization. The BPO loss adds only 1.3-1.7% relative improvement on the out-of-domain sets. Therefore the abstract and introduction claim that the approach 'outperforms previous state-of-the-art models' is largely explained by the choice of OWSM v3.1, a 180k-hour supervised ASR model, rather than by bifocal preference optimization. This matters for the central claim because it changes what the paper has demonstrated: it demonstrates a strong recipe (OWSM + visual CLIP + SFT) that already surpasses AVFormer, with a smaller additional contribution from BPO. The paper should present the same-backbone SFT baseline in the main table, report variance, and either claim the whole system (which is legitimate) or isolate BPO's contribution. With those additions, a conditional accept remains appropriate; without them, the reader may over-credit the novelty. We therefore keep the reader's CONDITIONAL verdict but for a stronger reason than the DPO derivation issue.","tokens_in":12950,"tokens_out":8915,"duration_ms":88260,"concrete_test":"Add OWSM-visual small (the SFT-only, no-BPO baseline from Table 2) to Table 1 as a row, and compute the marginal BPO gain and its significance from at least 3 seeds with bootstrap confidence intervals on VisSpeech and Ego4D. If BPO-AVASR's improvements over OWSM-visual small are not statistically significant (especially the 0.2/0.7 WER differences on VisSpeech/Ego4D), then the SOTA superiority claim must be attributed to the OWSM backbone, and the paper's central claim should be narrowed accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 credits BPO-AVASR with beating AVFormer (13.6/16.4/64.6) using only 300h audiovisual data, implying preference optimization drives the gains. Table 2 contains the control that settles attribution: OWSM-visual small, the same OWSM backbone and CLIP visual encoder with SFT on How2 and no BPO, already achieves 10.5/15.8/59.9 on How2/VisSpeech/Ego4D, beating AVFormer on every dataset. BPO-AVASR small then gives 9.3/15.6/59.2, i.e., marginal absolute gains of 1.2/0.2/0.7 WER (11.4%/1.3%/1.7% relative). Thus the headline 'superior performance compared to previous SOTA' is mostly a property of the OWSM v3.1 ASR backbone, not of the proposed bifocal preference loss. The central claim as stated in Section 1 (and in the abstract) is technically about the full system, but the evidence that preference optimization is the cause of the SOTA result is weak: a same-backbone SFT baseline is already SOTA, and the added BPO gains are small, with no error bars or multi-seed significance testing. The theoretical concern in Eq. 4 (input-side loss is not a valid DPO objective because the partition function does not cancel across different inputs) is real but secondary; it affects interpretation and novelty, not the raw comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BPO-AVASR, an audiovisual ASR method that starts from the audio-only OWSM v3.1 model, adds a CLIP-based visual encoder, performs supervised fine-tuning (SFT) on How2, and then applies a 'bifocal preference optimization' stage. Preference pairs are constructed by masking audio frames, flipping video frames, and using ChatGPT to rewrite transcripts (homophone, spontaneous-speech, and vision-related errors). Experiments on How2, VisSpeech, and Ego4D report WER reductions relative to an SFT-only baseline, and the authors claim state-of-the-art results relative to AVFormer and other prior systems. Ablations compare eight preference construction strategies.","tokens_in":13336,"tokens_out":7958,"duration_ms":58579,"significance":"The paper's main positive contribution is a practical recipe for adapting a strong audio-only ASR model to audiovisual input with modest additional data: the preference data construction pipeline is concrete, and the same-backbone control (OWSM-visual small) in Table 2 is a commendable experimental design. If the small WER gains are reproducible across seeds, the method could be useful for AV-ASR practitioners. However, the headline claim of outperforming previous SOTA is weakened by two facts: (i) the SFT baseline already surpasses AVFormer on all three datasets, so the gains attributed to preference optimization are marginal (1.3%-11.4% relative), and (ii) on How2, AVATAR† reports a lower WER (9.1) than either BPO model. The theoretical derivation of the input-side DPO loss is also questionable. The pledge to release code and data is positive for reproducibility.","major_comments":[{"comment":"The central claim that BPO-AVASR 'outperforms previous state-of-the-art models across three datasets' is not supported by the paper's own control. Table 2 shows that OWSM-visual small, i.e., the same OWSM v3.1 backbone and CLIP encoder trained with SFT on How2 and no preference optimization, already achieves 10.5/15.8/59.9 WER on How2/VisSpeech/Ego4D, beating AVFormer (13.6/16.4/64.6) on every dataset. BPO-AVASR small improves these to 9.3/15.6/59.2, i.e., absolute gains of 1.2/0.2/0.7 WER. The abstract and Section 1 should attribute the SOTA-level performance primarily to the OWSM backbone and the SFT stage, and state that preference optimization provides a further smaller improvement.","section":"Section 1 and Section 5.3 (Table 2)"},{"comment":"The abstract's claim of superior performance over previous SOTA is also factually contradicted on How2: AVATAR† achieves 9.1 WER, while BPO-AVASR small and medium achieve 9.3 and 9.2, respectively. The paper acknowledges this in Section 5.1 ('AVATAR† achieves the best performance on How2'), so the blanket statement 'outperforming previous state-of-the-art models' should be qualified to refer to AVFormer specifically, or the authors should argue why AVATAR† is not the relevant SOTA baseline.","section":"Section 5.1 (Table 1)"},{"comment":"The input-side preference loss L_input is not a valid DPO objective. In the standard DPO derivation (Eqs. 1-2), the input x is fixed across the chosen and rejected responses, so the partition function Z(x) cancels in the Bradley-Terry ratio. In L_input, the chosen pair (tw|cw) and the rejected pair (tw|cl) have different inputs (cw vs cl), so the reward model r(x,y) has different partition functions Z(cw) and Z(cl), and the logistic loss does not follow from the DPO derivation. The paper should either derive a principled preference objective for the input-side case or explicitly present L_input as a contrastive/regularization term distinct from DPO. This is load-bearing because the paper's novelty rests on 'bifocal preference optimization' as preference optimization.","section":"Section 3.3 (Eq. 4)"},{"comment":"No error bars, confidence intervals, or multi-seed results are reported. Several conclusions rely on tiny absolute WER differences; for example, in Table 2 the differences between Flipped vision (9.7/15.5/59.3), Random cropped vision (12.1/15.7/61.5), and Masked audio (9.4/15.7/59.2) are small on VisSpeech/Ego4D, and the statement in Section 5.3 that 'the Ego4D dataset benefits significantly' from spontaneous-based generation (59.9 to 59.0) is unsupported without significance testing.","section":"Sections 5.1-5.3"},{"comment":"The How2 data split is not defined. The text states 'Following AVFormer, we use the 300-hour version of How2' but does not specify the train/development/test partition used for the SFT stage and for the WER evaluation. Since prior baselines in Table 1 may use different splits, the authors must report the exact split (e.g., the standard How2 dev5 partition) and the number of clips/utterances used.","section":"Section 4.2"},{"comment":"The protocol for Table 3 is ambiguous. The caption says 'All models are fine-tuned on the Ego4D training set,' but the main evaluation of Ego4D in Table 1 is zero-shot. It is unclear whether OWSM-visual small is SFT on Ego4D for the same number of epochs as the How2 SFT, whether the preference data for this experiment are constructed from the Ego4D training set as described in the text, and what hyperparameters are used. The authors should specify the full training setup for Table 3.","section":"Section 5.3 (Table 3)"}],"minor_comments":[{"comment":"The phrase '1 V100 GPU' should be rewritten as 'one V100 GPU' or 'a single V100 GPU.'","section":"Section 4.1"},{"comment":"The notation τ i is used for an element of D, but the preceding definition lists triples (ai, vi, ti); consider writing the tuple explicitly to avoid ambiguity.","section":"Section 3.1"},{"comment":"The labels 'Focal I' and 'Focal II' are confusing; consider using 'Input-side preference' and 'Output-side preference' directly in the figure.","section":"Figure 1"},{"comment":"The word 'significantly' is used without statistical support (e.g., 'the Ego4D dataset benefits significantly'); replace it with 'numerically' or add significance tests.","section":"Section 5.3"},{"comment":"The mixture row labels the model as 'BPO-AVASR small,' but the reader must cross-reference Table 1 for the model definition; consider adding a footnote clarifying that this is the full method.","section":"Table 2"},{"comment":"The baseline 'OWSM-ft small' is cited as Peng et al. 2023b, but the text elsewhere refers to OWSM v3.1 (Peng et al. 2024); ensure the baseline reference matches the exact model version used.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is borderline in novelty: the main theoretical contribution (bifocal preference optimization) has a flawed derivation for the input-side loss, and the empirical gains over the same-backbone SFT baseline are modest. The SOTA claim is overstated because the backbone does most of the work and AVATAR† beats the proposed models on How2. That said, the preference data construction and the controlled ablation are useful; with honest reframing, significance testing, and a corrected or heuristic framing of the input-side loss, the paper could be publishable. I recommend major revision rather than rejection because the issues are fixable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The new thing is a training objective for AV-ASR that combines input-side preference (masked audio, flipped video) with output-side preference (ChatGPT-rewritten transcripts). That pairing is genuinely new for this task, and the preference-data construction is thoughtful—hard negatives via frame flipping and homophone rewrites make sense. The paper also shows consistent WER improvements across three datasets, including zero-shot transfer to Ego4D. I believe the empirical result: preference optimization does help, at least at the margin.\n\nThe bigger problem is attribution. Table 2 is the control that matters, and it undercuts the headline. The SFT-only baseline, OWSM-visual small, already beats AVFormer on all three datasets (10.5 vs 13.6 on How2, 15.8 vs 16.4 on VisSpeech, 59.9 vs 64.6 on Ego4D). BPO then adds gains of 1.2/0.2/0.7 WER absolute. So the 'superior performance compared to previous state-of-the-art' claim is mostly inherited from OWSM v3.1, not from bifocal preference optimization. The paper acknowledges this baseline only in the ablation, not in the abstract or intro, which is misleading.\n\nSecond, the theory is shaky. Equation 4's L_input plugs different inputs into the DPO objective. In standard DPO, the partition function Z(x) cancels only when x is fixed; with different cw and cl, that cancellation fails. So the input-side loss is not a valid DPO objective. It might still work as a generic contrastive regularizer, but the paper doesn't frame it that way. That's a real flaw in the derivation, though not necessarily fatal to the empirical claims.\n\nMinor but worth noting: no error bars or multi-seed runs, the How2 train/test split is undefined, and Table 3's fine-tuning protocol on Ego4D is underspecified (how much data, what visual encoder). These are fixable.\n\nWho benefits: anyone working on AV-ASR or applying preference optimization to ASR. The method is simple and the paper gives a good recipe for preference-data construction. It deserves a serious referee, but the authors need to reposition the claims—present BPO as a modest refinement over a strong SFT baseline, not as the reason they beat AVFormer.","headline":"New preference-optimization recipe for AV-ASR with real but modest gains; the SOTA headline is mostly the OWSM backbone, not the method.","tokens_in":13819,"tokens_out":1699,"would_cite":true,"duration_ms":16265,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Preference optimization beats 131k-hour audiovisual training","keywords":["audiovisual speech recognition","preference optimization","direct preference optimization","multimodal ASR","hard negative construction","real-world video","word error rate","visual features"],"falsifier":"Run BPO-AVASR on a test set where horizontally flipping the frame removes no information relevant to the transcript (e.g., symmetric scenes). If WER still improves relative to the SFT baseline, the input-side loss is acting as a generic regularizer, not as a preference over visual informativeness.","tokens_in":12785,"feed_emoji":"🎙️","tokens_out":7745,"duration_ms":65564,"temperature":0.7,"pith_summary":"This paper argues that audiovisual speech recognition (AV-ASR) in unconstrained real-world videos can be improved by framing the task as a preference optimization problem rather than as a purely supervised fine-tuning problem. The authors construct preference pairs that simulate the errors an AV-ASR system actually makes — masked or noisy audio, flipped video frames, homophone substitutions, and spontaneous-speech rewrites — and then train the model to prefer the correct transcript over each corrupted variant. On the model side, they propose BPO-AVASR, a 'bifocal' DPO-style objective that combines an input-side preference loss (rewarding correct transcripts when the input is clean rather than corrupted) with an output-side preference loss (rewarding correct transcripts over error-rewritten ones). The authors claim that this approach reaches state-of-the-art word error rates on How2, VisSpeech, and Ego4D, outperforming prior AV-ASR models even when those models were trained on hundreds of times more audiovisual data.","feed_headline":"Preference optimization beats 131k-hour audiovisual training","feed_subtitle":"A bifocal preference objective cuts word error rates on How2, VisSpeech, and Ego4D using just 300 hours of video.","key_machinery":"The central object is the bifocal preference dataset and its associated loss. Input-side preference pairs $(t_w | a_w, v_w) \\succ (t_w | a_l, v_w)$ and $(t_w | a_w, v_w) \\succ (t_w | a_w, v_l)$ hold the transcript fixed and degrade the audio or video input; output-side pairs $(t_w | a_w, v_w) \\succ (t_l | a_w, v_w)$ hold the input fixed and degrade the transcript through homophone, spontaneous-speech, or vision-omission rewrites. The training objective is $L_{BPO} = L_{input} + L_{output}$, where each term is a logistic contrast of the policy's log-probability ratio for chosen versus rejected pairs, anchored against the reference model $\\pi_{\\text{ref}}$. The construction of hard negatives — choosing input corruptions that genuinely remove information (flipped vision rather than random cropping) and LLM-generated transcript rewrites rather than simple dictionary swaps — is what makes the preference signal informative.","core_discovery":"The central claim is that BPO-AVASR, trained with the proposed bifocal preference optimization, achieves superior performance compared to previous state-of-the-art models across three datasets. Specifically, BPO-AVASR small and medium reach 9.3% and 9.2% WER on How2, 15.6% and 14.5% on VisSpeech, and 59.2% and 56.5% on Ego4D, while the prior AVFormer baseline obtains 13.6%, 16.4%, and 64.6% respectively — and AVFormer used 131,000 hours of audiovisual fine-tuning data, whereas BPO-AVASR uses only 300 hours. The authors attribute the improvement to the bifocal preference construction, which teaches the model to avoid the specific error patterns that dominate unconstrained video speech: homophone confusions, spontaneous-speech disfluencies, and reliance on missing or misleading visual cues.","pith_inferences":["If the input-side loss is not a valid DPO objective because the partition function does not cancel, the empirical gains might be replicable with a simpler contrastive regularizer; a head-to-head test with such a regularizer would clarify whether the preference framing is essential.","The reliance on LLM-based transcript rewrites for preference construction makes the method's performance partly a function of those generators' quality; automated, self-supervised preference generation would make it more portable to low-resource languages.","The reported relative reductions are computed against AVFormer, which was trained on different data; a controlled comparison with the same backbone and data would isolate the effect of the preference objective."],"forward_implications":["AV-ASR no longer needs hundred-thousand-hour audiovisual corpora: preference optimization on 300 hours of How2 matches or beats models trained on 131k hours.","The same preference-construction recipe transfers to other domains: building preference pairs from Ego4D's training set improves Ego4D WER from 52.3% to 50.0% without changing the method.","The method suggests that ASR can be treated as a preference-alignment problem, opening the door to using human or model feedback beyond correct/wrong transcript labels.","Because the construction targets homophone and spontaneous-speech errors, the gains are likely to be largest in noisy, spontaneous, egocentric video — precisely the regime where conventional AV-ASR fails."],"supporting_citations":[{"why":"Supplies the DPO objective and Bradley–Terry grounding that the BPO loss extends to the input and output sides.","marker":"Rafailov et al. 2023"},{"why":"AVFormer, the previous state-of-the-art AV-ASR model whose 131k-hour training scale and WER results define the comparison.","marker":"Seo et al. 2023"},{"why":"OWSM v3.1, the open audio-only ASR backbone that BPO-AVASR fine-tunes with visual conditions.","marker":"Peng et al. 2024"},{"why":"How2, the 300-hour dataset used for supervised fine-tuning, preference construction, and evaluation.","marker":"Sanabria et al. 2018"},{"why":"ShareGPT4Video, the captioner that produces dense video captions used to create vision-based output-side preference samples.","marker":"Chen et al. 2024"},{"why":"Ego4D, the egocentric benchmark used for zero-shot evaluation and same-domain preference-construction generalization.","marker":"Grauman et al. 2022"}],"fun_headline_variants":["Bifocal preference cuts WER with 300 hours vs 131k","300 hours beats 131k: bifocal preference for AV-ASR","Preference optimization: 300h beats 131k-hour training","AV-ASR: small data, big gains via bifocal preference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The input-side preference loss assumes that the DPO derivation remains valid when chosen and rejected pairs share the same transcript but have different inputs; in the standard DPO objective the partition function cancels only when the input is fixed, so this loss may not be a true preference objective.","fun_headline_variants_meta":{"raw":{"variants":["Bifocal preference cuts WER with 300 hours vs 131k","300 hours beats 131k: bifocal preference for AV-ASR","Preference optimization: 300h beats 131k-hour training","AV-ASR: small data, big gains via bifocal preference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000554,"raw_usage":{"total_tokens":2631,"prompt_tokens":931,"completion_tokens":1700,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":1620}},"tokens_in":547,"tokens_out":1700,"duration_ms":12395,"temperature":1.0,"reasoning_tokens":1620,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:57:54.482417+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BPO-AVASR on a test set where horizontally flipping the frame removes no information relevant to the transcript (e.g., symmetric scenes). If WER still improves relative to the SFT baseline, the input-side loss is acting as a generic regularizer, not as a preference over visual informativeness.","supporting_citations":[{"cited_title":"H.; et al","cited_arxiv_id":null,"evidence_quote":"AVFormer, the previous state-of-the-art AV-ASR model whose 131k-hour training scale and WER results define the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Ego4D, the egocentric benchmark used for zero-shot evaluation and same-domain preference-construction generalization."}],"review_version":1}