{"id":"dca66385-a382-471f-969d-d7c360543932","arxiv_id":"2412.08548","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A bilevel training method that jointly optimizes supervised and unsupervised losses outperforms pretraining-then-finetuning for ASR on LibriSpeech, Switchboard, and an in-house dataset.","lead":"This paper introduces a training method for speech recognition that mixes labeled and unlabeled data in one optimization, rather than the usual two-step pretrain-then-finetune approach. Across three speech datasets, the method improves word error rates compared to pretraining and fine-tuning, with the largest gains on conversational speech.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline LibriSpeech gain over PT+FT comes from a penalty schedule chosen directly on test WER; across the four schedules the spread is as large as the claimed advantage, so the 4.1% vs 5.1% margin is not yet robust.","rationale":"I focused on the empirical evaluation rather than the unverified PL/Lipschitz assumptions because the paper's central claim as stated is about WER improvement; the theory issue would weaken the 'matched local optima' interpretation but would not by itself falsify the empirical result. The test-set schedule selection is the most direct threat to the main numbers: the method's key hyperparameter is selected on the test sets, and the variance across schedules is comparable to the effect size. A dev-based selection and multiple seeds would settle this cheaply. If the margin survives, I would regard the empirical claim as established; if not, the paper should be reported with the selection caveat. I agree partially with the reader: the reader identified test-set selection in the rationale but made the unverified theory the weakest assumption. The theory concern remains secondary but real: Lemma 1's conditions are not checked, so the 'bilevel' interpretation is not fully supported even if the WER results hold.","tokens_in":15980,"tokens_out":11217,"duration_ms":124734,"concrete_test":"Re-run the LibriSpeech 100h/860h condition with the same four penalty schedules of Table V, but select the schedule on a held-out dev split (e.g., dev-clean/dev-other) and then report test WER for that schedule; repeat with at least 3 seeds. If the dev-selected schedule differs from 0.002/0.20, or if the test advantage over PT+FT shrinks by more than about 0.5% absolute, the headline should be revised to reflect selection bias; if the margin persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table V of Section V-C tunes the BL-JUST penalty schedule (increment rate and maximum gamma) for the LibriSpeech 100h/860h condition and reports only the best of four runs: 0.002/0.20 gives 4.1/11.3, while the other three schedules give 5.2-6.7 on test-clean and 12.1-14.8 on test-other. There is no held-out validation split; the choice is made on the same test sets used in Table I, and Appendix A's learning-rate scheduler also monitors 'test loss' every 10 epochs. PT+FT has no comparable free hyperparameter, so the reported 1.0% and 1.9% absolute advantages over PT+FT are optimistic upper bounds. The across-schedule spread is as large as the claimed gain; with no seeds or error bars, the 'consistently outperforms' claim rests on a single selected configuration. The Switchboard and Payload results are consistent and mitigate this, but they cannot repair the lack of uncertainty quantification on the headline comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BL-JUST, a bilevel training framework for ASR in which supervised training on labeled data is the upper-level problem and unsupervised training on unlabeled data is the lower-level problem, with the two coupled through a penalty-based bilevel gradient descent method borrowed from Shen and Chen [22]. The authors evaluate BL-JUST on LibriSpeech, Switchboard, and an internal payload dataset, using CPC and BEST-RQ as unsupervised losses and CTC and RNNT as supervised losses, and report that BL-JUST consistently outperforms PT+FT as well as JUST, pseudo-labeling, and alternating optimization. The headline results are 4.1% vs. 5.1% WER on LibriSpeech test-clean under a 100h labeled / 860h unlabeled split, and 8.2% vs. 9.6% on the Switchboard average. The paper also includes an ablation study and an analysis of loss and gradient norms to support the 'matched local optima' interpretation.","tokens_in":16289,"tokens_out":3986,"duration_ms":43831,"significance":"If the empirical claim is robust, the result is practically significant: it suggests that, for semi-supervised ASR, jointly optimizing supervised and unsupervised objectives within a bilevel formulation can beat the standard disconnected pretraining-then-finetuning recipe. The breadth of the evaluation—three datasets, two unsupervised losses, two supervised losses, and multiple architectures—is a genuine strength, as is the inclusion of ablations and comparisons with several semi-supervised baselines. However, the current evidence is weakened by test-set-based hyperparameter selection, the absence of variance or significance information, and theoretical assumptions that are not verified for the actual losses and models. The central idea is plausible and worth publishing if the empirical protocol is tightened, but the present manuscript does not yet support the strength of the stated claims.","major_comments":[{"comment":"The penalty schedule for LibriSpeech is selected on the test sets. Table V reports four increasing-rate schedules with test-clean WERs of 6.7, 4.1, 4.7, and 5.2, so the spread across configurations (2.6 percentage points) is larger than the claimed BL-JUST advantage over PT+FT on test-clean (1.0 percentage point). Furthermore, Appendix A states that the learning-rate scheduler 'monitor[s] the test loss every 10 epochs' for the baseline, PT+FT fine-tuning, and BL-JUST training. This is test-set contamination for the entire LibriSpeech comparison, not just for the penalty value. The manuscript needs a held-out validation split for schedule selection and learning-rate scheduling, plus multiple seeds with error bars or significance tests, before the 'consistently outperforms' claim is supported by the LibriSpeech experiments.","section":"§V-C, Table V and Appendix A"},{"comment":"The theoretical equivalence and convergence guarantee rely on Assumptions (a)-(c): L-Lipschitz continuity of f, the Polyak-Łojasiewicz inequality for g, and Lipschitz continuity of ∇Fγ. These assumptions are never verified for CPC, BEST-RQ, CTC, or RNNT losses on Conformer networks. Since the lower-level problems in this paper are trained with SGD rather than solved exactly, and since the PL condition is a strong global condition, the penalty-based reformulation is not guaranteed to approximate the bilevel problem in this setting. The paper should either verify the assumptions empirically (for example, by estimating the PL constant along the training trajectory) or explicitly recast the theory as a heuristic motivation and place the burden on the empirical comparisons.","section":"§III, Lemma 1 and Theorem 1"},{"comment":"The final supervised fine-tuning step (the N3 loop) is outside the penalty-based bilevel formulation analyzed in Section III. After this step, the model is no longer guaranteed to satisfy the lower-level constraint, so Lemma 1 and Theorem 1 do not apply to the final model whose WER is reported. The ablation study shows that removing fine-tuning degrades WER, but it also means the reported system is a hybrid of the penalty method and a final PT+FT-like stage. Please clarify the role of this step relative to the theoretical claims, or report the performance before the final fine-tuning as the BL-JUST solution and the fine-tuned version as a post-processing variant.","section":"§IV, Algorithm 1"},{"comment":"No variance, confidence intervals, or significance tests are reported for any table. All comparisons appear to be single runs, and the claimed gains on LibriSpeech (1.0-1.9 absolute WER) and Switchboard (1.4 absolute on average) are moderate in size. Given the test-set selection issue in Table V, the lack of uncertainty quantification makes it impossible to determine whether the headline gains are robust. At minimum, the authors should report multiple random seeds for the key comparisons and, ideally, a small hyperparameter sensitivity analysis on a validation set.","section":"§V, Tables I-III"}],"minor_comments":[{"comment":"The text says 'BL-JUST also reaches a better CTC loss than PF+FT'; 'PF+FT' should be 'PT+FT'.","section":"§V-D"},{"comment":"'The masked frames are replaced with Gaussian noise with with 0 mean and 0.1 variance' contains a duplicated 'with' in both appendices.","section":"Appendices B and C"},{"comment":"The row label 'simi' appears to be a placeholder and should be replaced with the intended method description.","section":"Table V"},{"comment":"The penalty update γk = (k - 1)γm/K reaches (K-1)/K γm at k=K rather than γm; either the formula or the description 'increases to a pre-defined maximum value' should be corrected. Also, the values of N1, N2, and N3 are not specified in the appendices, although they are part of the algorithm's input.","section":"Algorithm 1"},{"comment":"Reference [2] and [27] contain the typo 'Maganize' for 'Magazine'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The theoretical foundation is taken from [22], whose authors overlap with the present paper. This is not inherently a problem, but the manuscript should make the overlap explicit and should not present the theory as independent validation without noting the provenance. The previous work [25] is an earlier conference version; the novelty of this journal submission relative to [25] should be stated more clearly in the introduction. The main barrier is empirical rigor: the LibriSpeech headline comparison is compromised by test-set-based selection, and the absence of any uncertainty quantification across all tables is a concern for a journal-level claim of consistent improvement. These issues are fixable with additional experiments, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe quick take: the paper's claim that a bilevel formulation (supervised loss as leader, unsupervised loss as follower) beats disconnected pretrain+finetune is supported by a wide set of experiments, but the LibriSpeech number quoted in the abstract is the best of four penalty schedules chosen directly on test WER. The spread across schedules is as large as the claimed advantage, so the 4.1% vs 5.1% margin is not robust as reported.\n\nWhat's actually new: the method generalizes JUST and extends the authors' ICASSP 2024 paper with genuinely new evidence—Switchboard with RNNT, Payload with 90k hours, BEST-RQ, larger conformers, and direct comparisons to pseudo-labeling and alternating optimization. The ablations (removing self-supervised exploration, removing final fine-tuning) are clean and show both pieces matter. The loss curves in Fig. 3 provide a nice supporting detail. The empirical pattern across three datasets and two loss families is consistent, which is more than most short ASR papers offer.\n\nWhere it's soft. The stress-test concern lands. Table V sweeps four penalty schedules and reports the best one for LibriSpeech 100h/860h; the other three give 4.7–6.7 on test-clean, putting two of them at or below PT+FT's 5.1. No held-out split, no multiple seeds, no error bars. On top of that, all methods use a learning-rate scheduler that monitors test loss, so test-set selection is baked into the whole pipeline. The theory in Section III cites [22] for the penalty equivalence; the PL and Lipschitz assumptions are stated but never verified for CPC, BEST-RQ, CTC, or RNNT on conformers. That is a real limitation, though the empirical claim does not depend on the theory being verified—only on the algorithm working. The self-citation to [22] and [25] is not a problem here: the cited work is a separate theory paper, and the comparisons are against external baselines.\n\nBottom line: this deserves a serious referee. The method is coherent, the experimental breadth is useful, and the qualitative conclusion—joint bilevel training can use unlabeled data more effectively than disconnected PT+FT—is credible. What needs fixing before the numbers are trusted: a proper held-out validation split for hyperparameters, at least three seeds or a significance statement, and a toned-down \"consistently outperforms\" claim. I'd send it to review, and I'd want the revised version to address uncertainty quantification before acceptance.\n\nMy recommendation: take it, but require the validation work.","headline":"BL-JUST is a plausible, broadened extension of the authors' earlier bilevel ASR work, but the LibriSpeech headline gain is the best of four penalty schedules selected on the test set, so read the magnitude as optimistic.","tokens_in":16770,"tokens_out":3903,"would_cite":true,"duration_ms":40683,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that joint bilevel optimization of supervised and unsupervised speech losses, called BL-JUST, consistently outperforms the standard pre-training plus fine-tuning recipe.","keywords":["bilevel optimization","joint unsupervised and supervised training","automatic speech recognition","penalty-based gradient descent","semi-supervised learning","pre-training and fine-tuning","self-supervised speech representation learning","Conformer acoustic model"],"falsifier":"A concrete check: train the same Conformer models with BL-JUST and PT+FT on LibriSpeech with 100 labeled and 860 unlabeled hours, measure the final gradient norm of the CPC loss, and also run BL-JUST with a constant penalty factor after the unsupervised gap has narrowed. If the final CPC gradient norm is not substantially smaller under BL-JUST than under PT+FT, or if the constant-penalty version matches the scheduled-penalty version in WER, the matched-optimum story would be contradicted.","tokens_in":15816,"feed_emoji":"🎙️","tokens_out":10232,"duration_ms":93819,"temperature":0.7,"pith_summary":"This paper tries to establish that automatic speech recognition models are better trained by solving the supervised and unsupervised objectives together in one bilevel optimization than by the standard two-stage recipe of unsupervised pre-training followed by supervised fine-tuning. The proposed method, BL-JUST, treats the supervised loss on labeled speech as an upper-level problem whose solution is required to stay near a local optimum of the unsupervised loss on unlabeled speech, and it solves that pair with penalty-based bilevel gradient descent. Across LibriSpeech, Switchboard, and an industrial payload dataset, BL-JUST reports lower word error rates than PT+FT in every configuration tested, including 4.1% versus 5.1% on LibriSpeech test-clean with 100 labeled and 860 unlabeled hours. The stakes are practical: labeled speech is expensive, unlabeled speech is abundant, and if joint training indeed finds matched local optima of both losses, it offers a more efficient use of that unlabeled data for building task-ready acoustic models.","feed_headline":"Joint bilevel training beats pre-then-finetune in speech recognition","feed_subtitle":"On LibriSpeech, BL-JUST cuts WER from 5.1% to 4.1% with 100 labeled and 860 unlabeled hours.","key_machinery":"The carrying object is the penalty reformulation $F_\\gamma(\\theta,\\phi,\\eta) = f(\\theta,\\phi) + \\gamma\\,(g(\\theta,\\eta) - v)$, where $v = \\min_{\\theta,\\eta} g(\\theta,\\eta)$ is the optimal value of the unsupervised loss and the parenthesis is the value-function gap. The algorithm maintains a shared backbone $\\theta$, a supervised head $\\phi$, and an unsupervised head $\\eta$; each epoch it first runs self-supervised exploration to approach a local optimum of the unsupervised loss, then applies the penalty-based bilevel gradient descent update with the penalty coefficient $\\gamma$ increasing linearly from 0 to a preset maximum, then ends with a small-learning-rate supervised fine-tuning. Lemma 1 is the theoretical link: for $\\gamma$ large enough, every local or global solution of the penalty problem is an $\\epsilon$-approximate solution of the bilevel problem, so the procedure is meant to converge to a matched local optimum of both losses.","core_discovery":"The central claim is that the disconnection in the conventional PT+FT pipeline carries a real cost: pre-training ignores the labeled task and fine-tuning ignores the unlabeled data, so the final model is not required to be good for both objectives and can land in a supervised local optimum with a high unsupervised loss. BL-JUST avoids this by solving $\\min_{\\theta,\\phi} f(\\theta,\\phi)$ subject to $\\theta \\in \\arg\\min_{\\theta',\\eta} g(\\theta',\\eta)$, where $f$ is the supervised loss (CTC or RNN-Transducer) and $g$ is the unsupervised loss (CPC or BEST-RQ). The paper reports that this joint formulation outperforms PT+FT in every experiment and also beats constant-penalty joint training (JUST), pseudo-labeling, and alternating optimization. It takes the simultaneously faster drop of both loss curves and the smaller final unsupervised-gradient norm under BL-JUST as evidence that the training reaches matched local optima of the two losses, yielding representations that remain useful for unlabeled speech while being adapted to the ASR task.","pith_inferences":["Beyond the paper's speech experiments, the same value-function-gap penalty is modality-agnostic, so the bilevel coupling is a natural candidate for any field with a large unlabeled pool and a small labeled pool, such as computer vision or language modeling; this is an editorial inference, not a claim of the paper.","The paper tests only a linear penalty schedule, but its theory only requires $\\gamma$ to be large enough; an adaptive schedule that grows $\\gamma$ in response to the measured unsupervised gap could, in principle, reach the same optimum with less tuning, though the paper does not test this.","If the matched-optimum interpretation is correct, BL-JUST models should transfer better to out-of-domain speech than PT+FT models at equal primary-test WER, because the unsupervised constraint keeps the shared representations from drifting too far from the unlabeled distribution; this is a testable prediction the paper does not report."],"forward_implications":["If the paper is right, the widely used PT+FT recipe leaves accuracy on the table: coupling supervised and unsupervised losses with a growing penalty reduces WER at the same data and model sizes.","The reported gains are not tied to one self-supervised loss; they appear with CPC plus CTC on LibriSpeech and with BEST-RQ plus RNNT or CTC on Switchboard and Payload, which points to the bilevel coupling itself as the active ingredient.","The ablation results imply that the self-supervised exploration step is the most important component of the implementation, and that the final fine-tuning step still adds a further improvement; removing both raises test-clean WER from 4.1% to 5.9%.","Scaling within the method is a direct continuation of the paper's results: a 15-block Conformer trained with BL-JUST reaches 8.1% average WER on Switchboard, better than the 8.2% of the 74M-parameter version."],"supporting_citations":[{"why":"Supplies the penalty-based bilevel gradient descent solver, including Lemma 1 and Theorem 1 that justify the penalty reformulation and its convergence rate.","marker":"[22]"},{"why":"Defines the JUST baseline, joint training with a constant penalty and no unsupervised exploration, which BL-JUST is compared against and extends.","marker":"[23]"},{"why":"Defines the Contrastive Predictive Coding loss used as the lower-level unsupervised objective in the LibriSpeech experiments.","marker":"[36]"},{"why":"Defines the Connectionist Temporal Classification loss used as the upper-level supervised objective in LibriSpeech and Payload.","marker":"[37]"},{"why":"Defines BEST-RQ self-supervised training with a random-projection quantizer, the lower-level unsupervised objective on Switchboard and Payload.","marker":"[11]"},{"why":"Inspires the frame-label re-alignment procedure that the paper uses to improve BEST-RQ pre-training for PT+FT and BL-JUST on Switchboard.","marker":"[10]"},{"why":"Provides the LibriSpeech dataset and the train-clean-100/360 and train-other-500 splits used in the main comparisons.","marker":"[32]"},{"why":"Defines the Conformer architecture used for all acoustic models in the experiments.","marker":"[33]"},{"why":"Supplies the iterative pseudo-labeling baseline compared on Switchboard.","marker":"[24]"}],"fun_headline_variants":["Bilevel joint training beats pre-then-finetune for ASR","Cut speech WER 20% with bilevel joint training","One training pass couples unsupervised and supervised ASR","Joint bilevel training surpasses two-stage ASR on LibriSpeech","With 100 labeled hours, bilevel joint training beats two-stage ASR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The theoretical guarantee that the penalty method exactly solves the bilevel problem requires the unsupervised loss to satisfy a strong-gradient condition (the Polyak-Łojasiewicz inequality, which says the gradient does not vanish while the loss is still far from its minimum) and requires the supervised loss and the penalty gradient to be smooth; the paper never verifies these conditions for CPC, BEST-RQ, CTC, or RNNT losses on Conformer networks, so if they fail the matched-optimum interpretation rests on measured gradient norms rather than on the convergence proof.","fun_headline_variants_meta":{"raw":{"variants":["Bilevel joint training beats pre-then-finetune for ASR","Cut speech WER 20% with bilevel joint training","One training pass couples unsupervised and supervised ASR","Joint bilevel training surpasses two-stage ASR on LibriSpeech","With 100 labeled hours, bilevel joint training beats two-stage ASR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001077,"raw_usage":{"total_tokens":4490,"prompt_tokens":912,"completion_tokens":3578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":3488}},"tokens_in":528,"tokens_out":3578,"duration_ms":25313,"temperature":1.0,"reasoning_tokens":3488,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:44:33.705759+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: train the same Conformer models with BL-JUST and PT+FT on LibriSpeech with 100 labeled and 860 unlabeled hours, measure the final gradient norm of the CPC loss, and also run BL-JUST with a constant penalty factor after the unsupervised gap has narrowed. If the final CPC gradient norm is not substantially smaller under BL-JUST than under PT+FT, or if the constant-penalty version matches the scheduled-penalty version in WER, the matched-optimum story would be contradicted.","supporting_citations":[{"cited_title":"Joint unsupervised and supervised training for multilingual ASR,","cited_arxiv_id":null,"evidence_quote":"Defines the JUST baseline, joint training with a constant penalty and no unsupervised exploration, which BL-JUST is compared against and extends."},{"cited_title":"Self-supervised learning with random-projection quantizer for speech recognition,","cited_arxiv_id":null,"evidence_quote":"Defines BEST-RQ self-supervised training with a random-projection quantizer, the lower-level unsupervised objective on Switchboard and Payload."},{"cited_title":"Librispeech: an asr corpus based on public domain audio books,","cited_arxiv_id":null,"evidence_quote":"Provides the LibriSpeech dataset and the train-clean-100/360 and train-other-500 splits used in the main comparisons."},{"cited_title":"Iterative pseudo-labeling for speech recognition,","cited_arxiv_id":null,"evidence_quote":"Supplies the iterative pseudo-labeling baseline compared on Switchboard."}],"review_version":1}