{"id":"0f13ab67-103e-4ec4-ab0a-21e7ae792e9b","arxiv_id":"2505.15333","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A text-like 'unit language' mined from discrete speech units via n-gram modeling, plus task-prompt multi-task training, improves textless speech-to-speech translation to near text-supervised performance.","lead":"The paper introduces 'unit language', a text-like tokenization of speech units built from n-gram statistics, and shows that training a textless speech-to-speech translation model with it as an auxiliary objective improves translation quality. The method closes most of the gap between models trained without any text and models trained with text transcriptions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported BLEU gains (0.4–1.2) are single-run with no significance testing; the central claim of 'significant improvements' may not survive run-to-run variance.","rationale":"The reader's weakest_assumption focused on the norm-unit dependency, but the more load-bearing issue is the evidentiary basis for the central claim: no statistical significance testing. If the gains are within noise, the method's contribution vanishes, even if the implementation is correct. The norm-unit limitation is a scope restriction, not a falsifier for the stated setting. The proposed concrete test would settle whether the improvements are real. The reader already implicitly attached a 'multiple seeds' condition in the verdict, so the reader's CONDITIONAL verdict remains appropriate.","tokens_in":14444,"tokens_out":7155,"duration_ms":56589,"concrete_test":"Run each configuration in Table 2 (Baseline, +LCM, +LCL, +LCM&LCL, +Task prompt) with at least 3 different random seeds, and report mean BLEU ± std. Apply a paired significance test (e.g., bootstrap over test sentences or paired t-test across seeds) comparing Baseline vs +Task prompt. If the improvement is not significant (p<0.05), the central claim fails. Also report the variance for the BPE comparison in Table 6 to confirm the unit language advantage over BPE exceeds noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 reports average BLEU improvements from +0.4 to +1.2 over the baseline, with no error bars, multiple seeds, or significance tests. The abstract's claim of 'significant improvements' is therefore not statistically supported. In S2ST, BLEU differences of this size are comparable to typical seed-to-seed variation; the hyperparameter sweeps in Tables 4 and 5 show fluctuations of the same magnitude (En-Es +CM: 23.5 vs 22.3 between r=2 and r=4; K=2 vs K=3 yields 23.3 vs 23.5). The final +1.2 average with task prompt also depends on an unablated negative weight (-3.0) for the prompt-difference loss. Without a variance estimate, the 'consistent across four language pairs' pattern could arise from a systematic training-schedule effect rather than the unit language itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a \"unit language\" for textless speech-to-speech translation (S2ST): discrete speech units are merged into pseudo-words via n-gram language modeling, yielding a text-like representation that requires no transcriptions. This unit language is used as auxiliary supervision through two additional decoders and losses, LCM (cross-modal) and LCL (cross-lingual), which guide the encoder of a S2UT-style model. Since applying source and target unit languages together initially degrades performance, the authors introduce learnable task prompts to reconcile the two tasks. Experiments on four VoxPopuli language pairs report average BLEU improvements of +0.4 to +1.2 over a reproduced strong baseline, reaching an average of 21.5 BLEU that is comparable to multi-task training with recognized text.","tokens_in":14625,"tokens_out":6218,"duration_ms":55089,"significance":"If the reported improvements are robust, the paper makes a useful contribution: it shows that an unsupervised, text-like tokenization of speech units can provide auxiliary supervision for textless S2ST, reducing the need for parallel text or ASR transcripts. The construction is self-supervised (the unit language is derived from the same unlabeled unit corpus used for training), and the main BLEU evaluations are on held-out test sets, so the reported gains are not circular. The paper also includes a controlled comparison against a BPE-based pseudo-language (Table 6), an analysis of hyperparameter sensitivity (Tables 4 and 5), and diagnostic sparseness/localness analyses intended to explain why CM and CL help. The main reservations are that all results come from single runs with no significance testing, the benefit is strongly dependent on external norm-unit preprocessing (Appendix G), and the final task-prompt loss includes an unablated negative-weight term.","major_comments":[{"comment":"The abstract and Section 3.2 describe the gains as \"significant\" and \"consistent\", but Table 2 reports only single-run BLEU scores with no error bars, multiple seeds, or significance tests. The claimed average improvements of +0.4 to +1.2 BLEU are comparable in magnitude to the hyperparameter sensitivity shown in Tables 4 and 5 (e.g., En-Es +CM ranges from 22.3 to 23.5 as r varies from 4 to 2; K=2 versus K=3 changes En-Es +CM from 23.3 to 23.5). Without variance estimates or paired significance tests, the central quantitative claim is not supported. Please provide at least three seeds per configuration or bootstrap confidence intervals for the main comparisons.","section":"Table 2 / Abstract"},{"comment":"The value of the proposed unit language is almost entirely contingent on norm-unit preprocessing: with non-normalized units, the average LCM&LCL gain drops from +0.9 to +0.2 BLEU (Table 11). Since norm units require an additional speaker-normalization model (Lee et al., 2022b) that may not exist or transfer to new or unwritten languages, the paper should explicitly scope the central claims to \"unit language built on norm units\". As written, the abstract and conclusion state a general claim about unit-language supervision that the non-norm condition does not support.","section":"Appendix G, Table 11"},{"comment":"The task-prompt mechanism is underspecified, which harms reproducibility. The text says b_CM is concatenated at the first position and later \"replaced\" by b_CL after the r-th layer, but it does not state whether the prompt is an extra token in the sequence (changing length), whether it is added to all layer inputs or only the r-th layer, or how the replacement is implemented. In addition, the negative MSE loss between b_CM and b_CL with weight −3.0 is not ablated: without removing this term or varying its weight, we cannot tell whether the +1.2 BLEU of \"+Task prompt\" comes from the prompt mechanism itself or from this auxiliary loss. Please provide a precise formal definition and an ablation over the MSE weight (including 0).","section":"Section 2.4, Figure 3"},{"comment":"The paper claims \"new state-of-the-art performance on the textless S2ST task using the VoxPopuli dataset\", but Table 2 only compares against the reproduced S2UT baseline and the text-based Seamless model. No published textless S2ST systems (e.g., UnitY or other direct S2ST methods evaluated on VoxPopuli) are included. To support the state-of-the-art claim, the authors should compare with published textless S2ST results on the same data or temper the claim to \"comparable to S2UT with text supervision\".","section":"Section 3.2 / Introduction"}],"minor_comments":[{"comment":"The conditional-probability notation in Eq. (8) is ambiguous: the index ranges in the numerator and denominator do not make the conditioning event explicit, and l is defined rather indirectly. Please rewrite with a clear definition of l and the segment lengths involved.","section":"Section 2.1, Eq. (8)"},{"comment":"There are repeated typos such as \"V oxpupil\" (abstract) and \"langauge\" (Section 4.6). Please proofread the manuscript.","section":"Abstract and throughout"},{"comment":"The caption label \"FrameUnitCharUnit_lText\" is difficult to parse; separate the entries as \"Frame, Unit, Char, Unit_l, Text\" for clarity.","section":"Figure 4 caption"},{"comment":"The BPE comparison is performed for the LCM&LCL setting without the task prompt, while the final system includes the task prompt. Reporting BPE with the same task-prompt augmentation would make the comparison against the proposed method fairer.","section":"Table 6"},{"comment":"The sparseness and localness analyses use only 200 training samples and no error bars; they are suggestive diagnostics but should be described as such rather than as definitive explanations of the underlying mechanisms.","section":"Sections 4.2–4.3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is within the journal's scope and the self-supervised construction is not circular. The main risk is statistical: all quantitative claims rest on single-run BLEU scores, and the reported gains are within the range of the paper's own hyperparameter sensitivity. If the authors can provide multi-seed results or significance tests, and clarify the task-prompt ablation, the paper could be acceptable. The norm-unit dependency and the unsupported state-of-the-art claim should also be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper introduces a 'unit language'—an unsupervised n-gram segmentation of discrete speech units into pseudo-words, used as auxiliary multi-task supervision in textless speech-to-speech translation. The construction, maximum-likelihood segmentation with dynamic programming, is genuinely different from the BPE-based pseudo-languages in prior work, and the authors include a reproduced BPE baseline as a control. That is the real contribution, and it is a reasonable one.\n\nThe experimental core is honest in scope: four VoxPopuli directions, consistent gains of 0.4–1.2 BLEU over a reproduced S2UT baseline, and performance matching text-supervised multi-task training. The unit language edges out BPE by about 0.5 BLEU on average. The sparseness and localness analyses are a bit hand-wavy but give a plausible mechanism for why cross-modal and cross-lingual losses help and why they conflict.\n\nThe soft spots are real but not fatal. The biggest is statistical: every number in the main tables is a single run with no error bars or significance tests. A 0.4–1.2 BLEU difference is often within seed-to-seed variance in S2ST, so the abstract's 'significant' overstates what is shown. Second, the method depends heavily on speaker-normalized units; Appendix G shows the average gain collapses to +0.2 BLEU without them. That makes the headline result conditional on a preprocessing step that may not transfer to other languages or datasets. Third, the task prompt uses a negative-weight MSE loss (-3.0) that is never ablated, so the final +1.2 average is partly driven by an unexamined hyperparameter. Fourth, the 'state-of-the-art' claim should be read narrowly: Seamless scores far higher on the same pairs, so this is SOTA only among textless baselines, and even that is caveated by the norm-unit dependency.\n\nBottom line: the underlying idea is new, clearly described, and the experiments include a useful control. The paper deserves a serious referee, but the authors should be asked for multi-seed runs, confidence intervals, an ablation of the negative prompt weight, and a qualified SOTA claim. People working on unit-based speech models or unwritten languages will get value from this.","headline":"A genuinely new pseudo-text construction for textless S2ST with modest, single-run gains; worth a serious referee if the authors address variance and the norm-unit dependency.","tokens_in":15239,"tokens_out":3157,"would_cite":true,"duration_ms":25835,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an unsupervised pseudo-text mined from discrete speech units — the 'unit language' — lets a textless speech-to-speech translation model match the BLEU of a model trained with real text on VoxPopuli.","keywords":["speech-to-speech translation","textless S2ST","unit language","n-gram language modeling","multi-task learning","task prompt modeling","discrete speech units","VoxPopuli"],"falsifier":"Run the full recipe on a language pair where no speaker-normalization model exists, using raw pretrained discrete units only; if the average BLEU gain over the baseline stays near the +0.2 reported in Table 11 rather than +0.9, the unit-language gains are driven by the unit normalizer, not by the n-gram pseudo-text. A second check is to vary K and r as in Tables 4 and 5 and see whether any setting on a new pair recovers the +0.9 gain.","tokens_in":14174,"feed_emoji":"🎙️","tokens_out":9137,"duration_ms":68920,"temperature":0.7,"pith_summary":"Textless speech-to-speech translation (S2ST) usually has to learn both what speech means and how languages align without any written text. This paper claims that a text-like tokenization of discrete speech units, which it calls a unit language, can provide that missing guidance. The unit language is mined unsupervised with 2-gram language modeling: contiguous speech units are grouped into pseudo-words by maximizing their probability, giving a compressed, text-like sequence. The authors train their textless S2ST model with two auxiliary losses — one predicting the source unit language to sharpen acoustic-to-meaning (cross-modal) learning, one predicting the target unit language to sharpen language alignment (cross-lingual learning) — and add task prompts to stop these two losses from fighting. On four VoxPopuli translation directions, the method reaches 21.5 average BLEU, matching a model trained with real text and beating the 20.3 strong baseline.","feed_headline":"Unit language lifts textless speech translation to text-level BLEU","feed_subtitle":"Mining n-gram 'unit words' from discrete speech adds 1.2 BLEU over a strong baseline","key_machinery":"The load-bearing object is the unit language, a text-like transcription built by n-gram language modeling over discrete speech units. Given a unit sequence, the algorithm merges at most K contiguous units into pseudo-words, choosing the segmentation that maximizes the sequence probability under a 2-gram model whose conditional probabilities are estimated by counting in the unit corpus; equations (4)-(8) give the dynamic-programming recursion. The pseudo-words are then fed to two extra decoder heads: a source text decoder for the cross-modal loss and a target text decoder for the cross-lingual loss, with a learnable task prompt (bCM or bCL) inserted at the r-th encoding layer to keep the two tasks from interfering. The unit language's compression (sequence length between characters and text) is what makes alignment tractable, and the norm-unit preprocessing makes the units clean enough for the n-gram statistics to be meaningful.","core_discovery":"The central claim is that unsupervised pseudo-text can substitute for real transcription in S2ST training. The unit language groups every sequence of discrete speech units into pseudo-words via a maximum-likelihood n-gram segmentation (2-gram, at most K=3 units per word), using dynamic programming over the recursion in equations (4) and (7). When these pseudo-words are used as auxiliary decoder targets — the source unit language for cross-modal modeling and the target unit language for cross-lingual modeling — the textless model improves by 1.2 average BLEU over the reproduced strong textless baseline and reaches parity with the same model trained on recognized text (21.5 vs. 21.5). The paper further claims that the two auxiliary tasks operate differently — CM filters noise, CL captures semantics — and that their simultaneous use conflicts; task-prompt modeling, two learnable vectors inserted at the r-th encoder layer with a negative mean-square-error diversity term, resolves the conflict and yields the best results. The authors also report that the gain depends on speaker-normalized units: without normalization the average improvement falls from +0.9 to +0.2 BLEU.","pith_inferences":["Editorial extension: the same n-gram mining could produce pseudo-text for other speech tasks — speech summarization, speech-to-text translation without transcripts, and TTS from unwritten languages — since the segmentation is unsupervised and language-agnostic.","Editorial extension: the observed CM/CL conflict suggests a design rule for multi-task speech models: inject text-like supervision at different encoder depths, with task prompts and a diversity penalty as one generalizable instance.","Editorial extension: a testable next step is higher-order n-grams with pruning; the paper keeps to 2-gram for computational cost (K=4 already needs about two days), and a pruned 3-gram unit language could compress further and improve cross-lingual alignment.","Editorial extension: the claimed parity with text-trained models should be checked beyond BLEU; the paper itself notes the lack of human evaluation of tone and fluency."],"forward_implications":["If the claim holds, textless S2ST no longer needs real transcriptions to reach text-level BLEU: the unit language provides the same guidance that recognized text provides.","The conflict between cross-modal and cross-lingual auxiliary losses is real and general, and task prompts offer a mechanism to reconcile depth-split supervision in a single encoder.","Unit language is a drop-in auxiliary target: source unit language improves noise filtering, target unit language improves semantic alignment, and both contribute to the full gain.","Because the unit language compresses unit sequences toward text length, it can serve as pseudo-text for any speech sequence, not only translation pairs.","Gains depend on speaker-normalized units, so the method inherits the portability of unit normalization; languages without a norm-unit model may need an unsupervised normalizer first."],"supporting_citations":[{"why":"Defines the textless S2UT baseline, the speaker-normalized 'norm unit' preprocessing, and the unit vocabulary that the unit language is mined from.","marker":"Lee et al. (2022b)"},{"why":"Supplies the HuBERT self-supervised model whose discrete units are the raw material for the unit language.","marker":"Hsu et al. (2021)"},{"why":"Provides the VoxPopuli multilingual corpus used for all four translation-direction experiments.","marker":"Wang et al. (2021)"},{"why":"Serves as the text-based S2ST system whose BLEU scores mark the 'comparable to text' target.","marker":"Barrault et al. (2023)"},{"why":"The Transformer backbone that hosts the acoustic encoder, textual encoder, and all decoders.","marker":"Vaswani et al. (2017)"},{"why":"SentencePiece limits the unit-language vocabulary to 10k tokens for training.","marker":"Kudo and Richardson (2018)"},{"why":"Acoustic BPE pseudo-language is the reproduced baseline that the unit language must beat in Table 6.","marker":"Shen et al. (2024)"},{"why":"Supplies the analysis that speech representations are converted to text in the textual encoder, motivating the r-layer insertion and length-reduction benefit.","marker":"Zhang et al. (2023)"}],"fun_headline_variants":["Pseudo-text from speech units matches text-trained BLEU","Task-prompting settles unit conflict, +1.2 BLEU on textless S2ST","Unit language: textless S2ST achieves text-level parity without transcripts","N-gram pseudo-words guide speech translation to text-level BLEU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire improvement rests on having speaker-normalized discrete units; when the unit language is built from unnormalized units, the average BLEU gain collapses from +0.9 to +0.2, so if normalization is unavailable or fails to transfer to a new language the method's benefit mostly disappears even though the algorithm is unchanged.","fun_headline_variants_meta":{"raw":{"variants":["Pseudo-text from speech units matches text-trained BLEU","Task-prompting settles unit conflict, +1.2 BLEU on textless S2ST","Unit language: textless S2ST achieves text-level parity without transcripts","N-gram pseudo-words guide speech translation to text-level BLEU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1448,"prompt_tokens":942,"completion_tokens":506,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":421}},"tokens_in":558,"tokens_out":506,"duration_ms":4767,"temperature":1.0,"reasoning_tokens":421,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:19:39.352611+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full recipe on a language pair where no speaker-normalization model exists, using raw pretrained discrete units only; if the average BLEU gain over the baseline stays near the +0.2 reported in Table 11 rather than +0.9, the unit-language gains are driven by the unit normalizer, not by the n-gram pseudo-text. A second check is to vary K and r as in Tables 4 and 5 and see whether any setting on a new pair recovers the +0.9 gain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Acoustic BPE pseudo-language is the reproduced baseline that the unit language must beat in Table 6."}],"review_version":1}