{"id":"3753d68b-4a90-4dca-ab6e-e13afc98ba2e","arxiv_id":"2506.02414","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"StarVC is an autoregressive voice conversion model that generates text tokens before acoustic tokens, improving linguistic fidelity while retaining speaker similarity.","lead":"StarVC is a voice conversion system that first predicts the text of what was said, then generates the audio in a target speaker's voice. It reports better word accuracy and comparable perceived quality than three existing systems, while also producing a transcription as a byproduct.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'w/o text token' ablation in Table 1 / §3.3.3 removes ASR pretraining and the joint ASR-VC objective along with text-token conditioning, so the central claim that explicit text prediction drives the gains is not yet isolated.","rationale":"The reader's weakest assumption concerns the sufficiency of frozen pretrained components; my load-bearing concern is different: the main ablation intended to demonstrate the causal benefit of text-token prediction is confounded with the presence of ASR pretraining and the joint ASR-VC objective. The paper is a plausible, clearly described system, but the evidence for its central design claim is weaker than the prose suggests. This does not require rejecting the paper; it requires one controlled ablation or an explicit statement of what the 'w/o text token' run actually trained. Since the reader already issued a CONDITIONAL verdict, I would keep that verdict rather than moving to ACCEPT or REJECT. The concern is internal to the experimental design, not an attack on the authors, and it is testable without retraining the entire system from scratch if the original ablation artifacts are available.","tokens_in":8172,"tokens_out":6095,"duration_ms":59377,"concrete_test":"Specify and run an ablation that keeps the text branch and its losses identical (same ASR pretraining, same 20% ASR batches, same text-token loss) but prevents acoustic-token positions from conditioning on the generated text tokens—e.g., mask text tokens in the context for acoustic positions while leaving them in the context for the text loss. Compare WER, CER, SECS-Res, and SECS-WavLM on the same 200 source/target LibriTTS test-clean split. If this variant matches the current 'w/o text token' row (WER 7.30, SECS-Res 0.771), the central claim is unsupported; if it matches the full model (WER 6.27, SECS-Res 0.835), the concern is resolved. If the original 'w/o text token' run is still available, reporting its exact training schedule (specifically whether Stage 1 ASR pretraining was run) would also help.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3.3 presents the key evidence for the text-conditioning claim: dropping text tokens raises WER from 6.27% to 7.30% and lowers SECS-Res from 0.835 to 0.771. But in the architecture described in §2.1.2 and the three-stage recipe in §2.2, text tokens are not an add-on that can be removed while keeping the rest of the pipeline fixed. Stage 1 (ASR pretraining) optimizes CE(y_t, ŷ_t) on text tokens; Stage 3 allocates 20% of batches to the same ASR objective. Removing the text-token output eliminates both objectives and, as described, also eliminates the ASR pretraining stage itself. The ablation therefore conflates (a) acoustic tokens being conditioned on predicted text, (b) the extra ASR pretraining, and (c) the multi-task ASR loss during joint training. Since (b) and (c) are commonly beneficial regularizers/pretraining for semantic encoding, the observed WER/SECS drop cannot be attributed specifically to text-conditioned acoustics. The baseline comparisons in Table 1 do not resolve this: CosyVoice, OpenVoice V2, and TriAAN-VC differ from StarVC in architecture, training data, and codec, so they are not controlled tests of the text-token mechanism. Thus the paper's central design claim is currently supported only by a confounded ablation.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes StarVC, an autoregressive voice-conversion framework that generates text tokens before acoustic tokens, using a Whisper-small encoder for semantic features, an ERes2Net-large encoder for target-speaker features, a Qwen2.5-style language model, and Mimi codec layers for reconstruction. Training is split into three stages: ASR pretraining, VC training, and joint ASR-VC training, with additional data augmentation from OpenVoice V2. On 400 LibriTTS test-clean utterances, StarVC reports the best WER (6.27%), CER (4.09%), and MOS scores among the compared methods, while also producing text tokens with WER-Text 4.95% and CER-Text 1.51%. The central design claim is that explicitly predicting text before acoustic tokens improves linguistic consistency and speaker preservation.","tokens_in":8457,"tokens_out":4229,"duration_ms":43391,"significance":"The idea of inserting an explicit text-token prediction step into an autoregressive voice-conversion pipeline is timely and potentially useful, especially as speech-text foundation models become more common. If the mechanism were cleanly isolated, the paper would offer a practical recipe for unifying ASR and VC in one model. Strengths of the reported study include a reasonably detailed architecture description, a multi-metric evaluation protocol, an audio demo, and the reporting of both objective and subjective results. Weaker points are the confounded ablation behind the central claim, an abstract-level claim about speaker similarity that Table 1 does not support, and the absence of controlled comparisons that could separate the text-conditioning mechanism from multi-task ASR training. The paper is empirical rather than theoretical; none of the reported losses are circular with the evaluation metrics, which are computed with external tools (Whisper-large, Resemblyzer, WavLM).","major_comments":[{"comment":"The abstract and Section 3.3.1 claim that StarVC outperforms conventional VC methods in preserving speaker characteristics, but Table 1 shows CosyVoice achieving higher SECS-Res (0.839 vs. 0.835) and SECS-Wavlm (0.478 vs. 0.472). The claim is true only for the subjective SMOS in Table 2. Please restrict the speaker-characteristic claim to the metrics that actually support it, or provide confidence intervals or significance tests showing that the small SECS differences are not meaningful.","section":"Abstract and Table 1"},{"comment":"The 'w/o text token' ablation is confounded. In the described training recipe, removing the text-token output also removes Stage 1 ASR pretraining (L_ASR = CE(y_t, ŷ_t)) and the ASR component of Stage 3 joint training. Therefore the observed WER increase from 6.27% to 7.30% and the SECS-Res drop from 0.835 to 0.771 cannot be attributed specifically to text-conditioned acoustic generation; they may equally reflect the loss of ASR pretraining or multi-task regularization. Since the central claim is that explicit text prediction drives the gains, please add an ablation that retains the ASR objectives (or at least the ASR pretraining stage) while not conditioning acoustic tokens on predicted text, or clearly report a 'w/o text token but with ASR pretraining' condition.","section":"Section 3.3.3 and Section 2.2"},{"comment":"The comparison to CosyVoice, OpenVoice V2, and TriAAN-VC is not a controlled test of the text-token mechanism, because these systems differ in architecture, codec, training data, and objective functions. The paper cites LM-VC and DualVC3 in the introduction but does not include them in the evaluation. To support the claim that explicit text prediction is what improves content preservation, the authors should either include LM-based VC baselines that are closer in architecture or rely on within-model ablations that isolate the text-conditioning component.","section":"Section 3.2 and Table 1"},{"comment":"The loss weights w and w' and the schedule of the multi-stage training are never specified, and the 'w/o multi-stage' ablation is not defined precisely. Without knowing what remains in that ablation (only VC training? only joint training? which loss weights?) and what the exact coefficients are, the claimed contribution of multi-stage training cannot be assessed. Please report the numeric values of w and w', the definition of the 'w/o multi-stage' condition, and the training configuration of the 'smaller model' row.","section":"Section 2.2 and Table 1"}],"minor_comments":[{"comment":"The delay method from MusicGen is mentioned but not precisely specified: it would help to state the exact delay pattern used between text tokens and the eight acoustic codebook layers, and to clarify how subword text tokenization interacts with frame-level acoustic tokens.","section":"Section 2.1.2"},{"comment":"The ratio between real and OpenVoice V2-synthesized parallel data is reported as 50%/50% in VC training and 80%/20% in joint training, but it is not stated whether the same target speaker can appear with both real and synthesized references or how the model handles speaker-identity conflicts; a sentence clarifying the data construction would be useful.","section":"Section 2.3"},{"comment":"The evaluation set is limited to 400 LibriTTS test-clean utterances with 200 source and 200 target utterances; this is acceptable for a conference paper, but the authors should note that the test set is a single-domain clean English condition and that the generalization to noisy or multilingual speech is untested.","section":"Section 3.2"},{"comment":"The table format for WER and CER columns (e.g., '8.24% / 4.27%') is ambiguous: it appears to show two metrics in one column. Separating WER and CER into two clear columns would improve readability.","section":"Table 1"},{"comment":"The MOS confidence intervals are reported, but no significance tests are provided for the objective metrics. Given that some SECS differences are small (e.g., 0.835 vs. 0.839), a paired test or bootstrap confidence intervals would strengthen the objective claims.","section":"Section 3.3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a system paper with an interesting premise and mostly clean experimental reporting, but the central claim relies on an ablation that conflates text-token conditioning with ASR pretraining and multi-task learning. The abstract also overstates the speaker-similarity results relative to Table 1. I would be willing to support acceptance after the authors isolate the text-token mechanism with a better ablation and correct the overclaims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. StarVC is a clean transfer of the Moshi-style text-then-speech autoregressive recipe to voice conversion, and on the reported numbers it does improve intelligibility over the baselines. The main caveat is that its flagship ablation doesn't isolate the thing it claims to isolate: dropping text tokens also removes the ASR pretraining stage and the 20% ASR objective in joint training, so the WER gain could be from extra ASR training rather than from text-conditioned acoustics.\n\nWhat's actually new: among the cited VC methods, none explicitly predict text tokens before acoustic tokens. The architecture—Whisper-small encoder, ERes2Net speaker embedding, Mimi codec, a Qwen-style LM, three-stage training—is described clearly and is sensible. The results are credible: StarVC gets the lowest WER (6.27%) and CER (4.09%), close to CosyVoice on speaker similarity, and the best MOS. The 95% CIs on MOS are a plus. It also produces text tokens at 4.95% WER, which is a genuinely useful byproduct. The paper is honest about most of its numbers; the analysis section acknowledges CosyVoice's slightly higher SECS.\n\nSoft spots. First, the abstract overstates: \"outperforms conventional VC methods in preserving speaker characteristics\" is contradicted by Table 1, where CosyVoice edges StarVC on both SECS metrics. Second, and more important, the \"w/o text token\" ablation is confounded. In the described training recipe, text-token output is the target for Stage 1 ASR pretraining and for 20% of Stage 3 batches. Removing the text stream removes both of those, so the observed drop from 7.30% to 6.27% WER could reflect the value of the extra ASR training rather than the value of conditioning acoustics on predicted text. A cleaner ablation would keep the ASR objectives while replacing the text-token conditioning with, say, an unpredicted text embedding or a text-free encoder. Third, objective metrics have no confidence intervals and are computed on 200 conversion pairs. Fourth, no code or weights, and the demo page isn't enough for reproduction. The reliance on frozen pretrained components is a fair concern but not a fatal one.\n\nWho this is for: people working on VC or on speech-text joint modeling will get a useful data point. The paper deserves serious peer review. The reviewers should push for the cleaner ablation and for tempering the abstract. As it stands, the evidence supports \"text+ASR multi-task helps\" more strongly than \"explicit text prediction helps,\" and the authors need to disentangle those.","headline":"A clean transfer of the Moshi-style text-then-speech recipe to voice conversion, with credible gains in intelligibility, but the key ablation is confounded because removing text tokens also removes ASR pretraining and the joint ASR objective.","tokens_in":9043,"tokens_out":2620,"would_cite":true,"duration_ms":24771,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"StarVC is a unified autoregressive framework that generates text tokens before acoustic tokens, and it reports the lowest word and character error rates among the compared voice-conversion systems while matching the best speaker similarity.","keywords":["voice conversion","joint text and speech generation","autoregressive language model","text-first conditioning","speaker disentanglement","content preservation","audio codec tokens","multi-stage training"],"falsifier":"A concrete check would be to compare per-utterance accuracy of the model's generated text with accuracy of its converted audio: if the central claim is right, utterances where the predicted transcript is wrong should also show degraded acoustic WER, and corrupting the text prediction should propagate into the converted speech; if the two are decoupled, the text step is not the load-bearing mechanism.","tokens_in":7936,"feed_emoji":"🎤","tokens_out":9190,"duration_ms":77180,"temperature":0.7,"pith_summary":"This paper tries to establish that voice conversion improves when the model is forced to predict a written transcription before it generates the converted audio. The proposed system, StarVC, uses a language model that generates tokens one at a time: it emits text tokens first and then eight layers of audio codec tokens, grounding the speech synthesis in an explicit lexical plan rather than in soft semantic vectors. On a held-out set of 400 clean utterances, this design reports the lowest word error rate (6.27%) and character error rate (4.09%) among the systems compared, while its speaker similarity and naturalness scores are at least tied with the strongest alternatives. The same pipeline also outputs its own transcript of the converted speech, with 4.95% word error and 1.51% character error. If the claim holds, explicit text prediction is a structural choice that helps disentangle who is speaking from what is said.","feed_headline":"Text-first voice conversion cuts word errors to 6.27%","feed_subtitle":"Generating a transcript before the audio keeps words intact and speaker likeness strong.","key_machinery":"The load-bearing mechanism is a delayed text-first autoregressive sequence: the model writes one text token, then the first acoustic token, then the next text token, then the next acoustic tokens, using the MusicGen-style delay pattern so every acoustic decision is conditioned on an explicit lexical prefix. Around this sit three frozen components—a semantic encoder taken from a pretrained speech-recognition system, a speaker encoder, and an eight-layer open audio codec—together with trainable adapters and a 24-layer transformer language model. A three-stage training schedule (ASR pretraining, voice-conversion training, then joint ASR-VC training) is what lets the same weights serve both transcription and conversion.","core_discovery":"StarVC's central claim is that inserting an explicit text-generation step into an autoregressive voice-conversion pipeline improves both content preservation and speaker transfer. The model takes semantic features from a frozen speech-recognition encoder and a single speaker embedding from a frozen speaker encoder, then autoregressively produces a sequence of text tokens followed by acoustic tokens from an eight-layer neural audio codec. Because the text tokens are scheduled one step ahead of the first acoustic token, the audio stream is conditioned on an already-decided transcript rather than on latent semantic vectors. In the reported experiments this mechanism delivers the best intelligibility among the compared systems, with 6.27% word error rate and 4.09% character error rate on converted speech, while speaker similarity is effectively on par with the strongest baseline (SECS 0.835 and 0.472 versus 0.839 and 0.478) and subjective similarity and naturalness are the highest (SMOS 3.98, NMOS 4.17). The same pipeline emits its own transcript at 4.95% word error and 1.51% character error, which the paper reads as evidence that the text tokens themselves are accurate.","pith_inferences":["The paper leaves implicit that the same text-first design could transfer to speech-to-speech translation or zero-shot text-to-speech, where lexical fidelity matters as much as timbre transfer; the architecture does not depend on the voice-conversion setup specifically.","Because the generated-text error (4.95% WER) is lower than the converted-audio error (6.27% WER), a portion of the remaining intelligibility loss is probably introduced after text prediction, inside the acoustic codec or decoder; studying which acoustic tokens deviate would locate the bottleneck.","The comparison is against baselines that do not generate text; a fair boundary test would be to compare StarVC with a text-aware diffusion or semantic-token voice conversion system to see whether the advantage comes from text prediction itself or from the specific codec and training data.","The MOS test uses 20 listeners and 20 source-target pairs; a larger listening panel would be needed to confirm that the small speaker-similarity gap (3.98 versus 3.97) is perceptible rather than noise."],"forward_implications":["Voice conversion systems built this way can output a readable transcript as a natural byproduct, removing the need for a separate post-hoc ASR pass for captioning or dialogue logging.","Because acoustic tokens are decoded from an explicit word sequence, content preservation is less dependent on source prosody and timbre, which may reduce timbre leakage in cross-speaker transfers.","The ablations imply that multi-stage training and a sufficiently large language model are needed for the benefit; smaller or single-stage versions lose much of the intelligibility gain.","The text-token stream gives the model a direct handle for correction: a listener can repair a mis-transcribed word and have the audio resynthesized with the corrected content."],"supporting_citations":[{"why":"Supplies the frozen semantic encoder, a pretrained speech-recognition model whose features must carry exact lexical content into the language model.","marker":"[18]"},{"why":"Supplies the frozen speaker encoder used to extract the target speaker embedding from a single reference clip.","marker":"[19]"},{"why":"Motivates the eight-layer codec setting and demonstrates joint speech-text generation in a streaming speech model.","marker":"[15]"},{"why":"Provides the delayed-token scheduling that places text tokens one step ahead of the first acoustic token.","marker":"[20]"},{"why":"Serves both as a comparison baseline for intelligibility and as the synthesizer for parallel data augmentation during training.","marker":"[12]"},{"why":"Are the diffusion-based voice conversion baseline whose 8.24% WER anchors the main intelligibility comparison.","marker":"[33, 34]"},{"why":"Is the any-to-any voice conversion baseline whose high WER marks the cost of weaker content disentanglement.","marker":"[36]"},{"why":"Supplies the large transcribed multi-domain corpus used in ASR pretraining and joint training.","marker":"[30]"},{"why":"Supplies the clean multi-speaker corpus used for training and the held-out test utterances.","marker":"[31, 32]"}],"fun_headline_variants":["StarVC: text-first VC keeps words and speaker intact","Predict transcript, then audio: StarVC's VC edge","Text-first auto-regressive VC improves fidelity","Voice conversion that writes before it speaks","StarVC: lower WER via text-before-audio design"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system depends on its frozen pretrained components being the right carriers: the speech-recognition encoder must preserve exact words in its semantic features, the speaker encoder must capture the target voice from one reference clip, and the eight-layer audio codec must retain enough detail for natural reconstruction; if any of these representations is mismatched with the conversion objective, the text-conditioning benefit shrinks.","fun_headline_variants_meta":{"raw":{"variants":["StarVC: text-first VC keeps words and speaker intact","Predict transcript, then audio: StarVC's VC edge","Text-first auto-regressive VC improves fidelity","Voice conversion that writes before it speaks","StarVC: lower WER via text-before-audio design"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000424,"raw_usage":{"total_tokens":2158,"prompt_tokens":908,"completion_tokens":1250,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":1174}},"tokens_in":524,"tokens_out":1250,"duration_ms":11050,"temperature":1.0,"reasoning_tokens":1174,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:24:01.698476+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to compare per-utterance accuracy of the model's generated text with accuracy of its converted audio: if the central claim is right, utterances where the predicted transcript is wrong should also show degraded acoustic WER, and corrupting the text prediction should propagate into the converted speech; if the two are decoupled, the text step is not the load-bearing mechanism.","supporting_citations":[{"cited_title":"Lm-vc: Zero- shot voice conversion via speech generation based on language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen speaker encoder used to extract the target speaker embedding from a single reference clip."},{"cited_title":"Sef-vc: Speaker embedding free zero-shot voice conversion with cross attention,","cited_arxiv_id":null,"evidence_quote":"Motivates the eight-layer codec setting and demonstrates joint speech-text generation in a streaming speech model."},{"cited_title":"DualVC 3: Leveraging Language Model Generated Pseudo Context for End-to-end Low Latency Streaming Voice Conversion","cited_arxiv_id":"2406.07846","evidence_quote":"Provides the delayed-token scheduling that places text tokens one step ahead of the first acoustic token."},{"cited_title":"GigaSpeech covers diverse domains and acoustic conditions, while LibriTTS is cleaner and more consistent","cited_arxiv_id":null,"evidence_quote":"Supplies the large transcribed multi-domain corpus used in ASR pretraining and joint training."}],"review_version":1}