{"id":"b89f94cf-6662-4fa0-b719-b70dedb5d42b","arxiv_id":"2507.14534","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Conan achieves chunkwise online zero-shot voice conversion, preserving source content while adopting the reference speaker's timbre and style, with a latency as low as 37 milliseconds.","lead":"A new voice conversion system, Conan, turns a speaker's voice into another person's voice in real time, processing audio in 20 to 80 millisecond chunks. It is designed for live calls and entertainment, where you cannot wait for the whole sentence before starting to speak.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim of outperforming StreamVC rests on borrowed, unmatched metrics; without a matched comparison the headline superiority over the only online baseline is unverified.","rationale":"The reader's verdict is CONDITIONAL and identifies the unmatched StreamVC comparison as the weakest assumption. I agree: this is the single most load-bearing concern because the paper's abstract and conclusion claim superiority over baselines, and StreamVC is the only online baseline. If the StreamVC numbers are not directly comparable, the main real-time advantage claim is unsupported. The internal evidence for protocol mismatch is concrete: Conan's own Source CER is 2.43%, yet the borrowed StreamVC CER is 2.17%, which would imply the converted speech has lower character error than the original source under identical evaluation, a red flag that the evaluation sets or ASR pipelines differ. The proposed concrete test would settle the issue by computing all metrics on identical utterances with identical protocols. I do not see an internal inconsistency in the architecture itself: the streaming content extractor, the adaptive style encoder with CVQ, and the causal pixel-shuffle vocoder are coherently described, and the ablation study supports the contribution of each component. The missing code and model weights are secondary but real verification barriers. Thus I do not move the verdict; the appropriate disposition remains conditional on matched evaluation and artifact release.","tokens_in":11936,"tokens_out":3784,"duration_ms":52350,"concrete_test":"Obtain StreamVC outputs for the same LibriTTS test-clean source utterances and VCTK reference speakers, either from the authors, from an official release, or from a faithful third-party reproduction. Then recompute WER/CER with the same HuBERT-Large ASR pipeline, SIM with Resemblyzer, and run a matched MOS/CMOS listening test on the identical utterance pairs and with the same listeners. If StreamVC's numbers shift by more than the observed gaps (e.g., roughly 0.2 WER points or 0.1-0.2 MOS points), the claimed superiority over StreamVC is not established; if they remain within noise, the comparison survives. In the absence of access, the paper should explicitly downgrade the StreamVC comparison to non-matched and avoid claiming a performance advantage over it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Conan outperforms baseline models depends heavily on Table I, where the only online streaming baseline is StreamVC. Section IV-B states: 'Since StreamVC is not open-sourced, to ensure a fair comparison, we adopt the same dataset split and objective evaluation protocols that they used. We rely on the objective metrics reported in their paper and perform subjective assessments based on their demo pages.' This is not a matched comparison. Reported WER/CER/MOS values can shift with ASR version, resampling, utterance filtering, prompt wording, or listener pools; the observed differences over StreamVC are small (WER 6.02 vs 6.22, MOS-S 4.02 vs 3.86, MOS-Q 4.06 vs 4.01). The table itself hints at protocol mismatch: the reported Source CER is 2.43%, while StreamVC's CER is 2.17%, i.e., the converted StreamVC output is claimed to have lower character error than the original source on the same test set, which is implausible unless the evaluation sets or ASR pipelines differ. Because StreamVC is the only online baseline, an invalid StreamVC comparison would reduce the support for 'outperforms baseline models' to offline baselines only, weakening the paper's core real-time claim. The lack of released code or model weights further prevents external verification of the Conan numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Conan, a chunkwise online zero-shot voice conversion system with three main components: a Stream Content Extractor based on Emformer that distills HuBERT content labels for low-latency streaming content encoding, an Adaptive Style Encoder that uses clustering vector quantization and alignment attention to transfer reference-speaker style, and a Causal Shuffle Vocoder that makes HiFi-GAN fully causal via pixel-shuffle upsampling. The system is evaluated on LibriTTS and VCTK with objective metrics (WER, CER, SIM) and subjective MOS/CMOS tests, and the authors report a fast configuration with 37 ms latency. The central claim is that Conan outperforms baseline models in content accuracy, speaker similarity, and quality while operating online.","tokens_in":12238,"tokens_out":2465,"duration_ms":30064,"significance":"If the central claim is established, Conan would be a useful contribution to real-time zero-shot voice conversion, particularly because it addresses three recognized difficulties: streaming content extraction, fine-grained style transfer, and causal high-quality vocoding. The architectural ideas are clearly motivated, and the Causal Shuffle Vocoder is an elegant way to avoid transposed-convolution artifacts in a causal setting. The latency measurements are valuable and the ablation study is a strength, as it isolates the contribution of each component. However, the evaluation protocol for the only online baseline, StreamVC, is not a matched comparison, and the content metric is partly aligned with the training target; these issues currently weaken the evidence for the headline superiority claim.","major_comments":[{"comment":"The claim that Conan outperforms StreamVC, the only online baseline, rests on an unmatched comparison. The paper states that StreamVC is not open-sourced and that the authors 'rely on the objective metrics reported in their paper and perform subjective assessments based on their demo pages.' Different ASR versions, resampling, utterance filtering, and listener pools can shift WER, CER, and MOS by amounts comparable to the reported differences (e.g., WER 6.02 vs. 6.22, MOS-S 4.02 vs. 3.86). The implausible internal consistency check in Table I—StreamVC's CER (2.17%) is lower than the Source CER (2.43%) on the same test set—strongly suggests that the evaluation sets or ASR pipelines are not actually identical. This undermines the paper's central claim of superiority over the only streaming baseline. The authors should either re-run StreamVC under the same protocol, or explicitly restrict the superiority claim to the offline baselines and reframe the streaming comparison as suggestive.","section":"IV-B, Table I"},{"comment":"The content metric is partly circular. The Stream Content Extractor is trained by distilling HuBERT content labels (Section III-B), and content accuracy is measured with a HuBERT-Large ASR model (Section IV-A). Since both the training target and the evaluation metric are derived from HuBERT, low WER/CER may reflect how well the extractor mimics HuBERT rather than how faithfully linguistic content is preserved for human listeners. This is a correctness-risk concern for the content-accuracy claims, not merely a philosophical objection. The authors should add a human intelligibility test or use an ASR model from a different family (e.g., Whisper) to verify that the content-accuracy advantage is not an artifact of the HuBERT alignment.","section":"III-B and IV-A"},{"comment":"Objective results are reported without any measure of variance or statistical significance. WER, CER, and SIM in Table I are point estimates, and the subjective MOS values carry 95% confidence intervals, but the objective differences among the strongest systems are small (e.g., WER 6.02% vs. 7.14% for QuickVC, SIM 85.71% vs. 81.47% for Diff-VCTK). Without confidence intervals, significance tests, or multiple utterance-level resampling, the reader cannot tell whether these differences are stable. The authors should report objective metrics with confidence intervals or bootstrap significance, or at least specify the number of utterances and any test-set construction that would support the comparison.","section":"IV-A and IV-B"}],"minor_comments":[{"comment":"The abstract states a latency 'as low as 37 ms,' but Table II reports 139.71 ms for the full setting and 36.87 ms for the fast setting. The abstract should clarify that the 37 ms figure corresponds to the fast, strictly causal configuration.","section":"Abstract and Section IV-B"},{"comment":"In Eq. (6), the notation 'arg max_{1≤j≤J} [Softmax(U C_i^{(N)})]^{(j,t)}' is ambiguous because the softmax is taken over the class dimension but the (j,t) indexing is not fully defined. Please clarify the exact tensor dimensions and the role of t.","section":"III-B, Eq. (6)"},{"comment":"The component name is inconsistent: 'Stream Content Extractor' appears in the abstract and Section III, while 'Stream Content Encoder' appears in the ablation study (Table III). Please unify the terminology.","section":"Throughout"},{"comment":"The explanation that StreamVC's CER advantage comes from reusing source pitch and energy is presented without supporting evidence. If this claim is retained, it should be backed by an experiment or a citation; otherwise it should be softened.","section":"IV-B"},{"comment":"The related-work section lists many style-modeling methods, but the positioning of Conan relative to recent streaming style-transfer work (e.g., Yang et al. [5]) could be sharper: specifically, what does the Adaptive Style Encoder capture that the GST-based approach in [5] does not?","section":"II-C"}],"recommendation":"major_revision","confidential_remarks":"The architecture is well motivated and the latency results are useful, but the evaluation protocol for StreamVC is not matched and the content metric is partly circular. These are fixable within the manuscript's scope: the authors can strengthen the streaming comparison, add a non-HuBERT ASR or human intelligibility check, and provide variance estimates for objective metrics. I would also encourage the authors to consider releasing code or audio samples for all test utterances, since the current demo page does not substitute for a controlled evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the causal pixel-shuffle vocoder is a genuine architectural idea, and the Conan system is a clean integration of streaming content extraction, adaptive style encoding, and causal synthesis. The paper is well written and the ablation study gives each component a fair shake. But the central claim that Conan outperforms StreamVC is not supported by a matched comparison. The authors transparently say they borrowed StreamVC's reported metrics and did subjective ratings from demo pages. That is not a controlled test. Table I even has a red flag: StreamVC's CER (2.17%) is lower than the source CER (2.43%) on the same test set, which is implausible for any VC system unless the ASR or preprocessing differs. So the superiority over the only online baseline is unverified.\n\nThe offline baselines may have been run in-house, but the paper never says so explicitly, so the reader can't tell if those comparisons are matched either. Objective metrics come without variance or significance tests. The MOS numbers have confidence intervals but the ratings are based on demo pages, not a formal listening test.\n\nThe content metric is mildly circular: the content extractor is distilled from HuBERT, and content accuracy is measured with a HuBERT-Large ASR. That biases WER/CER in Conan's favor. Speaker similarity and naturalness are human-judged, which gives some independent grounding.\n\nThe authors don't release code or weights, so the numbers aren't externally checkable.\n\nStill, the architecture is sound. Replacing transposed convolutions with pixel shuffle in a causal HiFiGAN is worth building on. The latency numbers are plausible and clearly broken down. The ablation study shows each module contributes, though the w/o SCE result (WER 26.92%) is more a sanity check than an insight.\n\nThis paper is for researchers in streaming VC and speech synthesis. The empirical claims need a redo, but the architectural contribution deserves referee time. I'd recommend sending it to peer review with the requirement that the authors re-run StreamVC (or at least one online baseline) under matched conditions, report error bars, and release evaluation code. If that's impossible, the claims should be scaled back to the architectural contribution.","headline":"A genuinely novel causal pixel-shuffle vocoder and a sensible streaming VC architecture, but the only online baseline comparison rests on borrowed metrics that don't hold up.","tokens_in":12706,"tokens_out":4393,"would_cite":true,"duration_ms":43368,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Conan converts speech chunk by chunk, keeping the source words while matching an unseen reference voice, with latency as low as 37 ms.","keywords":["zero-shot voice conversion","online voice conversion","streaming speech processing","Emformer","adaptive style encoder","causal shuffle vocoder","pixel shuffle","low-latency synthesis"],"falsifier":"A matched head-to-head test would settle the claim: run Conan and StreamVC on identical source-reference pairs with the same ASR for WER and CER, the same speaker-similarity model, and the same listeners for MOS and CMOS. If Conan does not beat StreamVC on these metrics under identical protocols, the paper's central claim of outperforming baselines fails.","tokens_in":11766,"feed_emoji":"🎙️","tokens_out":7047,"duration_ms":78922,"temperature":0.7,"pith_summary":"Conan is a voice conversion system designed to work live: it takes ordinary speech in chunks, keeps the words and meaning of the original utterance, and makes the voice sound like an unseen reference speaker. The paper argues that the right combination of streaming content extraction, adaptive style encoding, and a fully causal vocoder solves the three hard problems of online zero-shot conversion: content fidelity under latency, transfer of timbre and speaking style, and naturalness at chunk boundaries. If the claims hold, real-time communications and interactive entertainment could use voice conversion with only tens of milliseconds of delay, without giving up the source content or the target voice identity. The paper reports that Conan beats the compared baselines on word error rate, speaker similarity, and naturalness, with latency as low as 37 ms in its fast setting.","feed_headline":"Voice changer streams in 37 ms, keeping words and new voice","feed_subtitle":"Conan converts speech chunk by chunk, preserving source content while matching an unseen speaker's timbre and style.","key_machinery":"The load-bearing mechanism is the chunkwise causal pipeline itself, but the component that carries the quality claim is the Causal Shuffle Vocoder: instead of transposed convolutions, it projects the mel features to more channels and rearranges them in time with pixel shuffle, so upsampling never looks at future frames and never creates checkerboard artifacts. The Stream Content Extractor supplies the same role HuBERT plays offline by using Emformer's memory-bank attention to keep cross-chunk context. The Adaptive Style Encoder supplies the unseen speaker's detailed style through a clustered vector-quantization bottleneck plus an align-attention step that fuses style with the content and timbre embeddings. Together these let each 20 ms or 80 ms chunk be converted independently while a sliding context window smooths the boundaries.","core_discovery":"The central claim is that a fully chunkwise, causal network can preserve the source utterance's content while adopting the timbre and style of a reference speaker, and that this design outperforms prior zero-shot voice conversion systems. Content is handled by a Stream Content Extractor: an Emformer trained to predict the same 20 ms HuBERT content labels an offline encoder would produce, so no future context is needed. Style is handled by an Adaptive Style Encoder that quantizes chunk-level reference representations with clustering-based vector quantization and then aligns them to the content and timbre via attention. Synthesis is handled by a Causal Shuffle Vocoder that rebuilds HiFi-GAN with causal convolutions and pixel-shuffle upsampling, avoiding the artifacts of zero-padded causal vocoders. On LibriTTS test-clean with VCTK reference speakers, the full Conan model reports the lowest WER and highest MOS-S among the compared systems, and the strictly causal fast model runs at 37 ms total latency.","pith_inferences":["The paper leaves implicit that the same chunkwise design could be ported to singing voice conversion or streaming text-to-speech, where the content labels and F0 path would carry melody instead of speech prosody.","Because the Adaptive Style Encoder uses a discrete codebook, a natural next experiment is to test whether individual codes correspond to interpretable emotions or prosody and whether manipulating them gives real-time style control.","Latency is bounded by chunk size plus per-module delay, so smaller chunks would push below 37 ms, presumably at some cost to content accuracy; the paper's full and fast settings bracket that trade-off.","The reported comparison with StreamVC relies on published numbers and demo pages, so a matched head-to-head test would be the cleanest way to confirm the superiority claim beyond the paper's own protocol."],"forward_implications":["Real-time services could add a natural-sounding voice conversion stage with only tens of milliseconds of latency while keeping the original words.","The same distillation recipe, training a streaming encoder to match offline HuBERT labels, could be reused for other streaming speech tasks that currently depend on non-causal self-supervised models.","The Causal Shuffle Vocoder's pixel-shuffle upsampling gives streaming synthesizers an alternative to zero-padded causal vocoders, removing a known source of spectral artifacts.","The configurable right-context chunks give deployers a direct latency-versus-quality trade-off: 80 ms chunks with two lookahead chunks for quality, 20 ms strictly causal chunks for speed."],"supporting_citations":[{"why":"VQMIVC is one of the offline baselines whose content accuracy, similarity, and quality numbers Conan is compared against in Table I.","marker":"[2]"},{"why":"StreamVC is the main online baseline; the paper adopts its dataset split and evaluation protocols for comparison.","marker":"[6]"},{"why":"HuBERT supplies the offline content representations that the Stream Content Extractor is trained to distill into 20 ms labels.","marker":"[25]"},{"why":"The clustered vector quantization codebook is used in the Adaptive Style Encoder to capture detailed style while avoiding codebook collapse.","marker":"[36]"},{"why":"The scaled dot-product attention mechanism is the basis of the Align Attention module that fuses style with content and timbre.","marker":"[40]"},{"why":"HiFi-GAN is the vocoder backbone the Causal Shuffle Vocoder makes fully causal, and its GAN, mel, and feature-map losses are reused.","marker":"[41]"},{"why":"LibriTTS provides the training data and the test-clean source speech, with VCTK speakers used as unseen references.","marker":"[44]"}],"fun_headline_variants":["Real-time voice swap keeps words, adopts new voice","Chunkwise voice conversion hits 37 ms with content intact","Zero-shot VC streams live, preserving what you say","Causal voice changer: 37 ms latency, true timbre transfer","Online voice conversion that keeps your words, changes your voice"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the numbers reported for the main competing streaming system, StreamVC, were measured under the same conditions as Conan's own evaluations; if the protocols and listening tests differ, the claimed superiority over StreamVC is not established.","fun_headline_variants_meta":{"raw":{"variants":["Real-time voice swap keeps words, adopts new voice","Chunkwise voice conversion hits 37 ms with content intact","Zero-shot VC streams live, preserving what you say","Causal voice changer: 37 ms latency, true timbre transfer","Online voice conversion that keeps your words, changes your voice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1506,"prompt_tokens":915,"completion_tokens":591,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":508}},"tokens_in":531,"tokens_out":591,"duration_ms":5953,"temperature":1.0,"reasoning_tokens":508,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:53:07.239640+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A matched head-to-head test would settle the claim: run Conan and StreamVC on identical source-reference pairs with the same ASR for WER and CER, the same speaker-similarity model, and the same listeners for MOS and CMOS. If Conan does not beat StreamVC on these metrics under identical protocols, the paper's central claim of outperforming baselines fails.","supporting_citations":[{"cited_title":"Vqmivc: Vector quantization and mutual information-based unsuper- vised speech representation disentanglement for one-shot voice conver- sion,","cited_arxiv_id":null,"evidence_quote":"VQMIVC is one of the offline baselines whose content accuracy, similarity, and quality numbers Conan is compared against in Table I."},{"cited_title":"Streamvc: Real-time low-latency voice conversion,","cited_arxiv_id":null,"evidence_quote":"StreamVC is the main online baseline; the paper adopts its dataset split and evaluation protocols for comparison."},{"cited_title":"Online clustered codebook,","cited_arxiv_id":null,"evidence_quote":"The clustered vector quantization codebook is used in the Adaptive Style Encoder to capture detailed style while avoiding codebook collapse."},{"cited_title":"Attention is all you need,","cited_arxiv_id":null,"evidence_quote":"The scaled dot-product attention mechanism is the basis of the Align Attention module that fuses style with content and timbre."},{"cited_title":"Libritts: A corpus derived from librispeech for text-to-speech,","cited_arxiv_id":null,"evidence_quote":"LibriTTS provides the training data and the test-clean source speech, with VCTK speakers used as unseen references."}],"review_version":1}