{"id":"4c0e22ff-37ff-4e83-82d3-b11d7dd33a97","arxiv_id":"2412.08312","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"A HuBERT and HiFi-GAN pipeline with f0 features is proposed for unified voice and accent conversion in speech and singing, but the claimed gains are not backed by reproducible evidence.","lead":"This preprint proposes a voice conversion model that combines a HuBERT encoder, a HiFi-GAN decoder, and pitch features to convert speech and singing across voices and accents. It reports high accuracy and performance numbers, but provides no code, data, audio samples, or evaluation details to support them.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Target HuBERT branch is internally contradictory: under the paper's own claim that HuBERT discards speaker identity, target HuBERT embeddings cannot supply target voice; if they do retain identity, source content is not identity-free.","rationale":"The reader's weakest assumption was that HuBERT embeddings discard speaker identity and that this property transfers to singing and accented hybrid data. My concern is more specific and, I think, more load-bearing: the paper's own architecture contradicts that assumption. The source branch is claimed to be identity-free, but the target branch is simultaneously claimed to provide target voice characteristics via the same kind of HuBERT embeddings. If the assumption is true, the target branch adds content, not voice; if it is false, the source branch leaks source identity. Either way, the described pipeline cannot support the headline accuracies. This is an internal inconsistency, not merely an unverified external claim, so it strengthens the rejection. I do not fully agree with the reader's framing because the reader treats the content-discarding property as an untested hypothesis; I see the paper as committing to both the truth and falsity of that property in the same architecture. The proposed ablation would settle whether the target branch is inert or content-leaking, and therefore whether the central mechanism is sound. The verdict remains unchanged because the manuscript as written still lacks the definitions, baselines, and evaluation protocols needed to support its comparative claims; my concern adds a technical reason why even the architecture itself may be unimplementable as described.","tokens_in":5776,"tokens_out":4531,"duration_ms":65712,"concrete_test":"Run an ablation on the released (or, if none, reimplemented) model with three configurations on a non-parallel singing set: (i) full target HuBERT branch, (ii) target branch replaced by a constant vector or removed, and (iii) target HuBERT embeddings taken from a different target utterance with different lyrics while keeping target f0 fixed. Measure ASR word error of the output against both source and target lyrics, and a speaker-verification similarity to the target singer. If configuration (ii) is indistinguishable from (i), the target branch is inert. If configuration (iii) degrades content preservation (output tracks target lyrics), the target branch leaks content. Either result contradicts the role assigned to HuBERT in §3.2, and would require the reported conversion scores to be re-evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim depends on the target branch of the encoder-decoder. Section 3.2 states that HuBERT embeddings 'represent the content of the source audio, capturing the phonetic information while discarding speaker identity,' yet the same section uses target HuBERT embeddings to 'provide additional voice characteristics' for the target speaker. These roles are mutually incompatible. If the embeddings discard speaker identity, target HuBERT embeddings cannot supply target timbre; they inject target content, which for non-parallel singing conversion (where source and target lyrics differ) would corrupt the content of the output. If instead the embeddings retain speaker information, the source branch does not discard source identity, so the converted output should retain source voice characteristics. The paper also promises a 'singer embedding module' in the introduction but never defines one, so the only target-identity information in the described pipeline is target f0 plus the target HuBERT branch. Thus, under the paper's own assumption, the model has no mechanism to transfer voice identity; under the opposite assumption, the source-content claim fails. Either way, the architecture as described cannot implement the reported 90.6%/95.6% results. No code, checkpoints, or hyperparameters are provided to disambiguate, so this is not merely a reproducibility gap but a correctness risk in the core mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a unified voice and accent conversion system for speech and singing. The architecture combines a HuBERT-based encoder with a HiFi-GAN vocoder, using source and target self-supervised embeddings and f0 features; three losses (reconstruction, adversarial, pitch consistency) are defined. The authors report two evaluation tables: Table 1 gives voice identification (90.6%) and accent classification (95.6%) accuracy for the proposed model against three baselines, and Table 2 gives per-scenario conversion scores. The paper concludes that the model is a significant advancement in voice conversion. The manuscript provides no code, no hyperparameters, no metric definitions, no test-set descriptions, and no baselines definitions; Section 3.2 contains an apparent internal contradiction about whether HuBERT embeddings carry speaker identity.","tokens_in":5934,"tokens_out":5315,"duration_ms":53109,"significance":"If the reported numbers were properly supported, a single model handling speech-to-speech, singing voice conversion, and accent conversion on hybrid samples would be a useful contribution to the voice conversion literature. The paper also addresses an underexplored task (accent conversion on mixed speech-singing samples). However, the present manuscript does not establish this result: the evaluation is not verifiable (Tables 1-2), the baselines are undefined, and the described architecture's target HuBERT branch is internally inconsistent with the stated properties of HuBERT embeddings. There is no code, checkpoint, or reproducible training recipe, so the findings cannot be checked. These issues are load-bearing for the central claim, not presentation-level.","major_comments":[{"comment":"The manuscript never defines the metrics reported in Tables 1 and 2, nor the evaluation protocol. For Table 1, it does not specify what is being classified (speaker, utterance, or accent), how many speakers or utterances are in the test set, whether the test speakers are seen during training, or how errors are aggregated. For Table 2, the column labels C2C, O2C, C2O, and O2O are introduced only by their acronyms, and the score is called 'performance score, from an evaluation metric' without naming the metric or the conversion test setup. No error bars, number of runs, or statistical significance tests are reported. These numbers are therefore unverifiable and cannot support the claim that the proposed model outperforms the baselines.","section":"§4.1-§4.2, Tables 1 and 2"},{"comment":"The baselines HuBERT, HuBERT-Inter, and ContentVEC are not defined or cited anywhere in the manuscript. 'HuBERT' is ambiguous (pretrained HuBERT, fine-tuned HuBERT, or a HuBERT-based conversion system), and no reference is given for 'HuBERT-Inter' or 'ContentVEC'. Without a precise definition of these systems and their training and test conditions, the comparative claims in both tables cannot be assessed.","section":"§4.1-§4.2, baselines"},{"comment":"The architecture description contradicts itself. It states that source HuBERT embeddings 'represent the content of the source audio, capturing the phonetic information while discarding speaker identity'; immediately afterward it states that a pretrained HuBERT encoder extracts target self-supervised embeddings 'providing additional voice characteristics.' If the embeddings discard speaker identity, they cannot supply the target speaker's voice; if they retain speaker identity, the source branch does not discard source identity as claimed. The introduction promises a 'singer embedding module,' but no such module is ever defined, so the only target-identity information in the described pipeline is target f0 plus the target HuBERT branch. This is a core-mechanism inconsistency, not a reproducibility gap, and it undermines the architectural explanation of the reported 90.6% and 95.6% results.","section":"§3.2, target HuBERT branch"},{"comment":"No training or inference details are reported. The losses in Eqs. (1)-(3) are listed without weighting coefficients; the inference-time f0 scaling and shifting mentioned in the introduction is not specified; and the optimizer, learning rate, batch size, number of epochs, and train/validation/test split for either training stage are absent. The dataset sizes are only qualitatively described ('approximately three hours of singing audio', '30 minutes to one hour of speaking audio per singer'). No code or checkpoints are provided. As a result, the experiments cannot be reproduced or independently checked.","section":"§3.2, Eqs (1)-(3)"},{"comment":"The paper's central application claim concerns accent conversion on 'hybrid voice samples that encompass both speech and singing,' but no evaluation of hybrid samples appears anywhere in the results. There are no listening tests, no subjective MOS scores, no objective metrics for content preservation or speaker similarity of converted audio, and no comparison of converted waveforms. The only quantitative evidence is the classification table and the unspecified conversion-score table, so the conclusion of 'a significant advancement' is not supported by the presented data.","section":"§4-§5"}],"minor_comments":[{"comment":"The first sentence of the introduction contains 'Even said' instead of 'Even so', and there are numerous typographical issues throughout, such as 'V oice' and 'Hifi-gan' (References [4]-[6]).","section":"Introduction"},{"comment":"The text 'his concatenated feature vector' should read 'This concatenated feature vector'.","section":"§3.2"},{"comment":"The sentence 'The table I presents' is ungrammatical; table and figure captions should use consistent capitalization and should describe axes, units, and data provenance.","section":"§4.1"},{"comment":"The paper does not state how the mel-spectrogram is tokenized or normalized before entering the HuBERT encoder, nor the dimensionality of the concatenated feature vector; this should be clarified.","section":"§3.2"},{"comment":"Several references have inconsistent formatting; for example, [4] mixes an arXiv identifier with an access URL, and capitalization differs across entries.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript is not at the standard of a peer-reviewed paper: the evaluation is unverifiable, the baselines are undefined, and the architectural description contains an internal contradiction about the role of target HuBERT embeddings. In my assessment, these are load-bearing flaws that cannot be repaired within the scope of a revision, so I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read for you on arXiv:2412.08312. The paper combines HuBERT-based content extraction, f0 features, and a HiFi-GAN vocoder for voice and accent conversion in speech and singing. The combination is not new—similar ingredients appear in the SVC Challenge 2023 work the author cites—and the only slight novelty is the accent-conversion-on-hybrid-samples framing. What the paper does well is describe the data preprocessing and loss functions clearly enough that someone could re-implement the pipeline in principle. The related work section is also reasonable for a preprint.\n\nThe soft spots are load-bearing, not cosmetic. The stress-test note is correct: Section 3.2 first says HuBERT embeddings \"capturing the phonetic information while discarding speaker identity,\" then says target HuBERT embeddings provide \"additional voice characteristics.\" Those two claims cannot both be true. If HuBERT discards identity, the target branch injects target content, not target timbre; if it retains identity, the source branch fails to strip the source voice. The introduction also promises a \"singer embedding module\" that never appears in the methodology. So the architecture as described has no clear mechanism to transfer vocal identity at all.\n\nThe evaluation is equally unsupported. Tables 1 and 2 report numbers without defining the metrics, describing the test set, citing the baselines (ContentVEC and HuBERT-Inter are never defined), or providing error bars or statistical tests. Table 2 doesn't even name the evaluation metric. No code, data, or audio samples are shared. The central comparative claim therefore rests on unverifiable numbers.\n\nI agree with the reader's verdict. This is not a circularity problem; it is a missing-evidence problem compounded by an internal contradiction. The author may have working software, but the manuscript does not demonstrate it. A serious editor should desk-reject this version and invite a resubmission with a corrected architecture description, a defined and reproducible evaluation protocol, and released code and audio samples. As it stands, it is not refereeable.\n\nWho is this for? Possibly someone working on practical voice-conversion systems who wants a quick survey of one naive baseline design. But as a research claim, it does not hold up.","headline":"A clearly written but unevaluable preprint whose core architecture is internally contradictory; the reported numbers cannot be trusted as written.","tokens_in":6568,"tokens_out":1510,"would_cite":false,"duration_ms":18571,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single HuBERT-HiFi-GAN encoder-decoder model, augmented with f0 features and target singer embeddings, can perform both voice conversion and accent conversion for speech and singing, reaching 90.6% voice…","keywords":["voice conversion","accent conversion","singing voice conversion","HuBERT","HiFi-GAN","self-supervised learning","fundamental frequency","speech synthesis"],"falsifier":"Run the trained model on held-out accented singing clips and measure how often a speaker-verification system attributes the converted audio to the intended target speaker, together with word or phoneme intelligibility of the converted content. If converted audio is identified as the target at near-chance levels while content remains intelligible, or if intelligibility collapses when pitch consistency loss is weighted up, the central claim would fail. A simpler check: remove the target self-supervised embeddings from the concatenated input and see whether output speaker identity changes; if it does not, the model is not actually using the component the paper credits.","tokens_in":5451,"feed_emoji":"🎙️","tokens_out":5356,"duration_ms":48238,"temperature":0.7,"pith_summary":"This paper claims that one encoder-decoder architecture can handle both voice conversion and accent conversion for ordinary speech and for singing. The proposed system uses a HuBERT encoder to pull out self-supervised embeddings that are supposed to carry linguistic content while dropping speaker identity, then a HiFi-GAN vocoder to resynthesize audio in a target voice, with f0 features and target embeddings added to control pitch and timbre. On the paper's reported tests, the model reaches 90.6% voice-identification accuracy and 95.6% accent-classification accuracy, exceeding HuBERT, HuBERT-Inter, and ContentVEC baselines in every conversion scenario in Tables 1 and 2. If this result holds, a single pipeline could serve voice dubbing, accent modification, TTS, and singing-voice conversion without retraining per task.","feed_headline":"HuBERT-HiFi-GAN converts voices and accents, spoken or sung","feed_subtitle":"Reported 90.6% voice ID and 95.6% accent accuracy, beating HuBERT, HuBERT-Inter, and ContentVEC baselines.","key_machinery":"The load-bearing component is the concatenated feature vector built from (1) source HuBERT self-supervised embeddings, (2) source f0 features, (3) target HuBERT embeddings, and (4) target f0 features; this vector is decoded by a HiFi-GAN vocoder with multi-scale discriminators. The HuBERT encoder is a 12-layer transformer with hidden size 768 and 12 attention heads; the HiFi-GAN upsampler converts the feature vector into a waveform. Training combines a reconstruction loss on mel-spectrograms, an adversarial loss from the GAN discriminators, and a pitch-consistency L1 loss on f0, which together are meant to keep spectral fidelity, perceptual naturalness, and pitch accuracy aligned with the target.","core_discovery":"On its own terms, the paper's central discovery is that concatenating source and target HuBERT embeddings with source and target f0 features, and decoding that vector with HiFi-GAN, yields a unified voice-conversion system that outperforms HuBERT, HuBERT-Inter, and ContentVEC on both classification and conversion tests. The reported numbers are 90.6% versus 73.7%, 73.4%, and 37.7% for voice identification, and 95.6% versus 81.6%, 72.7%, and 62.1% for accent classification; in the four conversion scenarios C2C, O2C, C2O, O2O the proposed model scores 0.9512, 0.9428, 0.9392, and 0.9432, each above the corresponding baseline. The paper presents this as evidence that a single framework can preserve content and prosody while changing voice identity and accent across both speech and singing.","pith_inferences":["(Editorial inference) The conversion scores in Table 2 are presented as an unnamed evaluation metric, so a direct comparison with listening-test or standard MOS results would be needed before treating the gains as perceptual improvements.","(Editorial inference) Because the paper relies on HuBERT embeddings 'discarding speaker identity' without a verification experiment, the architecture's success could equally be explained by the f0 and target-embedding branches doing most of the work; ablating each branch would settle this.","(Editorial inference) If the claims transfer to unseen accents and singers, the same concatenated-embedding recipe could be applied to emotion conversion or non-verbal vocalization transfer, which the paper lists as limitations but does not test."],"forward_implications":["Voice dubbing and content creation could use a single pipeline for speech and singing rather than separate systems, since the model converts both with the same encoder-decoder.","Accent conversion on hybrid samples that mix speech and singing becomes possible, letting a speaker's accent be changed while content and prosody are preserved.","TTS and IVR systems could adopt this architecture to produce voices in a chosen accent or target speaker without retraining per voice.","The reported classification gains suggest the same embeddings carry enough voice and accent information for downstream speaker and accent verification tasks, not just conversion."],"supporting_citations":[{"why":"Supplies the HiFi-GAN vocoder that decodes the concatenated feature vector into high-fidelity waveforms.","marker":"[6]"},{"why":"Establishes the HuBERT-based feature-extraction approach for singing voice conversion that the paper extends with accent control.","marker":"[8]"},{"why":"Provides the Speech Accent Archive parallel dataset used to fine-tune the model for accent conversion.","marker":"[11]"},{"why":"Defines the prior VAE-GAN voice conversion system whose limitations (non-verbal sounds, emotions, accents) motivate the proposed model.","marker":"[1]"},{"why":"Introduces pitch adversarial training for singing voice conversion, informing the f0-based pitch handling in this model.","marker":"[7]"},{"why":"Demonstrates that parallel voice conversion can work with limited training data, supporting the paper's use of a small parallel accent dataset.","marker":"[10]"}],"fun_headline_variants":["Unified voice and accent conversion for speech and singing","Single model swaps voice and accent while preserving prosody","90.6% voice ID and 95.6% accent accuracy in one converter","Self-supervised encoder-decoder converts voice and accent in song and speech","HuBERT and HiFi-GAN pair with f0 to convert voice and accent in singing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system depends on HuBERT embeddings trained on speech actually capturing phonetic content while discarding speaker identity, and on that property carrying over to singing and to accented hybrid speech-singing samples; the paper asserts this in Section 3.2 but reports no experiment that verifies it.","fun_headline_variants_meta":{"raw":{"variants":["Unified voice and accent conversion for speech and singing","Single model swaps voice and accent while preserving prosody","90.6% voice ID and 95.6% accent accuracy in one converter","Self-supervised encoder-decoder converts voice and accent in song and speech","HuBERT and HiFi-GAN pair with f0 to convert voice and accent in singing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001018,"raw_usage":{"total_tokens":4298,"prompt_tokens":947,"completion_tokens":3351,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":3256}},"tokens_in":563,"tokens_out":3351,"duration_ms":24312,"temperature":1.0,"reasoning_tokens":3256,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:57:24.184100+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained model on held-out accented singing clips and measure how often a speaker-verification system attributes the converted audio to the intended target speaker, together with word or phoneme intelligibility of the converted content. If converted audio is identified as the target at near-chance levels while content remains intelligible, or if intelligibility collapses when pitch consistency loss is weighted up, the central claim would fail. A simpler check: remove the target self-supervised embeddings from the concatenated input and see whether output speaker identity changes; if it does not, the model is not actually using the component the paper credits.","supporting_citations":[{"cited_title":"Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,","cited_arxiv_id":null,"evidence_quote":"Supplies the HiFi-GAN vocoder that decodes the concatenated feature vector into high-fidelity waveforms."},{"cited_title":"Speech Accent Archive,","cited_arxiv_id":null,"evidence_quote":"Provides the Speech Accent Archive parallel dataset used to fine-tune the model for accent conversion."},{"cited_title":"V oice Conversion Using Speech-to-Speech Neuro-Style Transfer,","cited_arxiv_id":null,"evidence_quote":"Defines the prior VAE-GAN voice conversion system whose limitations (non-verbal sounds, emotions, accents) motivate the proposed model."},{"cited_title":"Parallel voice conversion with limited training data using stochastic variational deep kernel learning,","cited_arxiv_id":null,"evidence_quote":"Demonstrates that parallel voice conversion can work with limited training data, supporting the paper's use of a small parallel accent dataset."}],"review_version":1}