VoiceMark: Zero-Shot Voice Cloning-Resistant Watermarking Approach Leveraging Speaker-Specific Latents

Haiyun Li; Hanyang Peng; Jingran Xie; Xiaofeng Xie; Yaoxun Xu; Zhiyong Wu

REVIEW 4 major objections 6 minor 1 cited by

VoiceMark: Zero-Shot Voice Cloning-Resistant Watermarking Approach Leveraging Speaker-Specific Latents

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VoiceMark embeds watermarks into speaker-specific audio features so they survive zero-shot voice cloning, reporting over 95% detection accuracy where prior methods hover near random chance.

desk verdict A genuinely new watermarking design with strong reported results, but the 'speaker-specific latent carrier' mechanism needs direct verification before the zero-shot VC-resistance claim is fully trusted. read the letter →

arxiv 2505.21568 v2 pith:KLERDUMW submitted 2025-05-27 cs.SD cs.AIcs.CReess.AS

Haiyun Li , Zhiyong Wu , Xiaofeng Xie , Jingran Xie , Yaoxun Xu , Hanyang Peng This is my paper

classification cs.SDcs.AIcs.CReess.AS

keywords audiowatermarkingzero-shotvoicecloningspeaker-specificlatentsresidualvectorquantizationdetectionspeechsecurityactivitywatermarkrobustness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that an audio watermark can survive zero-shot voice cloning, in which a model makes new speech from a few seconds of an unseen speaker's voice without being trained on it. Its proposed method, VoiceMark, hides the watermark in the compressed speaker-specific features that such a model must carry from prompt to output, so the mark is copied along with the voice. Tested against three zero-shot cloning systems, VoiceMark reports watermarks recovered at over 95% bitwise accuracy, while existing watermarking methods decode at chance level, around 50%. If this holds, cloned audio can be traced back to the specific original recording it was cloned from, giving artists and platforms a practical attribution tool against unauthorized voice cloning.

What carries the argument

The central object is the speaker-specific latent sequence from an 8-layer RVQ codec: layer 1 carries content, while layers 2–8 carry timbre, pitch, and prosody. A cross-attention transformer embedder uses the 16-bit watermark, converted to a hex sequence, as keys and values, and writes it into those speaker layers; a transformer decoder with learnable CLS tokens reads the watermark back from the speaker latents of any input audio. Training forces this behavior with four losses, including a voice activity detection (VAD)-based loss that labels only speech-containing, watermarked frames so the model learns to embed where speaker identity actually lives, and a cosine similarity loss that keeps the speaker latents close to their original values to preserve voice quality. A set of cloning-simulating augmentations—frame masking, 50-ms shuffling, segment replacement, EnCodec encoding/decoding, and speed/filter perturbation—makes the decoder robust to the distortions zero-shot VC introduces.

What would settle it

Take VoiceMark's watermarked audio and run it through a zero-shot voice cloning model that conditions on a learned speaker embedding rather than on codec-disentangled speaker latents; if the model produces high-quality clones but VoiceMark's bitwise accuracy falls to roughly 50%, the speaker-latent carrier assumption is falsified. A simpler same-model test would be to strip or randomly perturb VQ layers 2–8 before synthesis and check whether cloning quality stays high while watermark detection collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that zero-shot VC resistance comes not from training the cloning model but from changing where the watermark lives. VoiceMark separates each audio clip into content latents and speaker-specific latents using an eight-layer residual vector quantization (RVQ) codec, embeds the watermark only into the speaker-specific layers, and reconstructs watermarked audio. Because zero-shot cloning must preserve timbre, pitch, and prosody to sound like the prompt speaker, those speaker-specific latents are transferred into the synthesized audio and carry the watermark with them. A transformer decoder then reads the watermark back from the latents of the cloned audio, trained only on augmented watermarked audio that mimics cloning distortions. On CosyVoice, F5-TTS, and MaskGCT, the paper reports bitwise accuracy above 95% and false attribution rates of 0.07–0.14, against roughly 0.50 accuracy for AudioSeal, WavMark, and Timbre.

Load-bearing premise

The method works only if zero-shot cloning models carry the same speaker-specific compressed features that VoiceMark modifies into the synthesized audio; a cloning model that derives identity from different signals, or that heavily transforms those latents, would make the watermark undetectable.

Editorial extensions

If this is right

Watermarking no longer requires training the cloning model: VoiceMark is trained entirely on augmented audio and then works on the output of inference-only zero-shot VC systems.
On CosyVoice, F5-TTS, and MaskGCT, decoded bitwise accuracy exceeds 95% with false attribution rates of 0.07–0.14, so cloned audio can be traced to its source recording with high confidence.
VoiceMark also holds up under traditional editing transforms, including EnCodec, resampling, filters, and MP3 compression, with bitwise accuracy between 0.965 and 0.995 on those tests.
Subjective audio quality remains close to the unwatermarked SpeechTokenizer reconstruction (SMOS 4.25 vs 4.63), so the traceability is not bought with obviously audible distortion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's three test models, VoiceMark's success is tied to how faithfully a cloning model preserves the RVQ codec's speaker layers; a model that derives identity from a separate speaker embedding, or that re-quantizes latents in a different codebook, could erase the mark, and the paper does not test that regime.
The 16-bit payload limits VoiceMark to identifying one of 65,536 watermarks; tracing large catalogs of recordings would require longer payloads, and the tradeoff between payload length, accuracy, and audio quality is not measured here.
One testable extension is to attack the watermark deliberately—for example, applying a second speaker's prompt, voice conversion that overwrites timbre, or band-limiting the speaker-latent layers—to find the point where attribution fails; the paper reports robustness but not this adversarial upper bound.
A positive consequence of embedding in identity latents is that the same latent carrier could in principle authenticate legitimate cloned audio or carry per-speaker metadata, not only trace misuse, though VoiceMark's binary watermark decoding would need to be extended for such uses.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this is a real new idea, not a repackaging. Embedding the watermark in the residual VQ layers that SpeechTokenizer uses for speaker information, rather than in the waveform or spectrogram, is the natural place to survive zero-shot VC, and the paper is the first to report it working across CosyVoice, F5-TTS, and MaskGCT. The reported margins are large — 0.96+ ACC vs. 0.50 for baselines — and the ablations (AS-Emb + VM-Dec, no augmentation, no VAD loss) support the design choices. Quality metrics and spectrogram examples are also there, and the code and demos are linked. Credit where due: the evaluation is extensive for a conference paper, and the decoder is trained only on augmented audio, not on the tested VC outputs, so there is no circular fitting to the test models.

The soft spots are real but not fatal. First, Table 1 gives single-run ACC/FAR with no variance or significance tests; with margins this large that is a presentational issue, not a correctness one, but readers will want error bars. Second, there is no explicit false-positive control on clean, unwatermarked audio — the VAD-based probability in the case study is suggestive, but a systematic FAR on a held-out clean set should be reported. Third, the omitted related work [9] (speaker identity watermarking) is at least conceptually adjacent; code availability doesn't excuse explaining the difference. The bigger concern, which the stress-test note puts its finger on, is that the paper never directly demonstrates that the VQ2-8 latents are the actual carrier through the three VC models. The decoder is trained on augmentations meant to mimic VC distortions, and the high accuracy is consistent both with the latents really surviving and with some other prompt-dependent acoustic invariant that the decoder learns to exploit. The three VC models likely share codec-based design, so they do not establish breadth by themselves. This is a mechanism/scope risk, not an internal inconsistency: the empirical result for the tested models can stand without the stronger 'speaker-specific latent carrier' narrative, but the first-to-market claim should be scoped accordingly.

Bottom line: this deserves a serious referee. The idea is promising, the evidence is strong enough to warrant further work, and the missing pieces — latent-survival analysis, variance, false-positive control, and a conceptual comparison to [9] — are addressable in revision. I'd send it to a venue that cares about audio security and ask for those additions.

Referee Report

4 major / 6 minor

Summary. The paper proposes VoiceMark, a watermarking method intended to survive zero-shot voice cloning (VC). The watermark is embedded into the speaker-specific latents (VQ2-8) of a pretrained SpeechTokenizer RVQ codec, and is recovered by a transformer decoder trained on VC-simulated augmentations. Evaluations on CosyVoice, F5-TTS, and MaskGCT report watermark detection accuracy above 95%, far exceeding the near-chance performance of AudioSeal, WavMark, and the Timbre baseline. The paper also introduces a VAD-based loss and evaluates traditional audio editing robustness and audio quality.

Significance. If the results are reliable, VoiceMark would be a meaningful step toward traceability of zero-shot voice cloning, a scenario where existing watermarking methods fail. The paper's strengths include a clear and plausible mechanism (using latent carriers), a broad evaluation across three recent zero-shot VC models, ablations isolating the main components, and public code/demos. However, the central causal claim — that the watermark is carried specifically by the VQ2-8 speaker-specific latents through the VC process — is not empirically verified, and the evaluation lacks false-positive controls and statistical error bars. These gaps currently limit the strength of the 'first zero-shot VC-resistant watermarking' claim, although the reported margins over baselines make the approach promising.

major comments (4)

[§2.4, §3.5, Table 2] The paper does not demonstrate that the watermark is causally carried by the SpeechTokenizer VQ2-8 latents through the zero-shot VC pipeline. The decoder is trained only on augmented audio (as described in §2.3) and is never exposed to actual VC outputs, so its transferability to VC-synthesized audio is an assumption. The ablation in Table 2 shows that replacing the AudioSeal embedder with the latent-based embedder improves results, but it does not show that the decoded watermark in synthesized audio resides specifically in the VQ2-8 latents, as opposed to some other prompt-dependent acoustic invariant that the decoder exploits after re-encoding by SpeechTokenizer. To support the paper's title and central claim, the authors should provide direct evidence: for example, compare detection from VQ2-8 latents versus other layers of the re-encoded synthesized audio, test a zero-shot VC model that does not share a similar neural codec design, or demonstrate transferability by embedding the same watermark into different latent layers and showing that only VQ2-8 supports recovery.
[§3.4, §3.5] There is no false-positive control on unwatermarked audio. The FAR metric is computed only for watermarked audio by comparing the decoded watermark to 100 candidates (one ground truth and 99 random). In a practical tracing application, the decoder must not return a valid watermark for clean, non-watermarked speech. The paper should define a detection threshold (e.g., minimum bit accuracy or maximum Hamming distance to the nearest candidate) and report the false-positive rate on unwatermarked test audio. Without this, the reported FAR values do not establish that VoiceMark enables reliable tracing without falsely attributing clean audio.
[Table 1, §3.5] All watermark detection results are single-run ACC/FAR values without error bars, confidence intervals, or significance tests. Given that baselines are near chance (ACC ≈ 0.5) and VoiceMark is near 0.96, the margin is large, but the absence of variance reporting prevents assessment of stability. The authors should report means and standard deviations over at least three independent training runs or over bootstrapped resamples of the test set, and ideally perform a paired significance test against the best baseline for the FAR comparison.
[§2.3, §2.4] The 'VC-simulated augmentations' in §2.3 are ad hoc and are not validated against the actual distribution of distortions introduced by zero-shot VC models. The paper does not justify why 50 ms shuffling, masking, replacement, and EnCodec encoding/decoding are representative of the transformations performed by CosyVoice, F5-TTS, and MaskGCT. Since the decoder is trained exclusively on these augmentations, the high detection accuracy on VC outputs might be model-specific or coincidental. The authors should measure the distribution shift between the augmented audio and actual VC outputs (e.g., comparing the statistics of VQ2-8 latents) and/or include a small amount of real VC outputs in training or validation to confirm that the augmentations capture the relevant distortions.

minor comments (6)

[§3.4] The definition of FAR should clarify whether the 99 random candidates are sampled independently for each test sample or shared across all samples, and how ties in Hamming distance are broken when the closest match is not unique.
[§3.5] The paper should state how many distinct watermark messages are used in the test set (e.g., one per sample, a shared set, or randomly generated), as this affects the interpretation of ACC and FAR.
[§2.5, Eq. (3)] The loss weights λ_vad=1, λ_cos=2, λ_mel=2, λ_adv=1, λ_dec=1 are given without sensitivity analysis; a brief ablation or discussion of their robustness would strengthen the paper.
[§3.7, Table 3] The SMOS scores are reported with confidence intervals, but no statistical test (e.g., paired t-test) is provided to compare VoiceMark against SpeechTokenizer and AudioSeal; consider adding significance testing.
[Figure 3] The red boxes in the spectrograms are not clearly explained; the caption should specify what artifact is highlighted in each subfigure and how the readers should interpret the visual differences.
[§1] The claim that 'Audio synthesized by zero-shot VC models exhibits significant differences ... These transformations disrupt or filter out the watermark' is presented without supporting evidence or citation; adding a brief reference or a small demonstration would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VoiceMark's decoder is trained only on augmented watermarked audio, never on the tested VC outputs, so the reported >95% accuracies are independent empirical results rather than a fitted artifact.

full rationale

The derivation chain is self-contained with respect to the benchmark data. The watermark embedder (Eq. 1) modifies SpeechTokenizer VQ2-8 latents, and the decoder (Eq. 2) reads from the same latent representation, but that shared codec is a design choice, not a reduction: the decoder's training objective (L_dec in Eq. 3) is optimized on VC-simulated augmentations (Section 2.3), not on synthesized audio from CosyVoice, F5-TTS, or MaskGCT, and the test accuracies in Table 1 are measured on unseen VCTK/LibriSpeech samples through actual VC inference. No parameter is fitted to the test VC outputs, and no load-bearing result is imported from a self-citation chain. The key assumption that zero-shot VC models transfer speaker-specific latents is an empirical hypothesis tested on three models; while it is a scope and mechanism risk if those models share codec-level inductive biases, it is not circular because the paper does not define 'speaker-specific latents' in terms of the observed watermark survival. The ablation in Table 2 isolates the latent-based embedder but does not by construction force the accuracy. No circular step can be exhibited from the paper's equations.

Assumptions & free parameters 10 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the assumption that zero-shot VC models preserve speaker-specific latents and that SpeechTokenizer's disentanglement is reliable. No new physical entities are introduced.

free parameters (10)

λ_vad (VAD loss weight) = 1
Hand-set hyperparameter in Section 3.1; controls emphasis on VAD-based loss.
λ_cos (cosine similarity loss weight) = 2
Hand-set hyperparameter in Section 3.1; larger weight to preserve speaker consistency.
λ_mel (mel spectrogram loss weight) = 2
Hand-set hyperparameter in Section 3.1; larger weight to preserve perceptual quality.
λ_adv (adversarial loss weight) = 1
Hand-set hyperparameter in Section 3.1.
λ_dec (decoding loss weight) = 1
Hand-set hyperparameter in Section 3.1.
Frame mask probability = 20%
Augmentation schedule in Section 2.3, chosen by hand.
Shuffle probability = 50%
Augmentation schedule in Section 2.3, chosen by hand.
Replace probability = 50%
Augmentation schedule in Section 2.3, chosen by hand.
Audio perturbation probability = 10%
Augmentation schedule in Section 2.3, chosen by hand.
Watermark bit length = 16
Design choice stated in Section 3.1.

assumptions (3)

domain assumption Zero-shot VC models consistently transfer speaker-specific latents from the prompt to synthesized audio.
Stated in the Introduction as the key observation; the paper does not measure latent transfer directly, only end-to-end watermark detection.
domain assumption SpeechTokenizer's first VQ layer encodes content and layers 2-8 encode speaker-specific information, as trained with HuBERT semantic teacher.
Relied on in Section 2.1; adopted from prior work [17] without verification.
domain assumption The augmentations in Section 2.3 (masking, shuffling, replacing, codec, perturbation) approximate the distortions of zero-shot VC synthesis.
Training uses these augmentations instead of real VC outputs; generalization depends on this approximation being adequate.

how reviews work

0 comments

Cite this review

Pith. "Pith review of VoiceMark: Zero-Shot Voice Cloning-Resistant Watermarking Approach Leveraging Speaker-Specific Latents." pith.science (2026). https://pith.science/paper/KLERDUMW

@misc{pith2026250521568,
  author       = {Pith},
  title        = {Pith review of: VoiceMark: Zero-Shot Voice Cloning-Resistant Watermarking Approach Leveraging Speaker-Specific Latents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KLERDUMW}},
  note         = {Machine review of arXiv:2505.21568}
}

read the original abstract

Voice cloning (VC)-resistant watermarking is an emerging technique for tracing and preventing unauthorized cloning. Existing methods effectively trace traditional VC models by training them on watermarked audio but fail in zero-shot VC scenarios, where models synthesize audio from an audio prompt without training. To address this, we propose VoiceMark, the first zero-shot VC-resistant watermarking method that leverages speaker-specific latents as the watermark carrier, allowing the watermark to transfer through the zero-shot VC process into the synthesized audio. Additionally, we introduce VC-simulated augmentations and VAD-based loss to enhance robustness against distortions. Experiments on multiple zero-shot VC models demonstrate that VoiceMark achieves over 95% accuracy in watermark detection after zero-shot VC synthesis, significantly outperforming existing methods, which only reach around 50%. See our code and demos at: https://huggingface.co/spaces/haiyunli/VoiceMark

Figures

Figures reproduced from arXiv: 2505.21568 by the authors.

**Figure 1.** Comparison of data flow for embedding watermarks in traditional voice cloning and zero-shot voice cloning. pression, with limited focus on resistance to VC models. Some methods [1, 9] propose VC-resistant watermarking techniques designed to resist traditional VC models. They validates that training traditional VC models [10, 11, 12] or commercial tools [13, 14] on watermarked audio enables robust watermark detectio… view at source ↗

**Figure 2.** The overall architecture of our proposed VoiceMark. audio during the zero-shot VC inference, thereby enabling robust zero-shot VC-resistant watermarking. To achieve this goal, we must address two primary challenges: 1) Speaker-specific latents span only speech-containing frames, while a large portion of the audio consists of silence and voiceless frames, which lack such latents. This makes conventional frame-by-fr… view at source ↗

**Figure 3.** Visualization of Mel Spectrograms. other methods exhibit visible watermarking artifacts in certain frequency bands (highlighted in red boxes), which attackers can exploit to detect or remove the watermark. In addition, we visualize watermark probability detected by VoiceMark as a top color band, with red for high and blue for low probability. In VoiceMark’s watermarked audio, the detected probability aligns precise… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Investigating Codec-Internal Latent Audio Watermarking for Neural Codec Robustness
cs.SD 2026-07 conditional novelty 6.0 of 10

Embedding watermarks inside a codec-like autoencoder's continuous latent space improves EnCodec-24k bit accuracy to ~95–97%, but the gain is in-distribution and does not transfer to EnCodec-16k.

Reference graph

Works this paper leans on

31 extracted references · 18 canonical work pages · cited by 1 Pith paper

[9]

F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,

Y . Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. Zhao, K. Yu, and X. Chen, “F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,”arXiv preprint arXiv:2410.06885, 2024

arXiv 2024
[1]

Introduction V oice cloning (VC)-resistant watermarking is an emerging tech- nique for tracing and preventing unauthorized cloning. Artists can embed such watermarks into their copyrighted recordings, ensuring that even if their voice is cloned into new audio, the watermark remains intact, thereby tracing and preventing unau- thorized cloning. Recent rese...

arXiv 2025
[2]

The RVQ model disentangles speaker-specific la- tents and reconstructs watermarked audio

Methodology V oiceMark consists of three main components: an encoder- decoder RVQ model, a cross-attention-based watermark em- bedder, and a transformer-based watermark decoder, as shown in Figure 2. The RVQ model disentangles speaker-specific la- tents and reconstructs watermarked audio. The embedder em- beds the watermark into speaker-specific latents. ...

work page
[3]

Implementation Details For the RVQ model, we use the pretrained SpeechTokenizer *

Experiments 3.1. Implementation Details For the RVQ model, we use the pretrained SpeechTokenizer *. We employs a 4-layer, 1-head, 256-dimensional Transformer Decoder, whileW d adopts an 8-layer, 1-head, 512-dimensional Transformer Encoder [19]. The watermark bit length is set to

work page
[4]

Additionally, we incorporate VC-simulated augmenta- tions and V AD-based loss to further enhance the robustness of V oiceMark

Conclusion In this work, we propose V oiceMark, the first zero-shot VC- resistant watermarking method that embeds watermarks into speaker-specific latents to achieve resistance to zero-shot VC models. Additionally, we incorporate VC-simulated augmenta- tions and V AD-based loss to further enhance the robustness of V oiceMark. Experimental results show tha...

work page
[5]

Acknowledgements This work is supported by National Natural Science Foundation of China (62076144)

work page
[6]

Gruhl, A

D. Gruhl, A. Lu, and W. Bender,Echo hiding, Jan 1996, p. 295–315. [Online]. Available: http://dx.doi.org/10.1007/ 3-540-61996-8 48

work page 1996
[7]

De- tecting voice cloning attacks via timbre watermarking,

C. Liu, J. Zhang, T. Zhang, X. Yang, W. Zhang, and N. Yu, “De- tecting voice cloning attacks via timbre watermarking,” inNet- work and Distributed System Security Symposium, 2024

work page 2024

Show all 31 references

[8]

Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y . Yang, H. Hu, S. Zheng, Y . Gu, Z. Maet al., “Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,”arXiv preprint arXiv:2407.05407, 2024

2024 arXiv
[10]

Maskgct: Zero-shot text- to-speech with masked generative codec transformer,

Y . Wang, H. Zhan, L. Liu, R. Zeng, H. Guo, J. Zheng, Q. Zhang, X. Zhang, S. Zhang, and Z. Wu, “Maskgct: Zero-shot text- to-speech with masked generative codec transformer,” inICLR. OpenReview.net, 2025

2025
[11]

Techniques for data hiding,

W. Bender, D. Gruhl, N. Morimoto, and A. Lu, “Techniques for data hiding,”IBM Systems Journal, p. 313–336, Jan 1996. [Online]. Available: http://dx.doi.org/10.1147/sj.353.0313

1996 doi
[12]

Natural tts synthesis by conditioning wavenet on mel spectrogram pre- dictions,

J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y . Zhang, Y . Wang, R. Skerrv-Ryanet al., “Natural tts synthesis by conditioning wavenet on mel spectrogram pre- dictions,” in2018 IEEE international conference on acoustics, speech and signal processing...

2018
[13]

Wav- mark: Watermarking for audio generation,

G. Chen, Y . Wu, S. Liu, T. Liu, X. Du, and F. Wei, “Wav- mark: Watermarking for audio generation,”arXiv preprint arXiv:2308.12770, 2023

2023 arXiv
[14]

Proactive detection of voice cloning with localized watermarking,

R. San Roman, P. Fernandez, H. Elsahar, A. D ´efossez, T. Furon, and T. Tran, “Proactive detection of voice cloning with localized watermarking,” inInternational Conference on Machine Learn- ing, vol. 235, 2024

2024
[15]

Proactive audio authentication using speaker identity watermarking,

Q. Li and X. Lin, “Proactive audio authentication using speaker identity watermarking,” in2024 21st Annual International Con- ference on Privacy, Security and Trust (PST). IEEE, 2024, pp. 1–10

2024
[16]

The model is trained for 30 epochs using Adam [21] optimizer with a learning rate of5e −5

Hyperparameters are set as:λ vad = 1,λcos = 2,λmel = 2, λadv = 1, andλ dec = 1, with larger weights forλ cos andλ mel to preserve audio quality. The model is trained for 30 epochs using Adam [21] optimizer with a learning rate of5e −5. 3.2. Baselines We compare V oiceMark with...

2024
[17]

Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,

J. Kim, J. Kong, and J. Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” inInter- national Conference on Machine Learning. PMLR, 2021, pp. 5530–5540

2021
[18]

Fastspeech 2: Fast and high-quality end-to-end text to speech,

Y . Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” inInternational Conference on Learning Representations
[19]

Paddlespeech: An easy-to-use all-in-one speech toolkit,

H. Zhang, T. Yuan, J. Chen, X. Li, R. Zheng, Y . Huang, X. Chen, E. Gong, Z. Chen, X. Huet al., “Paddlespeech: An easy-to-use all-in-one speech toolkit,” inProceedings of the 2022 Confer- ence of the North American Chapter of the Association for Com- putational Linguistics: Hu...

2022
[20]

V oice cloning app,

“V oice cloning app,” https://github.com/BenAAndrew/ V oice-Cloning-App, 2023

2023
[21]

High fidelity neural audio compression,

A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”arXiv preprint arXiv:2210.13438, 2022

2022 arXiv
[22]

Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models,

Z. Ju, Y . Wang, K. Shen, X. Tan, D. Xin, D. Yang, E. Liu, Y . Leng, K. Song, S. Tanget al., “Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models,” inForty-first International Conference on Machine Learning
[23]

Speechtok- enizer: Unified speech tokenizer for speech large language mod- els,

X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtok- enizer: Unified speech tokenizer for speech large language mod- els,”arXiv preprint arXiv:2308.16692, 2023

2023 arXiv
[24]

Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

2021
[25]

Attention is all you need,

A. Vaswani, “Attention is all you need,”Advances in Neural In- formation Processing Systems, 2017

2017
[26]

L. R. Rabiner,Digital processing of speech signals. Pearson Education India, 1978

1978
[27]

Adam: A method for stochastic optimization,

D. P. Kingma, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014

2014 arXiv
[28]

English multi-speaker corpus for cstr voice cloning toolkit,

J. Yamagishi, “English multi-speaker corpus for cstr voice cloning toolkit,” 2012. [Online]. Available: https://datashare.ed. ac.uk/handle/10283/3443

2012
[29]

Lib- rispeech: an asr corpus based on public domain audio books,

V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

2015
[30]

Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,

A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,” in 2001 IEEE international conference on acoustics, speech, and signal processing. Proceed...

2001
[31]

A short- time objective intelligibility measure for time-frequency weighted noisy speech,

C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “A short- time objective intelligibility measure for time-frequency weighted noisy speech,” in2010 IEEE international conference on acous- tics, speech and signal processing. IEEE, 2010, pp. 4214–4217

2010

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [9]

F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,

Y . Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. Zhao, K. Yu, and X. Chen, “F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,”arXiv preprint arXiv:2410.06885, 2024

arXiv 2024

[2] [1]

Introduction V oice cloning (VC)-resistant watermarking is an emerging tech- nique for tracing and preventing unauthorized cloning. Artists can embed such watermarks into their copyrighted recordings, ensuring that even if their voice is cloned into new audio, the watermark remains intact, thereby tracing and preventing unau- thorized cloning. Recent rese...

arXiv 2025

[3] [2]

The RVQ model disentangles speaker-specific la- tents and reconstructs watermarked audio

Methodology V oiceMark consists of three main components: an encoder- decoder RVQ model, a cross-attention-based watermark em- bedder, and a transformer-based watermark decoder, as shown in Figure 2. The RVQ model disentangles speaker-specific la- tents and reconstructs watermarked audio. The embedder em- beds the watermark into speaker-specific latents. ...

work page

[4] [3]

Implementation Details For the RVQ model, we use the pretrained SpeechTokenizer *

Experiments 3.1. Implementation Details For the RVQ model, we use the pretrained SpeechTokenizer *. We employs a 4-layer, 1-head, 256-dimensional Transformer Decoder, whileW d adopts an 8-layer, 1-head, 512-dimensional Transformer Encoder [19]. The watermark bit length is set to

work page

[5] [4]

Additionally, we incorporate VC-simulated augmenta- tions and V AD-based loss to further enhance the robustness of V oiceMark

Conclusion In this work, we propose V oiceMark, the first zero-shot VC- resistant watermarking method that embeds watermarks into speaker-specific latents to achieve resistance to zero-shot VC models. Additionally, we incorporate VC-simulated augmenta- tions and V AD-based loss to further enhance the robustness of V oiceMark. Experimental results show tha...

work page

[6] [5]

Acknowledgements This work is supported by National Natural Science Foundation of China (62076144)

work page

[7] [6]

Gruhl, A

D. Gruhl, A. Lu, and W. Bender,Echo hiding, Jan 1996, p. 295–315. [Online]. Available: http://dx.doi.org/10.1007/ 3-540-61996-8 48

work page 1996

[8] [7]

De- tecting voice cloning attacks via timbre watermarking,

C. Liu, J. Zhang, T. Zhang, X. Yang, W. Zhang, and N. Yu, “De- tecting voice cloning attacks via timbre watermarking,” inNet- work and Distributed System Security Symposium, 2024

work page 2024