REVIEW 4 major objections 4 minor 32 references
Co-training the watermarking encoder, decoder, and source separator in one differentiable pipeline makes separation-first multi-stream audio watermarking feasible.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 17:58 UTC pith:L26HT66P
load-bearing objection Good problem and a useful negative result, but the paper's own Table 1 contradicts its title, and the described mechanism doesn't match the loss function. the 4 major comments →
Making Separation-First Multi-Stream Audio Watermarking Feasible via Joint Training
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is a mismatch: robustness to generic distortions does not imply robustness to source-separation artifacts, and a watermark decoder trained only against noise, filtering, and time-pitch distortions fails on separated stems. The fix is to make the separator part of the detector and optimize it jointly with the watermark encoder and decoder through a fully differentiable pipeline. The gradient from the watermark decoding loss flows backward through the separator, reshaping its outputs to retain watermark-relevant information, while the encoder learns to place watermarks where the separator's distortions do the least damage. With the co-trained pipeline, a 32-bit payload em
What carries the argument
The load-bearing mechanism is the joint training loop: 2-second watermarked segments are cut from full-length stems, reinserted into their original context, summed into a mixture, passed through a time-domain separator, sliced back out, and decoded. The total loss couples bit-recovery error with spectrogram and perceptual losses, while a stop-gradient separation term keeps the separator faithful to the watermarked stems so it does not collapse the watermark space. This single gradient path from the decoding loss back through the separator to the encoder is what forces the two systems to co-adapt; without it, the watermarking system has no way to learn the separator's specific distortion prof
Load-bearing premise
The method presupposes that the separator used at deployment time is the one co-trained with the watermark and is part of the decoding pipeline; with an arbitrary third-party separator, the same watermarking backbone fails.
What would settle it
Run the jointly trained pipeline but swap in a different, off-the-shelf separator (different architecture or pretrained weights) at test time and decode the same watermarked mixture. If the bit error rate jumps back to the 15-30% range on speech-plus-music, the feasibility result is contingent on the co-trained separator. Similarly, decoding after the co-trained separator on an unseen, more entangled stem pair, or after a dataset shift in mixing levels, would test whether the co-adaptation is general or memorized.
If this is right
- If the claim holds, multi-stem provenance verification becomes possible: each stem's watermark can be read independently after the stems have been mixed, as long as the separator in the verification pipeline is the co-trained one.
- Watermarking systems for generative audio should be evaluated against the actual separation or editing operations they will encounter, not only against generic attack suites.
- A separator can be adapted to preserve watermark cues without sacrificing separation quality; the reported separation metrics barely move after joint training.
- Two-stage fine-tuning, where the separator is adapted after the watermark is frozen, is a viable alternative in difficult entangled cases and in this paper reaches the best vocal-plus-accompaniment result.
Where Pith is reading between the lines
- Because the gain depends on the separator being part of the detector, deploying the same watermark with a different, off-the-shelf separator would likely push bit error rates back to the 15-20% failure range; cross-separator generalization is the natural next test.
- The discrepancy between the joint and fine-tuned variants on vocal-plus-accompaniment hints that end-to-end training can overfit the separator's own artifacts when stems are heavily spectrally entangled; a decoupled two-stage schedule may be more robust in that regime.
- A plausible extension is to train with a pool of separators or with separator dropout, forcing the watermark to survive arbitrary separation models; that would move the method from controlled verification toward real-world deployment.
- The same joint-training principle may apply to other non-differentiable content transformations, such as audio editing or restoration, where the 'separator' is any fixed processing step whose artifacts are structured.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies a 'separation-first, multi-stream' audio watermarking pipeline: each stem is independently watermarked with a distinct key, the stems are mixed, a source separator (Demucs) is applied, and the watermark is decoded from each separated stem. The authors first show that a standard neural watermarking backbone (AURA*) combined with an off-the-shelf separator yields high bit error rates, and then propose to co-train the watermarking encoder/decoder and the separator either jointly end-to-end ('AURA*-Demucs joint') or in a two-stage fashion ('AURA*-Demucs finetuned'). Experiments on speech+music and vocal+accompaniment mixtures report large BER reductions relative to WavMark, Audioseal, and AURA*, while ViSQOL scores remain above 4.5. The central claim is that end-to-end joint training of the watermarking system and separator is what makes the separation-first multi-stream framework feasible.
Significance. If the central claim were established, the paper would make a useful contribution: it formalizes an increasingly important setting (multi-stem audio provenance) and provides evidence that robustness to generic distortions does not transfer to separation artifacts. The two-stage fine-tuned variant's 1.25% BER on the realistic MUSDB18 vocal+accompaniment benchmark, with ViSQOL 4.51, is a substantive result that could support a narrower but valuable claim: adapting a separator to a fixed watermarking system can make separation-first decoding feasible. The paper also uses standard architecture choices (unmodified Demucs), separate train/test data, and standard perceptual metrics. However, the headline contribution—joint end-to-end training—is contradicted by the paper's own Table 1 on the more realistic benchmark, and the described training mechanism is internally inconsistent. As written, the paper overclaims its main result; the underlying evidence is promising but the central claim needs substantial revision, clarification, and additional analysis.
major comments (4)
- [Table 1 (Origin, Vocal+Accompaniment); §2.10] The paper's central claim is that 'end-to-end joint training' is what makes separation-first multi-stream watermarking feasible. Table 1 contradicts this on the vocal+accompaniment benchmark: AURA*-Demucs joint achieves 15.21% BER at Origin, only slightly better than AURA* alone (19.23%) and far worse than AURA*-Demucs finetuned (1.25%). The two-stage variant freezes the watermarking encoder/decoder and tunes only Demucs, so it does not exhibit the claimed end-to-end co-adaptation. Section 2.10 states that 'jointly optimized models generalize substantially better' and that 'end-to-end coupling enables...', but on MUSDB18 the jointly trained model is not feasible for reliable decoding. The contribution should be repositioned as 'separation-aware training', with the fine-tuned variant as the successful method, or the joint method must be analyzed and modified to explain/fix its failure.
- [§2.5, §2.7, Eq. (3), Model Variants] There is a fundamental inconsistency in the training objective. Section 2.5 claims that 'the gradient from the watermark decoding loss... flow[s] back through the separator, guiding it to preserve watermark-relevant features.' However, Section 2.7 states that 'the separator is updated exclusively via an independent waveform-domain L1 loss' with a stop-gradient on the watermarked targets (Eq. 3). This means no bit-level gradient reaches the separator parameters, so the proposed end-to-end coupling is not actually implemented as described. Moreover, the Model Variants paragraph says the fine-tuned variant 'fine-tune[s] only the Demucs separator with the watermark decoding loss,' which contradicts the exclusive Lsep update. The paper must specify exactly which losses update the separator in each variant and reconcile the mechanism claim with the actual implementation. This is load-bearing b
- [Abstract; §2.5] The feasibility claim is explicitly scoped to a 'controlled verification pipeline, where the separator is part of the detector and can be selected or optimized together with the watermarking system.' In real multi-stem audio, the separator is typically an arbitrary third-party model outside the watermarking system's control. The paper's own baseline shows that with an off-the-shelf separator, the same watermarking backbone fails (AURA* 15–19% BER at Origin). Thus the paper does not demonstrate feasibility in the general setting suggested by the title and introduction. The authors should either clearly delimit the claim to 'co-designed detector-separator pipelines' and discuss the practical implications, or evaluate with unseen separators (e.g., Open-Unmix, Spleeter, a different Demucs checkpoint) to test whether the learned watermarks transfer beyond the exact co-trained separator.
- [§2.2, §2.6, Ref. [16]] The entire watermarking backbone is 'derived from' AURA [16], an unpublished, anonymized ICASSP 2026 manuscript whose full text is only available in the reviewer supplement. This creates a substantial reproducibility barrier: a reader cannot reconstruct AURA*, compare its architecture or training details, or judge whether the reported results depend on undocumented properties of [16]. Since all experiments in the paper are built on this backbone, the authors should either release code/checkpoints for AURA* or include a self-contained description of the backbone architecture and training recipe in the main text or a public appendix. At minimum, the dependency on an unpublished manuscript should be disclosed prominently in the paper itself, not only in the supplementary material.
minor comments (4)
- [§2.2] There is a duplicated sentence: 'The modified spectrogram is then inverted to the time domain using the original phase to produce the watermarked waveform. The modified spectrogram is inverted to wavedomain combined with the original phase to produce the watermarked waveform.' Please remove the redundancy and fix the typo ('wavedomain').
- [§2.7, Eq. (2)] The loss weights λ_stft and λ_spec are introduced but their values are never specified. The text says spectral weights are 'increased significantly' after 20k steps, but no concrete schedule is given. Provide the exact values or a table of hyperparameters; otherwise the training configuration is not reproducible.
- [§2.7.1, Table 1] The attack simulator defines 18 distortions, but the precise parameters are not fully specified (e.g., 'random time-frequency spectrogram masking' and 'random amplitude scaling in [−1.0,1.0]' leave wide latitude). More importantly, the robustness columns in Table 1 evaluate attacks from the same distribution used during training, so they measure in-distribution robustness rather than generalization to unseen distortions. Please state this explicitly or add held-out attack types.
- [Table 3] Table 3 reports only point estimates of SDR/SI-SDR with no confidence intervals, significance tests, or information about the number of evaluation clips. The differences (e.g., vocal SDR 9.36→8.89) may be within run-to-run variability. Report variance or error bars, and clarify whether 'Before'/'After' refers to joint training, finetuning, or both.
Circularity Check
Attack-robustness columns report the trained attack distribution as prediction; the joint-training claim is undercut by the paper's own §2.5 vs §2.7 contradiction and Table 1 vocal+accompaniment numbers.
specific steps
-
fitted input called prediction
[§2.7.1 Attack Simulation; Table 1]
"After 20,000 training steps, we introduce an attack simulator applying 18 random distortions to the mixed, watermarked audio prior to separation. For a structured evaluation, the subsequent robustness results will be presented in Table 1 according to these three distinct categories: •Basic/Noise: ... •Filter: ... •Time/Pitch: ... During joint training, these simulated attacks force the network to explicitly optimize for watermark detectability and separation fidelity under severe, real-world acoustic degradations."
Table 1 reports BER under exactly the categories (Basic/Noise, Filter, Time/Pitch) that were used as training-time simulated attacks. The model's parameters are explicitly optimized on those 18 distortions, so the 'robustness' columns measure in-sample performance on the training attack distribution, not held-out prediction to new distortion types. These numbers are fitted inputs relabeled as robustness results; they do not independently support the claim that the framework generalizes to separation artifacts beyond the trained attack set. The Origin column is likewise the training pipeline (embed -> mix -> Demucs -> decode), so the headline BER values are convergence on the training objective, though the comparison against baselines retains some independent content.
full rationale
The paper's own Table 1 undercuts the title's joint-training claim: on vocal+accompaniment, AURA*-Demucs joint gives 15.21% BER at Origin while AURA*-Demucs finetuned gives 1.25%; the joint model barely beats AURA* alone (19.23%). So the 'end-to-end coupling' mechanism, not the two-stage fine-tuned variant, is what makes the framework feasible on that benchmark. Additionally, §2.5 says the decoding loss gradient flows back through the separator, but §2.7 says the separator is updated exclusively via Lsep with stop-gradient on the watermarked targets; no bit-level gradient reaches the separator, contradicting the described mechanism. These are correctness and consistency problems rather than definitional circularity. The genuine circular step is the attack-simulator reporting: 18 distortions used in training are then presented as the robustness categories in Table 1, making those predictions in-sample. The self-citation to the unpublished AURA [16] is a transparency concern (anonymized supplementary, likely overlapping authors), but the AURA* numbers in Table 1 are measured in this paper, so the citation is not the load-bearing evidence for the joint-training result. The abstract's controlled verification scope limits real-world generalization, but that limitation is stated, not a circular step. Because one class of reported predictions reduces to the training distribution while the central claim still has independent empirical content (held-out audio, baseline comparisons), the appropriate score is 5 rather than 6+.
Axiom & Free-Parameter Ledger
free parameters (5)
- loss weights λ_stft and λ_spec =
not specified
- two-phase training schedule threshold =
20,000 steps
- attack simulator definitions (18 distortions) =
ranges per category (N, PK, LP, RS, etc.)
- loudness normalization target =
-16 LUFS
- payload size and carrier segment length =
32 bits / 2 s
axioms (5)
- domain assumption Demucs is fully differentiable and can be fine-tuned with backpropagated watermark loss without collapsing its separation function.
- domain assumption Magnitude-spectrogram modification with original-phase inversion is invertible enough for watermark decoding after separators introduce phase and spectral artifacts.
- domain assumption A 2-second carrier segment is sufficient for reliable 32-bit decoding under separation artifacts.
- domain assumption Speech+music and vocal+accompaniment mixtures at -16 LUFS are representative of realistic multi-source generative pipelines.
- ad hoc to paper The unpublished AURA backbone [16] is correct and effective as described, and its trained behavior is taken as a given base.
read the original abstract
Modern audio is created by mixing stems from different sources, raising the question: can we independently watermark each stem and recover all watermarks after separation? We study a separation-first, multi-stream watermarking framework --embedding distinct information into stems using unique keys but a shared structure, mixing, separating, and decoding from each output. A naive pipeline (robust watermarking + off-the-shelf separation) yields poor bit recovery, showing robustness to generic distortions does not ensure robustness to separation artifacts. To enable this, we study separation-aware watermarking in a controlled verification pipeline, where the separator is part of the detector and can be selected or optimized together with the watermarking system. Experiments on speech+music and vocal+accompaniment mixtures show substantial gains in post-separation recovery while maintaining perceptual quality.
Figures
Reference graph
Works this paper leans on
-
[1]
Audio watermarking has therefore emerged as a key technology for copyright protection and tamper detection [5]
Introduction Recent advances in AI-generated speech [1, 2, 3, 4] and neu- ral audio editing have made it increasingly difficult to protect the copyright and provenance of digital audio content. Audio watermarking has therefore emerged as a key technology for copyright protection and tamper detection [5]. Classical audio watermarking methods usually operat...
-
[2]
Method 2.1. Problem Setup: Separation-First Multi-Stream Water- marking We consider mixtures formed by multiple stems (e.g., vocal and accompaniment) that may originate from different sources. Each stem carries an independent watermark embedded with a distinct secret key. Given stems{x i}, watermarks are embed- ded independently to obtain{˜xi}. The waterm...
Pith/arXiv arXiv 2026
-
[3]
Conclusion We studied the feasibility of separation-first multi-stream audio watermarking and showed that robustness to generic distortions does not guarantee robustness to source separation. By jointly training the watermarking system and the separator end-to-end, we substantially improve post-separation decoding accuracy while maintaining competitive pe...
-
[4]
The LLMs were not used for ideation, methodology, experimental design, data analysis, or result interpretation
Generative AI Use Disclosure Large LanguageModels (LLMs) were used solely for manuscript polishing (e.g., rephrasing and grammar checks) to improve clarity and readability. The LLMs were not used for ideation, methodology, experimental design, data analysis, or result interpretation. All scientific content was produced and verified by the authors
-
[5]
Neural codec language mod- els are zero-shot text to speech synthesizers,
C. Wang, S. Chenet al., “Neural codec language mod- els are zero-shot text to speech synthesizers,”arXiv preprint arXiv:2301.02111, 2023
Pith/arXiv arXiv 2023
-
[6]
Audioldm: Text-to-audio generation with latent diffusion models,
H. Liu, Z. Chenet al., “Audioldm: Text-to-audio generation with latent diffusion models,”arXiv preprint arXiv:2301.12503, 2023
Pith/arXiv arXiv 2023
-
[7]
V oicebox: Text-guided multilin- gual universal speech generation at scale,
M. Le, A. Vyaset al., “V oicebox: Text-guided multilin- gual universal speech generation at scale,”arXiv preprint arXiv:2306.15687, 2023
Pith/arXiv arXiv 2023
-
[8]
Better speech synthesis through scaling,
J. Betker, “Better speech synthesis through scaling,”arXiv preprint arXiv:2305.07243, 2023
Pith/arXiv arXiv 2023
-
[9]
Dear: A deep-learning-based audio re-recording resilient watermarking,
C. Liu, J. Zhanget al., “Dear: A deep-learning-based audio re-recording resilient watermarking,”arXiv preprint arXiv:2212.02339, 2023
Pith/arXiv arXiv 2023
-
[10]
Techniques for data hiding,
W. Bender, D. Gruhl, N. Morimoto, and A. Lu, “Techniques for data hiding,”IBM Systems Journal, vol. 35, no. 3.4, pp. 313–336, 1996
1996
-
[11]
Wav- mark: Watermarking for audio generation,
G. Chen, Y . Wu, S. Liu, T. Liu, X. Du, and F. Wei, “Wav- mark: Watermarking for audio generation,”arXiv preprint arXiv:2308.12770, 2024
Pith/arXiv arXiv 2024
-
[12]
Maskmark: Robust neu- ralwatermarking for real and synthetic speech,
P. O’Reilly, Z. Jin, J. Su, and B. Pardo, “Maskmark: Robust neu- ralwatermarking for real and synthetic speech,” inProc. ICASSP, 2024, pp. 4650–4654
2024
-
[13]
Silentci- pher: Deep audio watermarking,
M. K. Singh, N. Takahashi, W. Liao, and Y . Mitsufuji, “Silentci- pher: Deep audio watermarking,” inProc. Interspeech, 2024, pp. 2235–2239
2024
-
[14]
Hybrid transformers for music source separation,
S. Rouard, F. Massa, and A. Défossez, “Hybrid transformers for music source separation,”arXiv preprint arXiv:2211.08553, 2022
Pith/arXiv arXiv 2022
-
[15]
Tf-gridnet: Making time-frequency domain mod- els great again for monaural speaker separation,
Z.-Q. Wang, S. Cornell, S. Choi, Y . Lee, B.-Y . Kim, and S. Watanabe, “Tf-gridnet: Making time-frequency domain mod- els great again for monaural speaker separation,”arXiv preprint arXiv:2209.03952, 2023
Pith/arXiv arXiv 2023
-
[16]
Open-unmix - a reference imple- mentation for music source separation,
F.-R. Stöter, S. Uhlichet al., “Open-unmix - a reference imple- mentation for music source separation,”Journal of Open Source Software, vol. 4, no. 41, p. 1667, 2019
2019
-
[17]
Spleeter: a fast and efficient music source separation tool with pre-trained models,
R. Hennequin, A. Khlifet al., “Spleeter: a fast and efficient music source separation tool with pre-trained models,”Journal of Open Source Software, vol. 5, no. 50, p. 2154, 2020
2020
-
[18]
Hybrid spectrogram and waveform source separa- tion,
A. Défossez, “Hybrid spectrogram and waveform source separa- tion,”arXiv preprint arXiv:2111.03600, 2022
Pith/arXiv arXiv 2022
-
[19]
Con- former: Convolution-augmented transformer for speech recogni- tion,
A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Con- former: Convolution-augmented transformer for speech recogni- tion,”arXiv preprint arXiv:2005.08100, 2020
Pith/arXiv arXiv 2005
-
[20]
Aura: A stegaformer-based scalable deep audio wa- termark with extreme robustness,
Anonymous, “Aura: A stegaformer-based scalable deep audio wa- termark with extreme robustness,” inProc. ICASSP, 2026
2026
-
[21]
Film: Visual reasoning with a general conditioning layer,
E. Perez, F. Strub, H. de Vries, V . Dumoulin, and A. C. Courville, “Film: Visual reasoning with a general conditioning layer,”arXiv preprint arXiv:1709.07871, 2017
Pith/arXiv arXiv 2017
-
[22]
Robust speech watermarking by a jointly trained embedder and detector using a dnn,
K. Pavlovi ´c, S. Kova ˇcevi´c, I. Djurovi ´c, and A. Wojciechowski, “Robust speech watermarking by a jointly trained embedder and detector using a dnn,”Digital Signal Processing, vol. 122, p. 103381, 2022
2022
-
[23]
Wave-u-net: A multi-scale neural network for end-to-end audio source separation,
D. Stoller, S. Ewert, and S. Dixon, “Wave-u-net: A multi-scale neural network for end-to-end audio source separation,” inProc. ISMIR, 2018
2018
-
[24]
Bidirectional recurrent neural networks,
M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,”IEEE Trans. Signal Process., vol. 45, no. 11, pp. 2673– 2681, 1997
1997
-
[25]
Musical source separation: An introduction,
E. Cano, D. FitzGerald, A. Liutkus, M. D. Plumbley, and F.-R. Stöter, “Musical source separation: An introduction,”IEEE Sig- nal Processing Magazine, vol. 36, no. 1, pp. 31–40, 2019
2019
-
[26]
Performance mea- surement in blind audio source separation,
E. Vincent, R. Gribonval, and C. Févotte, “Performance mea- surement in blind audio source separation,”IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 4, pp. 1462–1469, 2006
2006
-
[27]
Fma: A dataset for music analysis,
M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “Fma: A dataset for music analysis,”arXiv preprint arXiv:1612.01840, 2016
Pith/arXiv arXiv 2016
-
[28]
Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,
H. He, Z. Shanget al., “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,”arXiv preprint arXiv:2407.05361, 2024
Pith/arXiv arXiv 2024
-
[29]
The musdb18 corpus for music separation,
Z. Rafii, A. Liutkus, F.-R. Stöter, S. I. Mimilakis, and R. Bittner, “The musdb18 corpus for music separation,” 2017
2017
-
[30]
Bigvgan: A universal neural vocoder with large-scale training,
S.-g. Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon, “Bigvgan: A universal neural vocoder with large-scale training,” arXiv preprint arXiv:2206.04658, 2023
Pith/arXiv arXiv 2023
-
[31]
Noise-to-mask ratio loss for deep neural network based audio watermarking,
M. Moritz, T. Olán, and T. Virtanen, “Noise-to-mask ratio loss for deep neural network based audio watermarking,” inProc. IS2, 2024, pp. 1–6
2024
-
[32]
Visqol v3: An open source production ready objec- tive speech and audio metric,
M. Chinen, F. S. C. Lim, J. Skoglund, N. Gureev, F. O’Gorman, and A. Hines, “Visqol v3: An open source production ready objec- tive speech and audio metric,”arXiv preprint arXiv:2004.09584, 2020
Pith/arXiv arXiv 2004
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.