Pith. sign in

REVIEW 4 major objections 6 minor 45 references

Adapting Diffusion-Based Music Synthesis to Speech and Singing Voice Conversion

T0 review · 4 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A diffusion model originally designed to synthesize multi-instrument music can be adapted into a voice conversion system that matches or exceeds a dedicated speech/singing converter in naturalness and performer similarity, while preserving

desk verdict Solid cross-domain transfer demonstration, but the 'naturalness at parity' claim rests on same-identity resynthesis, not true voice conversion. read the letter →

arxiv 2607.13278 v2 pith:FBUN3QMQ submitted 2026-07-14 cs.SD

Ben Maman , Frank Zalkow , Hans-Ulrich Berendes , Paolo Sani , Christian Dittmar , Meinard Müller This is my paper · ORCID
classification cs.SD
keywords diffusionmodelsvoiceconversionsingingmusicsynthesisphoneticposteriorgramspitchcontrolperformerconditioningcross-domaintransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a diffusion model built for instrumental music synthesis can be turned into a voice conversion system that handles both speech and singing. The authors extend the model's musical-score conditioning—a piano-roll-like representation—to include phonetic posteriorgrams (PPG) for linguistic content and f0 contours for pitch, and reinterpret the 'timbre' conditioning as speaker or singer identity via feature-wise linear modulation. In listening tests and objective metrics, the adapted music model matches or surpasses a purpose-built voice conversion model in naturalness and performer similarity, while preserving pitch contours. The same off-the-shelf feature extractors let the model train without manual annotations, though the authors report lower phonetic fidelity than the specialist and a quality drop when instrumental data is added. The overall case is that cross-domain transfer can unify speech, singing, and music generation in one framework.

What carries the argument

The load-bearing mechanism is the conditioning stack: time-varying conditions (a piano-roll channel for notes, a PPG channel for phonetics, an f0 channel for pitch) are concatenated along the channel axis and fused by a conditioning encoder, then injected into the diffusion decoder via cross-attention; global conditions (performer identity, diffusion timestep) are applied as feature-wise linear modulation (FiLM) scale-and-shift in each block. Condition dropout allows classifier-free guidance and partial-condition generation, and pitch-range adaptation mean-shifts the source f0 to the target performer's range in semitone units. This stack lets one model represent musical notes, phonetic conte

What would settle it

Run the same naturalness listening test in a true conversion setup—source speaker A, target speaker B—and compare scores to the paper's same-identity reconstruction scores. If naturalness drops substantially below the dedicated voice-conversion baseline (or below the paper's reported numbers), the claim that the adapted model matches a specialist in naturalness would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that a generative architecture originally tuned for multi-instrument music—an attention-based diffusion decoder conditioned by a fused time-varying feature representation—is a sufficient substrate for human voice conversion. Conditioned on phonetic posteriorgrams, f0 contours, and a performer embedding, the model generates speech and singing that is judged as natural as or more natural than a specialized voice-conversion system, and its performer conditioning is stronger. Pitch contours are reproduced within 1–3 percentage points of the specialist baseline. The paper also reports two trade-offs: phonetic fidelity is worse than the specialist, and mixing instrumental trai

Load-bearing premise

The naturalness tests resynthesize the same speaker who supplied the content features, so the paper's headline claim assumes that this same-identity reconstruction quality carries over to true cross-performer conversion, which the study only tests for similarity, not naturalness.

Editorial extensions

If this is right

  • A single conditional model can in principle generate speech, singing, and instrumental music without retraining, because the same conditioning stack handles notes, phonetics, pitch, and performer.
  • Off-the-shelf feature extractors suffice for training large-scale voice-conversion models without manual annotations, enabling self-supervised scaling across domains.
  • Pitch-range adaptation in the semitone domain enables gender- and range-consistent transfer, as shown by the similarity test where the source and target pitch ranges are aligned.
  • Including instrumental data degrades vocal naturalness by roughly 10–15 points, so unified models face a real quality–flexibility trade-off that needs architectural or training mitigation.
  • Performer conditioning via FiLM applied in both encoder and decoder yields stronger similarity than the specialist baseline, suggesting a concrete architectural lesson for future VC systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would replace the triplet-trained speaker/singer embedding with an open-vocabulary or text-promptable identity embedding; if the conditioning stack is general, zero-shot conversion to arbitrary named voices should work without retraining.
  • The phonetic-fidelity gap noted in the paper (guttural 'r' becoming a different 'r') suggests the model is learning a target-performer phonetic style; a listening study comparing phonetic preservation against stylistic adaptation would decide whether that is a bug or a feature.
  • Because the same conditioning stack handles notes, phonetics, pitch, and performer, a single model could in principle generate spoken dialogue, sung melody, and instrumental accompaniment from one fused condition vector; the paper does not yet demonstrate this end-to-end, but its mixed-data results are the first step.
  • Reconstruction and conversion may diverge; a cycle-consistency objective that enforces content preservation across identity swaps could close the gap between the paper's reconstruction-based naturalness scores and real conversion deployments.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes treating an existing T5-based diffusion music-synthesis model as a voice conversion backbone. The conditioning is extended from piano-roll notes to phonetic posteriorgrams (PPG) and f0 contours, and FiLM-based timbre conditioning is reinterpreted as performer identity via TRILL embeddings. The authors compare T5-Voc and T5-All with PAD-Voc and MAC-Voc on internal speech and singing corpora. They report perceptual naturalness tests (speech p=0.69, singing p=7.24e-3 for T5-Voc vs MAC-Voc), a singer similarity test (p<1e-7), FAD, f0 accuracy, PPG distance, and mixed audio FAD. They conclude that the adapted model matches or surpasses a dedicated VC system in naturalness and performer similarity, while noting lower phonetic fidelity and a quality drop when instrumental data is included.

Significance. The paper's central idea is attractive and the engineering contribution is solid: reusing a music-diffusion conditioning stack and off-the-shelf feature extractors for voice conversion is a plausible route toward unified audio generation. The evaluation is more comprehensive than typical VC submissions, with multiple listening tests and objective metrics, and the authors transparently report negative results (phonetic fidelity, T5-All degradation). If the cross-performer naturalness gap identified below is closed, the result would be significant for the audio-generation community. As it stands, the strongest headline claim is supported only by a resynthesis protocol.

major comments (4)
  1. [§4.2.1, Figs. 2–3] The naturalness listening tests use a reconstruction protocol in which f0, PPG, and performer identity are all extracted from the same source excerpt, so the target identity equals the source. This measures resynthesis quality, not conversion quality. The only true cross-performer perception test is the singer similarity test (§4.2.2), which does not measure naturalness. The abstract's claim that the model 'matches or surpasses a dedicated voice conversion system in terms of naturalness' is therefore not directly supported for actual voice conversion. I request either a cross-performer naturalness test (e.g., source speech/singing converted to a target identity and rated for naturalness) or a clearly qualified statement that naturalness was evaluated on same-identity reconstruction.
  2. [§4.2.1, Table 1] The 'matches' claim for speech rests on p=0.69 from a Wilcoxon test with 25 listeners. A non-significant difference is not evidence of equivalence. Report a confidence interval for the mean/median rating difference or use a non-inferiority/equivalence test; additionally, the pairwise p-values among four models are reported without multiple-comparison control. The singing comparison is significant at p=7.24e-3, but the speech equivalence claim should be made statistically explicit.
  3. [§4.2.2] The perceptual performer-similarity test covers singing only. The abstract promises 'performer similarity' across speech and singing; for speech, similarity rests only on Performer-FAD (Table 2). Add a speech similarity listening test or qualify the claim to singing similarity. Also, the test equalizes pitch range by transposing the conditioning f0 to the target range, which may interact with identity perception; a brief discussion of this potential confound would help.
  4. [§4.1] All training and test data are internal, and no code or model weights are released (only a project page). The headline comparisons are therefore not independently reproducible. Given that the underlying architecture and feature extractors are public, I strongly encourage releasing the inference code and, if possible, evaluating on a public VC/SVC benchmark so that readers can verify the magnitude of the reported differences.
minor comments (6)
  1. [§3.5] Typo: 'conditioned using the the PAD-VC' should read 'conditioned using the PAD-VC'.
  2. [§4.2] The naming is inconsistent: PAD-VC in Section 3.5 becomes PAD-Voc in Section 4.2. Please unify the notation.
  3. [Figs. 2–4] The p-values are not defined in the captions. State that they are pairwise two-sided Wilcoxon signed-rank test results.
  4. [§3.4] The source-separation quality is said to be verified by 'informal listening tests.' A short quantitative check or a more formal listening protocol would make this statement stronger.
  5. [§4.2.1] The protocol is described as similar to MUSHRA, but it is not explicitly stated which system is the lower anchor and how the hidden reference is used in grading. Clarify the anchor/reference setup.
  6. [§3.3, Eq. (1)] The target mean μ_tar_p is used for pitch range adaptation, but its estimation from the target performer's data is not specified. Please define it precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivational circularity; measured claims are supported by held-out tests and external baselines.

full rationale

The paper contains no fitted-input-called-prediction or definitional reduction: the headline naturalness, similarity, pitch, and phonetic results are measured on held-out test sets with independent listeners and off-the-shelf estimators, and no target metric is equal to a training input by construction. The self-citations ([6,7] for the T5 music-synthesis architecture, [12,29] for the PAD-VC conditioning/baseline) are provenance and implementation details, not load-bearing uniqueness or ansatz-justifying citations; the models are publicly available, and the central comparison includes an externally specified FlowMAC/MatchaTTS-based system and external feature extractors (CREPE, wav2vec2, TRILL, BigVGAN). Using CREPE both to condition on f0 and to evaluate pitch fidelity, or TRILL both for performer conditioning and for FAD, does not make the evaluation circular: the model must still generate audio that the extractor scores correctly, and FAD is computed against real reference recordings. The naturalness listening tests in §4.2.1 reconstruct the original audio with target identity equal to source, so the abstract's unqualified "voice conversion naturalness" claim is broader than the protocol directly tests; that is a construct-validity/external-validity concern, not an equation-level circularity. The paper itself flags limitations — phonetic fidelity deficits in §4.2.5 and vocal-quality degradation with instrumental data in the abstract — which further indicates the results are not forced to match the claims. No passage exhibits a specific reduction such as Eq. X = Eq. Y or a fitted parameter renamed as a prediction; therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central comparison rests on standard diffusion/CFG training; on three off-the-shelf extractors (wav2vec2 PPG, CREPE f0, TRILL performer) providing adequate conditioning despite being trained on speech or generic audio; on the BigVGAN vocoder being neutral; and on evaluation metrics built from the same extractor families used for conditioning. These are domain assumptions rather than derived results, and the paper's own phonetic-fidelity results partially undermine the PPG assumption. No new physical or formal entities are introduced.

free parameters (3)
  • p_drop (condition dropout probabilities) = unreported
    §3.2 sets per-condition dropout p_drop^c and joint dropout p_drop_all by hand; values are not given, yet they control conditioning strength and are required for classifier-free guidance, affecting all downstream quality/similarity results.
  • classifier-free guidance scale = unreported
    §3.2 enables CFG; the guidance weight used at sampling is not reported, and it directly trades naturalness against conditioning fidelity in the listening tests.
  • data-type one-hot condition (T5-All) = vocal-only vs vocal+instrumental
    §4.2 adds a manually designed one-hot data-type channel to the performer condition to handle mixed-domain training; an ad hoc architectural choice specific to this paper.
assumptions (5)
  • domain assumption wav2vec2 XLS-R TIMIT phoneme PPG provides meaningful phonetic conditioning for singing and non-English speech
    §3.4 conditions the model on PPG from a speech-only extractor; §4.2.5 evaluation shows T5-Voc has worse phonetic fidelity than MAC-Voc and converts German guttural 'r' to English 'r', partially contradicting this premise.
  • domain assumption CREPE f0 estimates are accurate enough to serve both as conditioning and as ground truth for pitch-control metrics
    §3.4 uses CREPE for conditioning; §4.2.4 uses CREPE to evaluate pitch preservation (RPA/Ov50/Ov25), so the metric measures re-extraction of the input feature (mild evaluator circularity); cites >95% accuracy [20].
  • domain assumption TRILL embeddings capture performer identity independent of pitch and content
    §3.4 uses TRILL for performer conditioning; §4.2.2 similarity test shows conditioning on reference vs other performers separates ratings (3.25 vs 1.76 for T5-Voc), supporting but not proving this.
  • standard math Diffusion training with L1 noise-prediction loss and classifier-free guidance is a valid generative backbone for spectrogram synthesis
    §3.1-3.2 follows established diffusion/CFG methodology (Ho & Salimans [19]); standard in the field.
  • domain assumption BigVGAN vocoder performance is neutral across compared models
    §3 uses a single off-the-shelf vocoder for all models; if BigVGAN mismatches one model's mel-space statistics, comparisons could be skewed. The paper itself notes quality may suffer for vocal-only synthesis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adapting Diffusion-Based Music Synthesis to Speech and Singing Voice Conversion." pith.science (2026). https://pith.science/paper/FBUN3QMQ

@misc{pith2026260713278,
  author       = {Pith},
  title        = {Pith review of: Adapting Diffusion-Based Music Synthesis to Speech and Singing Voice Conversion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FBUN3QMQ}},
  note         = {Machine review of arXiv:2607.13278}
}
read the original abstract

Recent diffusion-based generative models have achieved strong results in domain-specific audio generation tasks such as speech, singing, and instrumental music synthesis. However, these models are typically specialized and do not generalize well to mixed or intermediate audio types. In this work, we adapt a diffusion-based model originally designed for multi-instrument music synthesis to voice conversion, covering both speech and singing within a unified framework. Specifically, we extend musical note-based conditioning to include phonetic posteriorgrams (PPGs) and pitch contours, and reinterpret timbre conditioning as speaker or singer identity via feature-wise linear modulation. Experiments show that the adapted model matches or surpasses a dedicated voice conversion system in terms of naturalness and performer similarity, while maintaining accurate pitch control across speech and singing. At the same time, we observe limitations in phonetic fidelity and a degradation in vocal quality when incorporating instrumental training data. Furthermore, we demonstrate that off-the-shelf feature extractors provide effective conditioning signals, enabling large-scale self-supervised training without manual annotations. These results highlight the potential of cross-domain model transfer towards unified audio generation systems capable of handling speech, singing, and music. Qualitative samples can be found on our project page: https://benadar293.github.io/voice-conversion

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 2 linked inside Pith

  1. [1]

    INTRODUCTION Generative modeling has become a central paradigm in modern ma- chine learning, enabling the synthesis of complex data across modal- ities such as images and audio. In the audio domain, recent advances have led to highly realistic generation of speech [1], singing [2, 3, 4], and instrumental music [5, 6, 7], based on neural architectures that...

  2. [2]

    CAT”: Concatenation along the channel axis, “C.A

    RELA TED WORK Speech V oice Conversion (VC)aims to transform a source speech audio sample to match the speaker identity of a target speaker [8, 9]. High-quality conversion is achieved using various generative mod- els, such as generative adversarial networks (GANs) [8], diffusion models [1], or more recently flow-matching models [10, 11]. To capture time-...

    arXiv 2026

  3. [3]

    To convert the generated mel spectrogram into a waveform, we use an off-the-shelf BigVGAN vocoder [17]

    METHOD Following previous work in speech, singing, and instrumental music synthesis, our acoustic model is based on mel spectrogram diffusion. To convert the generated mel spectrogram into a waveform, we use an off-the-shelf BigVGAN vocoder [17]. We choose a general pur- pose vocoder rather than a vocal-only one—although this may come at the expense of qu...

  4. [4]

    completely different person,

    EXPERIMENTS 4.1. Datasets We use a large compound dataset of unannotated audio from dif- ferent domains, including speech, singing, instrumental music, and mixed recordings. While these datasets are internal, all feature ex- tractors used to automatically generate pseudo-annotations are pub- licly available, allowing replication of our approach with equiv...

    arXiv

  5. [5]

    CONCLUSION In this work we evaluated the performance of an attention-based dif- fusion model adapted from instrumental music synthesis to human voice conversion. Through an extensive evaluation across speech and singing we have shown it can match—or even surpass—a dedi- cated voice conversion model in terms of quality and performer sim- ilarity. While our...

  6. [6]

    500643750 (MU 2686/15-1)

    ACKNOWLEDGMENTS This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Grant No. 500643750 (MU 2686/15-1). The International Audio Laboratories Erlan- gen are a joint institution of the Friedrich-Alexander-Universit ¨at Erlangen-N¨urnberg (FAU) and Fraunhofer Institute for Integrated Circuits IIS. This work was...

  7. [7]

    Diffusion-based voice conversion with fast maximum likelihood sampling scheme,

    V . Popov, I. V ovk, V . Gogoryan, T. Sadekova, M. S. Kudinov, and J. Wei, “Diffusion-based voice conversion with fast maximum likelihood sampling scheme,” inProceedings of the International Conference on Learning Representations (ICLR), 2022

    2022

  8. [8]

    DiffSinger: Singing voice synthesis via shallow diffusion mechanism,

    J. Liu, C. Li, Y . Ren, F. Chen, and Z. Zhao, “DiffSinger: Singing voice synthesis via shallow diffusion mechanism,” in Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI

Show all 45 references
  1. [9]

    An overview of voice conversion and its challenges: From statistical modeling to deep learning,

    B. Sisman, J. Yamagishi, S. King, and H. Li, “An overview of voice conversion and its challenges: From statistical modeling to deep learning,”IEEE ACM Trans. Audio Speech Lang. Process., vol. 29, pp. 132–157, 2021

    2021

  2. [10]

    Expres- siveSinger: Multilingual and multi-style score-based singing voice synthesis with expressive performance control,

    S. Dai, M.-Y . Liu, R. Valle, and S. Gururani, “Expres- siveSinger: Multilingual and multi-style score-based singing voice synthesis with expressive performance control,” inACM Multimedia 2024

    2024

  3. [11]

    Everyone- Can-Sing: Zero-shot singing voice synthesis and conversion with speech reference,

    S. Dai, Y . Wang, R. B. Dannenberg, and Z. Jin, “Everyone- Can-Sing: Zero-shot singing voice synthesis and conversion with speech reference,” inProceedings of the IEEE In- ternational Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2025, pp. 1–5

    2025

  4. [12]

    Multi-instrument music synthesis with spectrogram diffusion,

    C. Hawthorne, I. Simon, A. Roberts, N. Zeghidour, J. Gardner, E. Manilow, and J. H. Engel, “Multi-instrument music synthesis with spectrogram diffusion,” inProceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2022, pp. 598–607

    2022

  5. [13]

    Per- formance conditioning for diffusion-based multi-instrument music synthesis,

    B. Maman, J. Zeitler, M. M ¨uller, and A. H. Bermano, “Per- formance conditioning for diffusion-based multi-instrument music synthesis,” inProceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024, pp. 5045–5049

    2024

  6. [14]

    Multi- aspect conditioning for diffusion-based music synthesis: En- hancing realism and acoustic control,

    B. Maman, J. Zeitler, M. M ¨uller, and A. H. Bermano, “Multi- aspect conditioning for diffusion-based music synthesis: En- hancing realism and acoustic control,”IEEE/ACM Trans. on Audio, Speech, and Lang. Process., pp. 1–14, 2024

    2024

  7. [15]

    StarGAN- VC: Non-parallel many-to-many voice conversion using star generative adversarial networks,

    H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo, “StarGAN- VC: Non-parallel many-to-many voice conversion using star generative adversarial networks,” inIEEE Spoken Language Technology Workshop (SLT), 2018, pp. 266–273

    2018

  8. [16]

    Simple and controllable music gen- eration,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. D´efossez, “Simple and controllable music gen- eration,”Advances in Neural Information Processing Systems, vol. 36, pp. 47 704–47 720, 2023

    2023

  9. [17]

    Matcha-TTS: A fast TTS architecture with conditional flow matching,

    S. Mehta, R. Tu, J. Beskow, ´E. Sz ´ekely, and G. E. Henter, “Matcha-TTS: A fast TTS architecture with conditional flow matching,” inProceedings of the IEEE International Confer- ence on Acoustics, Speech, and Signal Processing (ICASSP), 2024, pp. 11 341–11 345

    2024

  10. [18]

    FlowMac: Con- ditional flow matching for audio coding at low bit rates,

    N. Pia, M. Strauss, M. Multrus, and B. Edler, “FlowMac: Con- ditional flow matching for audio coding at low bit rates,” in IEEE International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), 2025, pp. 1–5

    2025

  11. [19]

    PAD-VC: A prosody-aware decoder for any- to-few voice conversion,

    A. K. Kalita, C. Dittmar, P. Sani, F. Zalkow, E. A. P. Habets, and R. Patra, “PAD-VC: A prosody-aware decoder for any- to-few voice conversion,” inProceedings of the International Workshop on Acoustic Signal Enhancement (IWAENC), 2024, pp. 389–393

    2024

  12. [20]

    High-fidelity neu- ral phonetic posteriorgrams,

    C. Churchwell, M. Morrison, and B. Pardo, “High-fidelity neu- ral phonetic posteriorgrams,” inProceedings of the IEEE Inter- national Conference on Acoustics, Speech, and Signal Process- ing (ICASSP), 2024, pp. 823–827

    2024

  13. [21]

    SingStyle111: A multilingual singing dataset with style trans- fer,

    S. Dai, Y . Wu, S. Chen, R. Huang, and R. B. Dannenberg, “SingStyle111: A multilingual singing dataset with style trans- fer,” inProceedings of the International Society for Music In- formation Retrieval Conference (ISMIR), 2023, pp. 765–773

    2023

  14. [22]

    NaturalSpeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers,

    K. Shen, Z. Ju, X. Tan, E. Liu, Y . Leng, L. He, T. Qin, S. Zhao, and J. Bian, “NaturalSpeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers,” inProceedings of the International Conference on Learning Representations (ICLR), 2024

    2024

  15. [23]

    Hybrid transformers for music source separation,

    S. Rouard, F. Massa, and A. D ´efossez, “Hybrid transformers for music source separation,” inProceedings of the IEEE In- ternational Conference on Acoustics, Speech, and Signal Pro- cessing (ICASSP), 2023

    2023

  16. [24]

    BigVGAN: A universal neural vocoder with large-scale train- ing,

    S. Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon, “BigVGAN: A universal neural vocoder with large-scale train- ing,” inProceedings of the International Conference on Learn- ing Representations (ICLR), 2023

    2023

  17. [25]

    FiLM: Visual reasoning with a general conditioning layer,

    E. Perez, F. Strub, H. de Vries, V . Dumoulin, and A. C. Courville, “FiLM: Visual reasoning with a general conditioning layer,” inProceedings of the AAAI Conference on Artificial Intelligence, 2018, pp. 3942–3951

    2018

  18. [26]

    Classifier-free diffusion guidance,

    J. Ho and T. Salimans, “Classifier-free diffusion guidance,” arXiv, vol. abs/2207.12598, 2022

    2022 arXiv

  19. [27]

    CREPE: A con- volutional representation for pitch estimation,

    J. W. Kim, J. Salamon, P. Li, and J. P. Bello, “CREPE: A con- volutional representation for pitch estimation,” inProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 161–165

    2018

  20. [28]

    wav2vec 2.0: A framework for self-supervised learning of speech rep- resentations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech rep- resentations,” inAdvances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  21. [29]

    XLS-R: Self-supervised cross-lingual speech representation learning at scale,

    A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y . Saraf, J. Pino, A. Baevski, A. Con- neau, and M. Auli, “XLS-R: Self-supervised cross-lingual speech representation learning at scale,” inProceedings of the Annual Conference of the Interna...

    2022

  22. [30]

    Schubert Winterreise dataset: A multimodal scenario for music analysis,

    C. Weiß, F. Zalkow, V . Arifi-M¨uller, M. M ¨uller, H. V . Koops, A. V olk, and H. Grohganz, “Schubert Winterreise dataset: A multimodal scenario for music analysis,”ACM Journal on Computing and Cultural Heritage (JOCCH), vol. 14, no. 2, pp. 25:1–18, 2021

    2021

  23. [31]

    Onsets and Frames: Dual-objective piano transcription,

    C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Raf- fel, J. H. Engel, S. Oore, and D. Eck, “Onsets and Frames: Dual-objective piano transcription,” inProceedings of the In- ternational Society for Music Information Retrieval Confer- ence, (ISMIR), 2018, pp. 50–57

    2018

  24. [32]

    Enabling factorized piano music modeling and generation with the MAESTRO dataset,

    C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C. A. Huang, S. Dieleman, E. Elsen, J. H. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” inProceedings of the International Conference on Learning Representations (ICLR), 2019

    2019

  25. [33]

    Unaligned supervision for au- tomatic music transcription in the wild,

    B. Maman and A. H. Bermano, “Unaligned supervision for au- tomatic music transcription in the wild,” inProceedings of the International Conference on Machine Learning (ICML), 2022, pp. 14 918–14 934

    2022

  26. [34]

    Count The Notes: Histogram-based supervision for automatic music tran- scription,

    J. Yaffe, B. Maman, M. M ¨uller, and A. Bermano, “Count The Notes: Histogram-based supervision for automatic music tran- scription,” inProceedings of the International Society for Mu- sic Information Retrieval Conference (ISMIR), Daejeon, South Korea, 2025, pp. 469–476

    2025

  27. [35]

    Towards learning a universal non-semantic representation of speech,

    J. Shor, A. Jansen, R. Maor, O. Lang, O. Tuval, F. de Chau- mont Quitry, M. Tagliasacchi, I. Shavitt, D. Emanuel, and Y . Haviv, “Towards learning a universal non-semantic representation of speech,” inProc. of the Annual Confer- ence of the International Speech Communication A...

    2020

  28. [36]

    Bridging the training–inference gap in TTS: Training strategies for robust generative postprocessing for low-resource speakers,

    F. Zalkow, P. Sani, K. K. Lakshminarayana, E. A. P. Habets, N. Pia, and C. Dittmar, “Bridging the training–inference gap in TTS: Training strategies for robust generative postprocessing for low-resource speakers,” inProceedings of the Conference of the International Speech Com...

    2025

  29. [37]

    Jensen–Shannon divergence and Hilbert space embedding,

    B. Fuglede and F. Topsoe, “Jensen–Shannon divergence and Hilbert space embedding,” inIEEE International symposium on Information theory (ISIT), 2004, p. 31

    2004

  30. [38]

    Fr ´echet Audio Distance: A reference-free metric for evaluating music enhancement algorithms,

    K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, “Fr ´echet Audio Distance: A reference-free metric for evaluating music enhancement algorithms,” inProceedings of the Annual Conference of the International Speech Communication Association (Interspeech), 2019, pp. 2350–2354

    2019

  31. [39]

    ITU-R Rec. BS.1534-3: Method for the subjective assessment of interme- diate quality levels of coding systems,

    International Telecommunications Union, “ITU-R Rec. BS.1534-3: Method for the subjective assessment of interme- diate quality levels of coding systems,” 2015

    2015

  32. [40]

    webMUSHRA—a comprehen- sive framework for web-based listening tests,

    M. Schoeffler, S. Bartoschek, F.-R. St ¨oter, M. Roess, S. West- phal, B. Edler, and J. Herre, “webMUSHRA—a comprehen- sive framework for web-based listening tests,”Journal of Open Research Software, vol. 6, no. 1, 2018

    2018

  33. [41]

    Melody transcription from music audio: Ap- proaches and evaluation,

    G. E. Poliner, D. P. Ellis, A. F. Ehmann, E. G ´omez, S. Stre- ich, and B. Ong, “Melody transcription from music audio: Ap- proaches and evaluation,”IEEE Trans. on Audio, Speech, and Lang. Process., vol. 15, no. 4, pp. 1247–1256, 2007

    2007

  34. [42]

    Melody extraction from poly- phonic music signals using pitch contour characteristics,

    J. Salamon and E. G ´omez, “Melody extraction from poly- phonic music signals using pitch contour characteristics,”IEEE Transactions on Audio, Speech, and Language Processing, vol. 20, no. 6, pp. 1759–1770, 2012

    2012

  35. [43]

    MIR EV AL: A transparent im- plementation of common MIR metrics,

    C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. P. W. Ellis, “MIR EV AL: A transparent im- plementation of common MIR metrics,” inProceedings of the International Society for Music Information Retrieval Confer- ence (ISMIR), 2014, pp. 367–372

    2014

  36. [45]

    Markov processes over denumerable prod- ucts of spaces, describing large systems of automata,

    L. N. Vaserstein, “Markov processes over denumerable prod- ucts of spaces, describing large systems of automata,”Prob- lemy Peredachi Informatsii, vol. 5, no. 3, pp. 64–72, 1969

    1969

  37. [2022]

    11 020–11 028

    AAAI Press, 2022, pp. 11 020–11 028

    2022

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.