Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

MIDI-VALLE: Improving Expressive Piano Performance Synthesis Through Neural Codec Language Modelling

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MIDI-VALLE claims that expressive piano audio can be synthesised from performance MIDI by treating both as discrete token streams, cutting Fréchet Audio Distance by more than 75 percent against the prior baseline.

desk verdict MIDI-VALLE is a real step forward for codec-LM piano synthesis, but the headline 75% FAD claim is partly circular and needs an independent-embedding check before it can be taken at face value. read the letter →

arxiv 2507.08530 v1 pith:ENWREVHZ submitted 2025-07-11 cs.SD cs.AIeess.AS

Jingjing Tang , Xin Wang , Zhe Zhang , Junichi Yamagishi , Geraint Wiggins , George Fazekas This is my paper
classification cs.SDcs.AIeess.AS
keywords expressivepianoperformancesynthesisneuralcodeclanguagemodelMIDI-to-audioOctupleMIDItokenizationPiano-Encodeczero-shotaudioconditioningFréchetDistancetwo-stagemusicpipeline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that expressive piano audio can be synthesised directly from performance MIDI by modelling both as discrete token streams, rather than through the usual piano-roll and spectrogram route. It adapts the VALLE speech-synthesis architecture so that an autoregressive transformer predicts the first audio-codec tokens from MIDI, while a non-autoregressive transformer predicts the remaining tokens conditioned on a three-second audio prompt. Trained on a large and acoustically varied set of transcribed piano performances, the resulting system, MIDI-VALLE, claims more than 75 percent lower Fréchet Audio Distance than the previous best baseline on the ATEPP and Maestro datasets, and it won the listening comparison 202 votes to 58. If true, the practical payoff is a single synthesis stage that accepts MIDI from any expressive-performance renderer without re-tuning, and that lets the user control timbre and room acoustics by choosing the reference audio.

What carries the argument

The load-bearing machinery is the pair of discrete tokenisations plus a two-decoder codec language model. Octuple MIDI tokenisation turns each note into parallel tokens for pitch, velocity, duration, inter-onset interval, and metrical position, giving finer timing than the fixed-grid piano roll used by earlier synthesis systems. Piano-Encodec, an Encodec-style codec fine-tuned on piano audio with four residual vector-quantisation codebooks, converts 32 kHz audio into 50 Hz token frames; the autoregressive decoder predicts the first codebook from the MIDI and MIDI prompt, and the non-autoregressive decoder predicts codebooks two through four under the influence of a three-second reference audio prompt. That prompt is what lets the model carry timbre, loudness, and room acoustics from a chosen performance into the synthesis.

What would settle it

Run the same FAD comparison on the generated samples using an independently trained audio encoder that the model never saw during training; if the 75 percent FAD reduction shrinks to a small margin while the 202-to-58 listening preference persists, the metric was biased, whereas a persistent FAD gap would corroborate the perceptual claim.

Watch

Extended reading notes

Core claim

MIDI-VALLE's central claim is that token-level alignment between MIDI and audio removes the generalisation bottleneck in piano performance synthesis. The model encodes a performance MIDI with Octuple tokenisation, which uses separate vocabularies for pitch, velocity, duration, inter-onset interval, and metrical position, and it encodes the paired audio with Piano-Encodec, a four-level residual vector-quantised codec fine-tuned on piano data. The autoregressive decoder predicts the first codebook from the MIDI alone; the non-autoregressive decoder predicts the remaining three codebooks conditioned on the first three seconds of a reference recording. In the paper's evaluation this design lowers FAD from 11.014 to 3.329 on ATEPP and from 34.479 to 11.281 on Maestro relative to the M2A baseline, a reduction of over 75 percent, and listeners preferred it 202 to 58 overall. The paper also reports that the same model handles transcribed MIDI, recorded MIDI, and MIDI from three different performance-rendering systems without fine-tuning, while its performance on jazz from the Pijama dataset degrades, which it attributes to the classical-only training data.

Load-bearing premise

The main result depends on the assumption that Fréchet Audio Distance computed with embeddings from Piano-Encodec, the exact codec MIDI-VALLE is trained to mimic, is a fair measure of perceptual audio quality; if that distance favours the model's own training target, the 75 percent gap overstates real-world superiority.

Editorial extensions

If this is right

  • A two-stage score-to-audio pipeline can be built with a single token-based synthesis stage that accepts performance MIDI from models with different note, timing, and pedal conventions, so the renderer and synthesizer no longer need to be fine-tuned as a pair.
  • Training synthesis on transcribed performance MIDI transfers to recorded MIDI, meaning large transcribed corpora can substitute for scarce recorded-MIDI data in real applications.
  • Users gain prompt-level control of acoustic environment and timbre: swapping the three-second reference audio changes the room and instrument character of the output.
  • The output ceiling is tied to the codec: since MIDI-VALLE scores closer to Piano-Encodec reconstructions than to ground truth, improvements in codec fidelity should flow directly into synthesis quality.
  • Classical-trained synthesis does not yet transfer across genre boundaries, so extending training data remains the limiting factor for jazz and other styles.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The headline 75 percent FAD gap may partly measure how well the model reproduces its own training-codec target, because FAD is computed with Piano-Encodec embeddings, the same codec whose tokens the model predicts; an independent audio embedding could yield a smaller gap.
  • Inference: The practical bottleneck for real deployments may be the codec rather than the language model, since outputs are closer to codec reconstructions than to original recordings, so a better piano codec should improve perceived quality without retraining the language model.
  • Inference: The three-second prompt alignment requirement suggests a concrete testable improvement: train with randomly misaligned prompt pairs or add a prompt-selection mechanism, then measure whether the note-onset errors at segment starts disappear.
  • Inference: Genre adaptation could be probed by fine-tuning on a small jazz corpus; a sharp FAD drop on Pijama would show the architecture is not the limiting factor, while a stubborn FAD would point to the tokenisation or conditioning design.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MIDI-VALLE, a neural codec language model adapted from the VALLE text-to-speech framework for expressive piano performance MIDI-to-audio synthesis. MIDI is tokenised with the Octuple method and audio with a fine-tuned Piano-Encodec; an autoregressive decoder predicts the first codebook and a non-autoregressive decoder predicts the remaining codebooks, conditioned on a reference audio prompt and its corresponding MIDI. The model is trained on the ATEPP dataset and evaluated against the M2A baseline on ATEPP, Maestro, and Pijama using FAD, spectrogram distortion, chroma distortion, and listening tests. The paper claims over 75% lower FAD on ATEPP and Maestro and a 202-to-58 preference in the listening test.

Significance. If the central claims hold, the paper makes a useful contribution: it demonstrates that codec language modelling with explicit MIDI conditioning can serve expressive piano performance synthesis, provides zero-shot acoustic-prompt control, and shows good generalisation from transcribed to recorded MIDI. The release of code and demos, the use of a large diverse training set, and the candid discussion of failures on jazz and of prompt-alignment issues are strengths. However, the headline objective result is weakened because the FAD metric is computed with embeddings from Piano-Encodec, the same codec used as the training target, and because the spectrogram and chroma distortions do not favour MIDI-VALLE. The significance of the work therefore depends on whether the reported FAD advantage survives an independent-embedding evaluation.

major comments (3)
  1. [§5.1, §3.2, §6.1] The headline FAD result is computed with embeddings extracted from Piano-Encodec, the same codec whose discrete tokens MIDI-VALLE is trained to predict (Section 3.2, max p(C | x, C~)). This creates an in-distribution advantage for MIDI-VALLE over the spectrogram/vocoder baseline M2A: MIDI-VALLE outputs are decoded from predicted Piano-Encodec tokens, so they lie near the codec's reconstruction manifold, while M2A outputs do not. Section 6.1 itself states that MIDI-VALLE's FAD to Piano-Encodec reconstructions is lower than to ground truth, 'suggesting that MIDI-VALLE aligns more with the quantised representations used in training than with the original audio.' The reported FAD therefore partly measures codec-matching rather than perceptual quality. I ask the authors to recompute FAD using independent audio embeddings (for example, VGGish, CLAP, or the original Encodec) and to report both the in-codec and out-of-codec values. Without such a check, the claim that MIDI-VALLE 'significantly outperforms' M2A on objective quality is not established.
  2. [Abstract, §5.1, Table 4] The abstract claims 'over 75% lower Fréchet Audio Distance on the ATEPP and Maestro datasets,' but Table 4 shows this is not accurate for the ground-truth reference rows. Against ground truth, the FAD reduction is approximately 70% on ATEPP (11.014 to 3.329) and approximately 67% on Maestro (34.479 to 11.281); only the ATEPP reconstruction-reference row exceeds 75% (11.463 to 2.659, a 76.8% reduction), while the Maestro reconstruction row is 72.8%. The claim should be corrected to match the reported numbers. Additionally, Table 4 reports confidence intervals for spectrogram and chroma distortions but not for FAD; given that FAD is the basis for the headline claim, confidence intervals or another uncertainty estimate should be provided.
  3. [§6.1, Table 4] The objective metrics other than FAD do not support the claim of superior synthesis quality. In Table 4, spectrogram distortion is essentially unchanged (ATEPP ground-truth: 0.219 vs. 0.218; Maestro: 0.231 vs. 0.230; Pijama: 0.322 vs. 0.312, where MIDI-VALLE is worse), and chroma distortion is consistently worse for MIDI-VALLE on all three datasets (e.g., Maestro ground-truth: 0.428 vs. 0.387; Pijama: 0.558 vs. 0.471). The paper explains the Maestro chroma result by noting that M2A was trained on Maestro, but this leaves the ATEPP and Pijama chroma differences unexplained. The conclusion that MIDI-VALLE 'provides a more accurate reconstruction across the full frequency spectrum' is not supported by the quantitative spectrogram/chroma tables and relies on the biased FAD. I recommend tempering the objective-claims section or providing supporting evidence from a metric that is not computed with the training codec.
minor comments (5)
  1. [§5.2, §6.2] The listening test reports raw vote counts (202 vs. 58) and states that MIDI-VALLE 'receives significantly more votes,' but no significance test or confidence interval is reported. Given that 20 participants each evaluated half of the stimuli and votes are not independent per participant, a per-comparison or per-participant analysis would strengthen the claim.
  2. [§3.1.1] The text says 'Instead of the original Encodec model [10]' but then cites MusicGen's Encodec fine-tuning approach as [21]; the following description of fine-tuning also references [10,25]. The citation usage here is confusing and should be clarified so that the original Encodec paper, the MusicGen adaptation, and the unofficial VALLE implementation are each cited at the appropriate point.
  3. [Throughout] There are several typographical issues: 'groundtruth' should be 'ground truth' or 'ground-truth' consistently; Table 4 footnote 1 contains 'groudtruth'; 'Frechet' appears without the accent in the abstract and Section 5.1; and 'V ALLE' is inconsistently spaced in the running text. These should be cleaned up before publication.
  4. [§6.1, Table 3] Table 3 reports FAD for Piano-Encodec reconstructions on ATEPP, Maestro, and Pijama, but no confidence intervals are given for FAD in this table either. Since the table is used to argue that Piano-Encodec is a reliable embedding extractor, reporting uncertainty would make that argument more convincing.
  5. [§6.2] The discussion of prompt-selection effects and truncation misalignment is valuable, but the claim that 'MIDI-VALLE can generate coherent and natural outputs from any prompt that is stylistically consistent and acoustically clear' is only anecdotal; a small systematic study of prompt choice would make this claim reproducible.

Circularity Check

1 steps flagged · score 6.0 of 10

Primary objective metric (FAD) is computed in the embedding space of Piano-Encodec, the very codec whose tokens MIDI-VALLE is trained to predict; the headline 75% improvement is therefore partly an in-distribution advantage over the spectrogram-based baseline, though the listening test provides independent evidence.

  1. fitted input called prediction [Section 5.1 (Objective Metrics) and Section 3.2 (Model Design); see also Section 6.1]
    "FAD measures the perceptual quality and realism of generated audio by comparing it to reference performances using embeddings extracted from Piano-Encodec."

    Section 3.2 specifies that MIDI-VALLE is trained to maximise p(C | x, C~), i.e., to predict Piano-Encodec token matrices, and synthesised audio is obtained by decodec(hat{C}). FAD is then computed from embeddings of the same Piano-Encodec encoder, so the metric effectively measures how well the predicted token distribution matches the codec's latent distribution of the reference—close to the training objective itself. The spectrogram-based baseline M2A is evaluated in this codec space without having been trained on it, creating an in-distribution advantage for MIDI-VALLE.

full rationale

The paper's central 'over 75% lower FAD' claim is not fully self-contained: the FAD metric is defined on embeddings from Piano-Encodec, the same codec whose token prediction is the training objective of MIDI-VALLE (Section 3.2). This makes the objective metric partly a measure of codec-matching rather than purely perceptual quality, and it disadvantages M2A, which does not operate in that token space. The paper itself acknowledges in Section 6.1 that MIDI-VALLE aligns more with quantised representations than with ground-truth audio. No other circular steps were found: the listening test (Section 5.2) is a genuine external evaluation, and the citations to the authors' prior work (ATEPP dataset, M2A baseline, Octuple tokenisation) are not load-bearing derivations. The abstract's 'over 75%' also overstates the ground-truth-reference FAD reductions in Table 4 (~70% on ATEPP and ~67% on Maestro), but that is an arithmetic inconsistency, not circularity.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central performance claim rests on several hand-selected architectural constants (codebook count, prompt length, segment length) and on domain assumptions about the sufficiency of codec tokens and the validity of the codec-based evaluation metric. No new physical or conceptual entities are introduced.

free parameters (7)
  • audio prompt length = 3 seconds
    Chosen as the conditioning context for zero-shot timbre and environment transfer; not systematically swept. Section 3.2.
  • number of RVQ codebooks = 4
    Matches MusicGen/Encodec setup; determines audio token granularity. Section 3.1.1.
  • codebook size = 2048
    Standard setting from Encodec; not tuned for this task. Section 4.2.
  • segment duration = 15-20 seconds
    Training and evaluation clips chosen for context; affects both context length and alignment at boundaries. Section 4.1.
  • transformer hidden size = 1024
    Model capacity taken from the VALLE implementation (12 layers, 16 heads). Section 3.2.
  • learning rate base = 0.05
    ScaledAdam base LR with Eden scheduler, adopted from prior TTS implementation. Section 4.2.
  • ATEPP curation = 8825 performances
    Excluding low-quality performances is a data filtering choice that affects training distribution; the criteria for 'low-quality' are not specified. Section 4.1.
assumptions (5)
  • domain assumption Four-level RVQ codec tokens at 50 Hz preserve sufficient acoustic and timbral information for expressive piano synthesis.
    Assumed throughout; audio tokens are the training target and output representation. Section 3.1.1.
  • domain assumption Octuple MIDI tokenization with extended IOI captures note timing, velocity, and articulation with enough fidelity for audio generation.
    The paper argues this is superior to piano rolls, but does not ablate tokenization choices. Section 3.1.2.
  • domain assumption ATEPP is a sufficiently diverse dataset that training on it generalizes to other classical datasets (Maestro) and partially to jazz (Pijama).
    Underlies the generalization claim; Pijama results show limits. Section 4.1, Section 6.1.
  • domain assumption Fréchet Audio Distance using Piano-Encodec embeddings is a valid measure of perceptual and acoustic similarity.
    This is the key metric for the headline result and is not validated against an independent embedding model. Section 5.1.
  • domain assumption Three-second audio prompt and its corresponding MIDI segment can be aligned accurately enough to transfer timbre and environment without introducing artifacts.
    The paper notes misalignments cause unexpected notes or omissions (Section 6.2), making this assumption load-bearing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MIDI-VALLE: Improving Expressive Piano Performance Synthesis Through Neural Codec Language Modelling." pith.science (2026). https://pith.science/paper/ENWREVHZ

@misc{pith2026250708530,
  author       = {Pith},
  title        = {Pith review of: MIDI-VALLE: Improving Expressive Piano Performance Synthesis Through Neural Codec Language Modelling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ENWREVHZ}},
  note         = {Machine review of arXiv:2507.08530}
}
read the original abstract

Generating expressive audio performances from music scores requires models to capture both instrument acoustics and human interpretation. Traditional music performance synthesis pipelines follow a two-stage approach, first generating expressive performance MIDI from a score, then synthesising the MIDI into audio. However, the synthesis models often struggle to generalise across diverse MIDI sources, musical styles, and recording environments. To address these challenges, we propose MIDI-VALLE, a neural codec language model adapted from the VALLE framework, which was originally designed for zero-shot personalised text-to-speech (TTS) synthesis. For performance MIDI-to-audio synthesis, we improve the architecture to condition on a reference audio performance and its corresponding MIDI. Unlike previous TTS-based systems that rely on piano rolls, MIDI-VALLE encodes both MIDI and audio as discrete tokens, facilitating a more consistent and robust modelling of piano performances. Furthermore, the model's generalisation ability is enhanced by training on an extensive and diverse piano performance dataset. Evaluation results show that MIDI-VALLE significantly outperforms a state-of-the-art baseline, achieving over 75% lower Frechet Audio Distance on the ATEPP and Maestro datasets. In the listening test, MIDI-VALLE received 202 votes compared to 58 for the baseline, demonstrating improved synthesis quality and generalisation across diverse performance MIDI inputs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Drum Synthesis from Expressive Drum Grids via Neural Audio Codecs

    cs.SD 2026-05 unverdicted novelty 5.0 of 10

    A Transformer predicts tokens from neural audio codecs (EnCodec, DAC, X-Codec) to convert expressive drum grids into audio, trained and evaluated on the E-GMD dataset using objective metrics.

Reference graph

Works this paper leans on

40 extracted references · 32 canonical work pages · cited by 1 Pith paper

  1. [1]

    MIDI-V ALLE: Im- proving Expressive Piano Performance Synthesis Through Neural Codec Language Modelling

    INTRODUCTION Music performance synthesis (MPS) refers to the process of generating expressive audio performances from mu- sic scores. This task requires models to capture acoustic characteristics of musical instruments and infuse human- © J. Tang, X. Wang, Z. Zhang, J. Yamagishi, G. Wiggins and G. Fazekas. Licensed under a Creative Commons Attribution 4.0...

    arXiv 2025

  2. [2]

    These TTS-inspired models typically process piano performance MIDIs as pi- ano rolls for audio synthesis

    RELA TED WORKS 2.1 Expressive Performance Synthesis In the EPS domain, several studies have explored various approaches for MPS, including DDSP-based modelling [1, 11] and TTS-inspired models [2–5]. These TTS-inspired models typically process piano performance MIDIs as pi- ano rolls for audio synthesis. Hawthorne et al. [6] em- ployed WaveNet to map piano...

    work page

  3. [3]

    MIDI-V ALLE FOR PIANO SYNTHESIS Our MIDI-V ALLE model focuses on performance MIDI- to-audio synthesis, drawing parallels to text-to-speech syn- thesis by V ALLE. The following sections discuss the to- kenisation strategies and key architectural differences be- tween MIDI-V ALLE and V ALLE, highlighting the similar- ities and distinctions between speech an...

    work page

  4. [4]

    A total of 8,825 perfor- mance recordings were selected and split into training, val- idation, and test sets in an 8:1:1 ratio

    EXPERIMENTS 4.1 Datasets We used the ATEPP [8] dataset, excluding low-quality per- formances and their transcriptions. A total of 8,825 perfor- mance recordings were selected and split into training, val- idation, and test sets in an 8:1:1 ratio. The repertoire has around 700 hours of audio recordings from 1,099 albums, featuring 1,523 compositions by 25 ...

    work page

  5. [5]

    FAD measures the percep- tual quality and realism of generated audio by comparing it to reference performances using embeddings extracted from Piano-Encodec

    EV ALUA TION 5.1 Objective Metrics To evaluate the performance of the proposed MIDI- V ALLE system, we employ three objective metrics: Fréchet Audio Distance (FAD) [29, 30], spectrogram dis- tortion, and chroma distortion. FAD measures the percep- tual quality and realism of generated audio by comparing it to reference performances using embeddings extrac...

    work page

  6. [6]

    In addition, Piano-Encodec achieves high-fidelity reconstruction of human performances, with much lower FAD, spectrogram, and chroma distortions than generative models

    RESULTS & DISCUSSION 6.1 Objective Evaluation As shown in Table 3, fine-tuning with the ATEPP dataset significantly enhanced Piano-Encodec compared to the original Encodec [21], reducing spectrogram distortion from 0.304 to 0.123 and chroma distortion from 0.478 to 0.140. In addition, Piano-Encodec achieves high-fidelity reconstruction of human performanc...

    work page

  7. [7]

    CONCLUSION We present MIDI-V ALLE, a novel EPS model adapted from the V ALLE framework, for performance MIDI-to- audio synthesis. Our results demonstrate that MIDI- V ALLE outperforms the existing EPS baseline in both adaptability and synthesis quality, producing more natural and expressive audio across a wide range of performance inputs and recording con...

    work page

  8. [8]

    Onderzoeksprogramma Artificiële Intelli- gentie (AI) Vlaanderen

    ACKNOWLEDGEMENT This work was supported by the UKRI Centre for Doc- toral Training in Artificial Intelligence and Music [grant number EP/S022694/1] and the National Institute of In- formatics, Japan. J. Tang is a research student jointly funded by the China Scholarship Council [grant number 202008440382] and Queen Mary University of London. G. Wiggins rec...

    work page

Show all 40 references
  1. [9]

    The datasets used in this study — ATEPP [8], Mae- stro [6], and Pijama [31] — contain audio recordings and corresponding MIDI annotations of piano performances

    ETHICS STA TEMENT No personal or sensitive user data is involved in this re- search. The datasets used in this study — ATEPP [8], Mae- stro [6], and Pijama [31] — contain audio recordings and corresponding MIDI annotations of piano performances. The MIDI files from all three d...

  2. [10]

    MIDI-DDSP: Detailed control of musical per- formance via hierarchical modeling,

    Y . Wu, E. Manilow, Y . Deng, R. Swavely, K. Kastner, T. Cooijmans, A. Courville, C.-Z. A. Huang, and J. En- gel, “MIDI-DDSP: Detailed control of musical per- formance via hierarchical modeling,” in International Conference on Learning Representations, 2022

    2022

  3. [11]

    Deep performer: Score-to-audio music performance synthesis,

    H.-W. Dong, C. Zhou, T. Berg-Kirkpatrick, and J. McAuley, “Deep performer: Score-to-audio music performance synthesis,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 951– 955

    2022

  4. [12]

    Towards an integrated approach for expressive piano performance synthesis from music scores,

    J. Tang, E. Cooper, X. Wang, J. Yamagishi, and G. Fazekas, “Towards an integrated approach for expressive piano performance synthesis from music scores,” in ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), 2025, pp. 1–5

    2025

  5. [13]

    Text-to- speech synthesis techniques for midi-to-audio synthe- sis,

    E. Cooper, X. Wang, and J. Yamagishi, “Text-to- speech synthesis techniques for midi-to-audio synthe- sis,” Proc. 11th ISCA Speech Synthesis Workshop (SSW 11), 2021

    2021

  6. [14]

    Can knowledge of end-to-end text-to- speech models improve neural midi-to-audio synthesis systems?

    X. Shi, E. Cooper, X. Wang, J. Yamagishi, and S. Narayanan, “Can knowledge of end-to-end text-to- speech models improve neural midi-to-audio synthesis systems?” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  7. [15]

    Enabling factorized piano music modeling and generation with the MAE- STRO dataset,

    C. Hawthorne, A. Stasyuk, A. Roberts, I. Si- mon, C.-Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the MAE- STRO dataset,” in International Conference on Learning Representations, 2019. [Online]. Availab...

    2019

  8. [16]

    Neural codec language models are zero-shot text to speech synthesizers,

    S. Chen, C. Wang, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei, “Neural codec language models are zero-shot text to speech synthesizers,”IEEE Transactions on Au- dio, Speech and Language Processing , vol. 33, pp. 705–718, 2025

    2025

  9. [17]

    ATEPP: A Dataset of Auto- matically Transcribed Expressive Piano Performance,

    H. Zhang, J. Tang, S. R. Rafee, S. Dixon, G. A. Wiggins, and G. Fazekas, “ATEPP: A Dataset of Auto- matically Transcribed Expressive Piano Performance,” in International Society for Music Information Re- trieval Conference, Dec. 2022, pp. 446–453. [Online]. Available: https://...

    2022 doi

  10. [18]

    MusicBERT: Symbolic music understanding with large-scale pre-training,

    M. Zeng, X. Tan, R. Wang, Z. Ju, T. Qin, and T.-Y . Liu, “MusicBERT: Symbolic music understanding with large-scale pre-training,” inFindings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , Online, Aug. 2021, pp. 791–800. [Online]. Available: https://aclanth...

    2021

  11. [19]

    High fidelity neural audio compression,

    A. Défossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” Transactions on Machine Learning Research, 2023, featured Certifica- tion, Reproducibility Certification. [Online]. Available: https://openreview.net/forum?id=ivCd8z8zR2

    2023

  12. [20]

    DDSP-Piano: a Neural Sound Synthesizer Informed by Instrument Knowledge,

    L. Renault, R. Mignot, and A. Roebel, “DDSP-Piano: a Neural Sound Synthesizer Informed by Instrument Knowledge,” AES - Journal of the Audio Engineering Society Audio-Accoustics-Application, vol. 71, no. 9, pp. 552–565, Sep. 2023. [Online]. Available: https: //hal.science/hal-04073770

    2023

  13. [21]

    Neural speech synthesis with transformer network,

    N. Li, S. Liu, Y . Liu, S. Zhao, and M. Liu, “Neural speech synthesis with transformer network,” in Pro- ceedings of the AAAI conference on artificial intelli- gence, vol. 33, no. 01, 2019, pp. 6706–6713

    2019

  14. [22]

    Fastspeech: fast, robust and control- lable text to speech,

    Y . Ren, Y . Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “Fastspeech: fast, robust and control- lable text to speech,” Proceedings of the 33rd Interna- tional Conference on Neural Information Processing Systems, 2019

    2019

  15. [23]

    Hifi-gan: Generative ad- versarial networks for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative ad- versarial networks for efficient and high fidelity speech synthesis,” Advances in neural information processing systems, vol. 33, pp. 17 022–17 033, 2020

    2020

  16. [24]

    Reconstructing human expressiveness in piano performances with a transformer network,

    J. Tang, G. Wiggins, and G. Fazekas, “Reconstructing human expressiveness in piano performances with a transformer network,” The 16th International Sympo- sium on Computer Music Multidisciplinary Research , 2023

    2023

  17. [25]

    Scoreperformer: Expressive piano performance rendering with fine-grained con- trol

    I. Borovik and V . Viro, “Scoreperformer: Expressive piano performance rendering with fine-grained con- trol.” in Proceedings of the 23rd International Society for Music Information Retrieval Conference, 2023, pp. 588–596

    2023

  18. [26]

    Expressive Piano Performance Rendering from Unpaired Data,

    L. Renault, R. Mignot, and A. Roebel, “Expressive Piano Performance Rendering from Unpaired Data,” in International Conference on Digital Audio Effects (DAFx23), Copenhague, Denmark, Sep. 2023, pp. 355–358. [Online]. Available: https://hal.science/ hal-04221612

    2023

  19. [27]

    Vir- tuosonet: A hierarchical rnn-based system for model- ing expressive piano performance,

    D. Jeong, T. Kwon, Y . Kim, K. Lee, and J. Nam, “Vir- tuosonet: A hierarchical rnn-based system for model- ing expressive piano performance,” in Proceedings of the 20th International Society for Music Information Retrieval Conference, 2019

    2019

  20. [28]

    Dexter: Learning and controlling performance expression with diffusion models,

    H. Zhang, S. Chowdhury, C. E. Cancino-Chacón, J. Liang, S. Dixon, and G. Widmer, “Dexter: Learning and controlling performance expression with diffusion models,” Applied Sciences , no. 15, 2024. [Online]. Available: https://www.mdpi.com/2076-3417/14/15/ 6543

    2024

  21. [29]

    Audiolm: a language modeling approach to audio generation,

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi et al. , “Audiolm: a language modeling approach to audio generation,” IEEE/ACM transactions on audio, speech, and lan- guage processing, vol. 31, pp. ...

    2023

  22. [30]

    Simple and controllable music generation,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. Défos- sez, “Simple and controllable music generation,” in Thirty-seventh Conference on Neural Informa- tion Processing Systems , 2023. [Online]. Available: https://openreview.net/forum?id=jtiQ26sCJi

    2023

  23. [31]

    Musiclm: Generating music from text,

    A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al. , “Musiclm: Generating music from text,” arXiv preprint arXiv:2301.11325, 2023

    2023 arXiv

  24. [32]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Trans. Audio, Speech and Lang. Proc. , vol. 30, p. 495–507, Nov. 2021. [Online]. Available: https://doi.org/10.1109/TASLP. 2021.3129994

    2021

  25. [33]

    Vector quantization,

    R. Gray, “Vector quantization,” IEEE ASSP Magazine, vol. 1, no. 2, pp. 4–29, 1984

    1984

  26. [34]

    Vall-e: A neural codec language model,

    F. Li, “Vall-e: A neural codec language model,” 2023. [Online]. Available: http://github.com/lifeiteng/vall-e

    2023

  27. [35]

    Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs,

    W.-Y . Hsiao, J.-Y . Liu, Y .-C. Yeh, and Y .-H. Yang, “Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs,” in Proceedings of the AAAI Conference on Artificial In- telligence, vol. 35, no. 1, 2021, pp. 178–186

    2021

  28. [36]

    Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions,

    Y .-S. Huang and Y .-H. Yang, “Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions,” in Proceedings of the 28th ACM International Conference on Multimedia, ser. MM ’20. New York, NY , USA: Association for Computing Machinery, 2020, p....

    2020

  29. [37]

    Zipformer: A faster and better encoder for automatic speech recognition,

    Z. Yao, L. Guo, X. Yang, W. Kang, F. Kuang, Y . Yang, Z. Jin, L. Lin, and D. Povey, “Zipformer: A faster and better encoder for automatic speech recognition,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https: //openreview.ne...

    2024

  30. [38]

    Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms,

    K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, “Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms,” in Interspeech, 2019. [Online]. Available: https://api. semanticscholar.org/CorpusID:202725406

    2019

  31. [39]

    Adapting frechet audio distance for generative music evaluation,

    A. Gui, H. Gamper, S. Braun, and D. Emmanouilidou, “Adapting frechet audio distance for generative music evaluation,” in Proc. IEEE ICASSP 2024 , 2024. [Online]. Available: https://arxiv.org/abs/2311.01616

    2024 arXiv

  32. [40]

    Pijama: Pi- ano jazz with automatic midi annotations,

    D. Edwards, S. Dixon, and E. Benetos, “Pijama: Pi- ano jazz with automatic midi annotations,” Transac- tions of the International Society for Music Informa- tion Retrieval, 2023

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.