REVIEW 2 major objections 5 minor 32 references
MI-MIDI: Mechanistic Interpretability of Text-to-MIDI Generation Models via Probing, Lenses and Steering
T0 review · 2 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Two text-to-MIDI models encode musical structure readably and steerably, and their architecture determines whether predictions form gradually or in a sharp late switch.
desk verdict First systematic interpretability pass on text-to-MIDI models; careful controls, but the headline steering numbers need a multiple-comparison correction. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the residual-stream activation $h^{(i)}_{\ell,t}$, the state from which the model predicts token $t$ at layer $\ell$, recorded in predictive alignment. On this object the paper stacks four instruments: linear probes (logistic regressions deliberately restricted to linear form, with shuffled-label control tasks); the logit lens and its trained variant, the tuned lens, which pass intermediate states through the final normalization and vocabulary projection to reveal what the model would predict if it stopped early; activation patching that replaces prompt-conditioned activations at a chosen layer to measure causal transfer; and difference-in-means steering vectors $d_\ell = (\mu^B_\ell - \mu^A_\ell)/\lVert \mu^B_\ell - \mu^A_\ell \rVert_2$ added at one layer or at all layers. The paper's bidirectional protocol decomposes the steering response into an antisymmetric directional part and a symmetric drift part, with specificity $\mathrm{spec} = |s_{\mathrm{dir}}|/(|s_{\mathrm{dir}}|+|s_{\mathrm{non}}|)$.
What would settle it
Apply the same single-layer and all-layer steering sweep to another cross-attention encoder-decoder text-to-MIDI model and another decoder-only model with a shared prompt-music residual stream; if the cross-attention model fails to tolerate all-layer steering, or the decoder-only model tolerates it without disruptive accumulation, the architecture-attribution claim is refuted.
Extended reading notes
Core claim
The central claim is that mechanistic interpretability transfers to symbolic music generation and reveals architecture-dependent routes for prediction formation. In text2midi, predictions are refined gradually across decoder depth, while in MIDI-LLM early layers operate largely in the inherited language-model basis and the readout rotates sharply into the MIDI vocabulary around layers 13 to 15, a transition also detected by activation patching. Linear probes recover musically meaningful structure in both models, including instrument family, pitch class, harmony, and texture, with controlled SynTheory-style probing showing high accuracy for intervals and chord progressions. Activation steering produces bidirectional changes in register and polyphony in both models and in tempo/energy in MIDI-LLM, and the paper's two-orientation protocol separates true directional response from symmetric drift, showing that all-layer steering is stable in text2midi but accumulates disruptively in MIDI-LLM.
Load-bearing premise
The load-bearing premise is that the observed difference in steering stability between the two models comes from their conditioning architecture (cross-attention memory versus a shared prompt-music residual stream), but only one model of each architecture is studied, so tokenization, model size, or training data could also explain it.
Editorial extensions
If this is right
- The probing protocol can be applied to any symbolic music generator to audit which musical concepts are linearly accessible at which layers.
- The architecture-aware intervention rule predicts that cross-attention-conditioned generators can be steered safely at all layers, while decoder-only models with a shared prompt-music residual stream should be steered at a single targeted layer.
- The tuned lens and vocabulary-mass analysis give a way to detect basis changes in mixed text-music tokenizers, not just in pure language models.
- The two-orientation steering protocol provides a measurable cleanliness criterion for any activation-steering direction: high antisymmetric share plus low symmetric drift.
- The patch-transfer experiment offers a causal way to locate where prompt conditioning stops influencing generated musical attributes.
Reading between the lines
- Editorial inference: the two-orientation protocol could be applied to text-to-audio models, where exact note-level metrics are unavailable, by comparing outcome distributions across orientations.
- Editorial inference: because MIDI-LLM's predictions form late, intervening only in its final few layers may be a cheaper way to control music attributes than full-layer steering or retraining.
- Editorial inference: the controlled SynTheory results show strong interval and chord-progression decodability, so one could test whether these representations are reused in full scores by patching away a chord-progression direction and measuring harmony changes.
- Editorial inference: the authors leave sparse autoencoders to future work; a natural next step is to check whether the linearly decoded concepts correspond to single sparse features or to directions distributed across many features.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript studies two public text-to-MIDI models, text2midi (encoder-decoder, REMI+ tokenization) and MIDI-LLM (decoder-only Llama 3.2 1B with extended vocabulary), using linear probing, logit and tuned lenses, activation patching, and difference-in-means activation steering. The authors report that pitch, instrumentation, harmony, and texture are linearly decodable in both models; that text2midi refines predictions gradually across depth while MIDI-LLM exhibits a sharp late transition into the musical vocabulary; that patching locates a matching late attenuation of prompt-driven instrument transfer in MIDI-LLM; and that steering produces bidirectional changes in register and polyphony in both models and tempo/energy in MIDI-LLM. They propose a bidirectional evaluation protocol that decomposes steering responses into directional and symmetric components, and they derive an architecture-dependent intervention recipe (all-layer steering for text2midi, single-layer steering for MIDI-LLM).
Significance. If the causal steering results survive scrutiny, this is a substantial contribution to the interpretability of symbolic music generation. The experimental design is generally careful: grouped cross-validation prevents sequence leakage in probing; label-shuffled controls confirm that the probes read information rather than probe capacity; the tuned lens is trained on disjoint sequences and evaluated against the classic lens; patching uses paired seeds and bootstrap confidence intervals; and steering is guarded by note-count stability and a bidirectional decomposition. The paper also explicitly acknowledges important limitations, including confounded contrastive prompts, proxy metrics, and the single-model-per-architecture comparison. The principal weakness is the post-hoc selection of steering layers without multiple-comparison correction, which directly affects the headline causal claims and the architecture-dependent recipe.
major comments (2)
- [§7.2–7.3, Tables 6–7] The headline steering results are selected post hoc over source layers without a multiple-comparison correction. For each model, concept, and injection strategy, Table 7 reports the single cell that clears the exploratory |s_dir|>2 SE rule, and Table 6 reports the 'best single-layer configuration per concept' at exactly that layer. With nine source layers in the sweep, the family-wise error rate of the 2 SE rule is on the order of 0.34–0.37 even under independence, so a substantial fraction of the reported effects could arise from extreme noise draws. The bidirectional decomposition and note-count guard reduce symmetric drift but do not remove selection-driven inflation of |s_dir|, because selection is performed on |s_dir| (or on specificity, which depends on it). The paper does not show the full per-layer distribution, so the reader cannot judge how many cells survive a correction. In addition, the Δ columns of Table 6 are recorded at the α giving the largest shift within the stable range, adding a second level of selection over the α grid. Because Section 7.3 and Section 8 build the central causal claims and the intervention recipe on these selected cells, the paper should either apply a valid multiple-comparison correction, report the full per-layer response surface, or validate the layer selection on held-out data; otherwise the quantitative strength, and possibly the existence, of the bidirectional effects is overstated.
- [§8, abstract and conclusion] The architecture-dependent recipe is supported by only one model per architecture. The paper acknowledges in Section 8 that 'with one model per architecture we cannot isolate the conditioning pathway from every other design difference', yet the abstract and conclusion state more strongly that 'architecture shapes' prediction formation and control and that 'all-layer steering suits cross-attention-conditioned text2midi, while targeted layers provide robust control in MIDI-LLM'. Because text2midi and MIDI-LLM also differ in parameter count, tokenization, and training data, the observed differences in steering robustness cannot be attributed to the conditioning pathway without additional models or ablations. This should be reframed as a case-study hypothesis, or supported by at least one further model per architecture.
minor comments (5)
- [Tables 3–4] The 'best layer per concept' entries are selected as the peaks of per-layer accuracy curves without confidence intervals on the layer-wise differences; since Figures 2 and 3 show the full curves this is not load-bearing, but small adjacent-layer differences should not be overinterpreted.
- [§4.2] The pitch-class histogram control is reported only as summary lifts; reporting the distribution of histogram lifts, or a direct significance test against the mean-pooled activation lift, would make the 'suggestive rather than conclusive' conclusion easier to calibrate.
- [§7.2] The seed-clustered standard errors for the steering slopes are not described in enough detail; specifying the cluster-robust variance estimator, the clustering unit, and the effective degrees of freedom would improve reproducibility.
- [§4.1] Probe regularization is fixed at C=1.0 with no sensitivity analysis; the shuffled-label control mitigates probe-capacity concerns, but a short sentence on robustness to C would strengthen the probing claims.
- [§8] The acknowledged non-independence of the tempo/energy and polyphony metrics could be quantified, for example by reporting the correlation between note density and onset count on the generated scores, which would help interpret the relative sizes of the steering effects.
Circularity Check
No significant circularity: probing, lens, patching, and steering analyses are self-contained and do not reduce to their inputs.
full rationale
The paper's derivation chain is self-contained and does not exhibit circular reasoning. Probing labels are computed from the generated MIDI rather than from the prompts, and the paper explicitly checks the key probe against a pitch-class histogram control, acknowledging that most of the sequence-level signal may summarize the note distribution. The tuned lens is trained to distill the model's own final distribution, which is a standard diagnostic procedure, not a fitted input being presented as a prediction. Activation patching uses self-patch and neutral-prompt controls, and the steering directions are computed from contrastive prompt activations and then applied during generation, with the effect measured on the output score; the metric is not defined in terms of the direction itself. The bidirectional decomposition of steering responses into directional and symmetric components is an algebraic restatement, not a circular claim. The best-layer selection in Section 7.3 raises a multiple-comparisons concern that could inflate reported effect sizes, but this is a statistical robustness issue rather than a logical circularity: the selected cell's effect is still measured independently of the selection criterion. No self-citations are used to justify load-bearing premises, and no known result is renamed. Overall, the central claims are supported by independent measurements and controls.
Assumptions & free parameters
free parameters (5)
- Probe regularization C =
1.0
- Number of contrastive prompts per pole for steering directions =
25
- Tuned lens training hyperparameters =
learning rate 1e-3, 4 epochs
- Krumhansl-Schmuckler key-estimation window =
last 20 melodic pitches
- Steering alpha sweep and note-count stability threshold =
alpha in [0,2], step 0.25; threshold at half of baseline median note count
assumptions (4)
- domain assumption The unembedding (final vocabulary projection) is a useful basis for reading intermediate predictions via the logit lens and tuned lens.
- domain assumption Heuristic labels derived from decoded MIDI (e.g., Krumhansl-Schmuckler key estimation) are accurate enough to serve as ground truth for probing.
- ad hoc to paper The exploratory 2 SE rule is a valid significance criterion for steering effects.
- standard math Grouped cross-validation and bootstrap resampling provide valid uncertainty estimates.
Cite this review
Pith. "Pith review of MI-MIDI: Mechanistic Interpretability of Text-to-MIDI Generation Models via Probing, Lenses and Steering." pith.science (2026). https://pith.science/paper/D4GTU253
@misc{pith2026260806638,
author = {Pith},
title = {Pith review of: MI-MIDI: Mechanistic Interpretability of Text-to-MIDI Generation Models via Probing, Lenses and Steering},
year = {2026},
howpublished = {\url{https://pith.science/paper/D4GTU253}},
note = {Machine review of arXiv:2608.06638}
}
read the original abstract
Mechanistic interpretability of music generation has concentrated on audio models, leaving symbolic models largely unexplored. We analyze two public text-to-MIDI systems of contrasting design: the purpose-built encoder--decoder text2midi and MIDI-LLM, a Llama~3.2~1B model extended with MIDI tokens using linear probing, the logit and tuned lenses, activation patching and difference-in-means steering. Across these methods, we recover musically meaningful structure and show how architecture shapes its formation and control. Pitch, instrumentation, harmony and texture are linearly decodable in both models. text2midi refines predictions gradually across depth, whereas MIDI-LLM works largely in its inherited textual basis before a sharp late rotation into the musical vocabulary; patching identifies a matching late attenuation of prompt-driven instrument transfer. Steering produces bidirectional changes in register and polyphony in both systems, and in tempo/energy in MIDI-LLM. Our two-orientation protocol isolates directional control and shows that all-layer interventions are robust in text2midi but accumulate disruptively in MIDI-LLM. Together, the results provide a practical toolkit for tracing and controlling musical concepts in symbolic generators. Audio examples are available on a demo website.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes.arXiv preprint arXiv:1610.01644, 2016
arXiv 2016
-
[2]
Designing and interpreting probes with control tasks
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 2733–2743, 2019
work page 2019
-
[3]
Interpreting GPT: the logit lens
nostalgebraist. Interpreting GPT: the logit lens. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens, 2020. Accessed: 13.04.2026
work page 2020
-
[4]
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Ja- cob Steinhardt. Eliciting latent predictions from transformers with the tuned lens.arXiv preprint arXiv:2303.08112, 2023
arXiv 2023
-
[5]
Steering language models with activation engineering.arXiv preprint arXiv:2308.10248, 2023
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering.arXiv preprint arXiv:2308.10248, 2023
arXiv 2023
-
[6]
Extracting latent steering vectors from pretrained language models
Nishant Subramani, Nivedita Suresh, and Matthew E Peters. Extracting latent steering vectors from pretrained language models. InFindings of the Association for Computational Linguistics: ACL 2022, pages 566–581, 2022
2022
-
[7]
Sparse autoencoders find highly interpretable features in language models
Robert Huben, Hoagy Cunningham, Logan Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. InInternational Conference on Learning Representations, volume 2024, pages 7827–7845, 2024
work page 2024
-
[8]
Exploring the inner mechanisms of large generative music models
Marcel A Vélez Vásquez, Charlotte Pouw, John Ashley Burgoyne, and Willem Zuidema. Exploring the inner mechanisms of large generative music models. InProceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR), 2024
work page 2024
Show all 32 references
-
[9]
Activation patching for interpretable steering in music generation.arXiv preprint arXiv:2504.04479, 2025
Simone Facchiano, Giorgio Strano, Donato Crisostomi, Irene Tallini, Tommaso Mencattini, Fabio Galasso, and Emanuele Rodolà. Activation patching for interpretable steering in music generation.arXiv preprint arXiv:2504.04479, 2025. 12 MI-MIDI: Mechanistic Interpretability of Tex...
2025 arXiv
-
[10]
Discovering and steering interpretable concepts in large generative music models
Nikhil Singh, Manuel Cherep, and Pattie Maes. Discovering and steering interpretable concepts in large generative music models. InInternational Conference on Learning Representations, volume 2026, pages 84922–84941, 2026
2026
-
[11]
Text2midi: Generating symbolic music from captions
Keshav Bhandari, Abhinaba Roy, Kyra Wang, Geeta Puri, Simon Colton, and Dorien Herremans. Text2midi: Generating symbolic music from captions. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23478–23486, 2025
2025
-
[12]
Midi-llm: Adapting large language models for text-to-midi music generation.arXiv preprint arXiv:2511.03942, 2025
Shih-Lun Wu, Yoon Kim, and Cheng-Zhi Anna Huang. Midi-llm: Adapting large language models for text-to-midi music generation.arXiv preprint arXiv:2511.03942, 2025
2025 arXiv
-
[13]
Jukebox: A generative model for music.arXiv preprint arXiv:2005.00341, 2020
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music.arXiv preprint arXiv:2005.00341, 2020
2005 arXiv
-
[14]
Musiclm: Generating music from text.arXiv preprint arXiv:2301.11325, 2023
Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text.arXiv preprint arXiv:2301.11325, 2023
2023 arXiv
-
[15]
Simple and controllable music generation.arXiv preprint arXiv:2306.05284, 2023
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation.arXiv preprint arXiv:2306.05284, 2023
2023 arXiv
-
[16]
Midicaps: A large-scale midi dataset with text captions
Jan Melechovsky, Abhinaba Roy, and Dorien Herremans. Midicaps: A large-scale midi dataset with text captions. arXiv preprint arXiv:2406.02255, 2024
2024 arXiv
-
[17]
Towards a rigorous science of interpretable machine learning, 2017
Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning, 2017
2017
-
[18]
Explanation in artificial intelligence: Insights from the social sciences.Artificial intelligence, 267:1–38, 2019
Tim Miller. Explanation in artificial intelligence: Insights from the social sciences.Artificial intelligence, 267:1–38, 2019
2019
-
[19]
Mechanistic? InProceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 480–498, 2024
Naomi Saphra and Sarah Wiegreffe. Mechanistic? InProceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 480–498, 2024
2024
-
[20]
Probing classifiers: Promises, shortcomings, and advances.Computational Linguistics, 48(1):207–219, 2022
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances.Computational Linguistics, 48(1):207–219, 2022
2022
-
[21]
Instruct-musicgen: Unlocking text-to-music editing for music language models via instruction tuning.arXiv preprint arXiv:2405.18386, 2024
Yixiao Zhang, Yukara Ikemiya, Woosung Choi, Naoki Murata, Marco A Martínez-Ramírez, Liwei Lin, Gus Xia, Wei-Hsiang Liao, Yuki Mitsufuji, and Simon Dixon. Instruct-musicgen: Unlocking text-to-music editing for music language models via instruction tuning.arXiv preprint arXiv:24...
2024 arXiv
-
[22]
Do music generation models encode music theory? arXiv preprint arXiv:2410.00872, 2024
Megan Wei, Michael Freeman, Chris Donahue, and Chen Sun. Do music generation models encode music theory? arXiv preprint arXiv:2410.00872, 2024
2024 arXiv
-
[23]
Exploring xai for the arts: Explaining latent space in generative music.arXiv preprint arXiv:2308.05496, 2023
Nick Bryan-Kinns, Berker Banar, Corey Ford, Courtney N Reed, Yixiao Zhang, Simon Colton, and Jack Armitage. Exploring xai for the arts: Explaining latent space in generative music.arXiv preprint arXiv:2308.05496, 2023
2023 arXiv
-
[24]
Learning interpretable representation for controllable polyphonic music generation.arXiv preprint arXiv:2008.07122, 2020
Ziyu Wang, Dingsu Wang, Yixiao Zhang, and Gus Xia. Learning interpretable representation for controllable polyphonic music generation.arXiv preprint arXiv:2008.07122, 2020
2008 arXiv
-
[25]
Learning to traverse latent spaces for musical score inpainting
Ashis Pati, Alexander Lerch, and Gaëtan Hadjeres. Learning to traverse latent spaces for musical score inpainting. arXiv preprint arXiv:1907.01164, 2019
1907 arXiv
-
[26]
Latent space disentanglement via activation steering for interpretable attribute control in symbolic music generation.arXiv preprint arXiv:2605.31295, 2026
Ioannis Prokopiou, Pantelis Vikatos, Maximos Kaliakatsos-Papakostas, Theodoros Giannakopoulos, and Themos Stafylakis. Latent space disentanglement via activation steering for interpretable attribute control in symbolic music generation.arXiv preprint arXiv:2605.31295, 2026
2026 arXiv
-
[27]
Closing the loop: Pid feedback control for interpretable activation steering in symbolic music generation, 2026
Ioannis Prokopiou, Pantelis Vikatos, Maximos Kaliakatsos-Papakostas, Theodoros Giannakopoulos, and Themos Stafylakis. Closing the loop: Pid feedback control for interpretable activation steering in symbolic music generation, 2026
2026
-
[28]
Figaro: Controllable music generation using learned and expert features
Dimitri von Rütte, Luca Biggio, Yannic Kilcher, and Thomas Hofmann. Figaro: Controllable music generation using learned and expert features. InThe Eleventh International Conference on Learning Representations, 2023
2023
-
[29]
Anticipatory music transformer.arXiv preprint arXiv:2306.08620, 2023
John Thickstun, David Hall, Chris Donahue, and Percy Liang. Anticipatory music transformer.arXiv preprint arXiv:2306.08620, 2023
2023 arXiv
-
[30]
Oxford University Press, 1990
Carol L Krumhansl.Cognitive foundations of musical pitch. Oxford University Press, 1990
1990
-
[31]
music21: A toolkit for computer-aided musicology and symbolic music data
Michael Scott Cuthbert and Christopher Ariza. music21: A toolkit for computer-aided musicology and symbolic music data. InProceedings of the 11th International Society for Music Information Retrieval Conference (ISMIR), pages 637–642, 2010
2010
-
[32]
Toy models of superposition.arXiv preprint arXiv:2209.10652, 2022
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition.arXiv preprint arXiv:2209.10652, 2022. 13
2022 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.