{"id":"f356b3ad-a998-49a8-88f2-2f9c69caf823","arxiv_id":"2506.00003","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Text-only LLMs can synthesize simple musical notes via generated Python code, but their environmental sound outputs score near chance and speech generation fails entirely.","lead":"Researchers prompted text-only large language models to generate audio by writing Python code, testing musical notes, environmental sounds, and speech. The models produced recognizable simple notes but nearly failed on environmental sounds and completely failed on speech, raising questions about how much auditory knowledge text training captures.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Tier comparison is confounded: musical-note prompts include pitch, velocity, and synthesis metadata (Section 4.1, Appendix), while environmental prompts give only class names; the declining-quality trend may reflect prompt information content, not acoustic complexity or latent auditory…","rationale":"The reader's weakest_assumption concerns code memorization as an alternative to latent auditory understanding. I agree that memorization is a live threat, but I find the prompt-information confound more immediately load-bearing because it undermines the internal validity of the tiered comparison itself, independent of whether the generated code is novel or memorized. Even with perfect evaluation metrics, the observed decline from musical notes to environmental sounds to speech would not establish a complexity gradient unless the amount and kind of information in the prompts is controlled across tiers. The paper's own prompts show that the musical-note tier supplies quantitative synthesis parameters, while the environmental tier supplies only class labels and brief descriptions. This confound directly threatens the central claim that LLMs possess a latent understanding of the auditory world and that generation quality deteriorates with acoustic complexity. Because this is an experimental-design issue that can be addressed with a targeted control study, I keep the reader's CONDITIONAL verdict rather than escalating to rejection; however, if the authors cannot or will not run the crossover control, the central claim should be considered unverified rather than supported.","tokens_in":7347,"tokens_out":10504,"duration_ms":129241,"concrete_test":"Run a crossover control using the same models and evaluation pipeline: (1) prompt musical-note generation with only the instrument name, dropping pitch, velocity, amplitude, and quality descriptions; (2) prompt environmental-sound generation with detailed parametric metadata analogous to the note prompts (e.g., dominant frequency range, temporal envelope, harmonic content, duration). If stripping metadata collapses musical-note quality to the environmental level, or adding metadata lifts environmental quality to the musical-note level, the tiered trend in Section 4 is explained by prompt information content rather than acoustic complexity, and the latent-understanding conclusion in Section 5 is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states that each NSynth sample was annotated with MIDI pitch, velocity, and a qualitative description (e.g., 'waveshaping that produces a distinctive crunchy sound'), and these metadata were inserted into the prompt (Appendix 6). The model is therefore given the exact frequency, amplitude, duration, and timbral hints needed to write a standard synthesis script. Section 4.2, by contrast, uses only class names plus qualitative descriptions for FSD50K; Section 4.3 uses word/phonetic descriptions for speech. The paper's central result—quality declines from musical notes to environmental sounds to speech—is thus confounded: the tiers vary not only in acoustic complexity but also in the amount of parametric information provided in the prompt. The conclusion in Section 5 attributes the decline to 'inherent variability' and missing audio features in training data, but the design cannot separate those causes from prompt informativeness. A model could produce the observed pattern purely by following explicit synthesis parameters in the note prompts and guessing from class names in the environmental prompts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether text-only large language models can generate audio by writing Python synthesis code, and evaluates this capability across three tiers of increasing acoustic complexity: musical notes (NSynth), environmental sounds (FSD50K), and speech (Speech Commands). The authors report that musical-note generation partially succeeds, with median FAD scores for 6 of 10 instruments in a 'moderately similar' range, that environmental-sound generation achieves only 22.5% CLAP retrieval accuracy against a 20% chance baseline, and that no speech samples were successfully generated. They interpret the declining performance as evidence that LLMs have a 'latent understanding of the auditory world' whose translation into audio 'remains rudimentary' (Abstract, Section 5).","tokens_in":7500,"tokens_out":3211,"duration_ms":42225,"significance":"The paper addresses a timely and interesting question: whether text-only LLMs can access auditory knowledge through code as an intermediate representation. The three-tier design is sensible, and the paper reports failure rates and per-model compilation outcomes, which is useful descriptive information. If the central claim were established, the result would be a modest but real contribution to the literature on code-mediated modality generation. However, the current evidence does not support the paper's stronger interpretive claims. The evaluation pipeline has a confound between prompt informativeness and acoustic complexity, the FAD computation appears to be misapplied as a per-file distance, the environmental-sound result is statistically indistinguishable from chance, and the 'latent understanding' conclusion is not tested against the code-memorization alternative that the paper itself discusses in Section 2.3. Because the main empirical trend is plausibly real but the quantitative support is weak, the paper needs substantial revision before its claims can be accepted.","major_comments":[{"comment":"The central tier comparison is confounded by prompt information content. Musical-note prompts include pitch, velocity, note, amplitude, instrument, production method, and a qualitative description (Section 4.1, Appendix 6), whereas environmental-sound prompts provide only the class name and a generic template, or a brief description produced by GPT-4o (Section 4.2.1). Speech prompts include a word and a phonetic description. Thus the tiers vary not only in acoustic complexity but also in the amount of parametric guidance given to the model. The observed decline from musical notes to environmental sounds to speech could therefore reflect prompt informativeness rather than any property of the model's latent auditory knowledge. This directly affects the paper's main conclusion in Section 5, and the authors should either control for prompt information across tiers or substantially weaken the claim that performance degrades with 'complexity' or 'inherent variability.'","section":"Section 4.1 vs. 4.2, Appendix 6"},{"comment":"The FAD evaluation is not correctly specified. FAD is a distributional metric defined on sets of audio embeddings (Kilgour et al., 2019), but the paper states that 'FAD between two VGGish audio features is then calculated,' which suggests a per-file embedding distance rather than a Fréchet distance between two embedding distributions. It is also unclear how the human-evaluation thresholds at FAD scores 5, 10, 15, 20, and 25 were obtained if FAD was not computed over sets, and how a single generated file can be assigned a FAD score. Moreover, the thresholds are calibrated on the authors' own human judgments and then used to interpret the very same type of FAD scores, making the 'moderately similar' categorization partly circular. The authors need to specify the exact procedure, the number of reference and generated samples used for each FAD calculation, and whether standard implementation parameters from Kilgour et al. were followed.","section":"Section 4.1.2"},{"comment":"The environmental-sound result does not support the paper's claim of 'basic understanding.' CLAP identified 119 of 533 generated samples as belonging to the target class, or 22.5%, against a 20% chance level in a 5-way forced-choice design. No confidence interval, statistical test, or per-class breakdown is provided, and the difference from chance is 2.5 percentage points. The observation that 'even the samples that were incorrectly classified had some distinguishable characteristics' is anecdotal and not quantified. Without additional evidence, such as per-class retrieval rates, chance-level calibration, or human listening results, the environmental-sound tier should be reported as essentially failed generation rather than evidence of partial auditory knowledge.","section":"Section 4.2.2, Table 1"},{"comment":"The interpretation of the results as evidence of 'latent understanding' or 'cross-modal understanding' is not supported because the paper does not rule out the code-memorization alternative that it reviews in Section 2.3. Musical-note prompts contain standard synthesis parameters (frequency, amplitude, duration, instrument), and the model could be retrieving common code templates for sine waves, envelopes, and filters rather than drawing on internal auditory representations. The authors should test whether the generated code is novel composition or recombination of memorized snippets, for example by comparing generated code against training-era code corpora or by using obfuscated or parameter-free prompts. Without such a control, the conclusion in Section 5 overstates what the experiments establish.","section":"Section 5, Section 2.3"}],"minor_comments":[{"comment":"The sample description is inconsistent: the text refers to 10 instruments and also to 16 distinct classes, and states that most classes contained 110 examples while some had fewer than 60; the relationship between instruments, source types, and the final N=1500 should be stated precisely.","section":"Section 4.1"},{"comment":"The human evaluation is under-specified. The number of evaluators, the number of stimuli per participant, the exact instructions, and inter-rater agreement are not reported, which makes the threshold calibration ('below 10', '10 to 15', 'above 15') difficult to assess.","section":"Section 4.1.2"},{"comment":"The VGGish citation is inaccurate: VGGish was introduced by Hershey et al. (2017) in 'CNN Architectures for Large-Scale Audio Classification,' not by Nagrani et al. (2017); the latter is the VGGSound paper.","section":"References"},{"comment":"The appendix contains example prompts and a summary table, but the 'deep dive' into correctly generated audio reports descriptive statistics without stating the number of samples per class or providing qualitative examples, so the reader cannot verify the claim that misclassified samples still had distinguishable characteristics.","section":"Appendix 6"},{"comment":"The comparison with GPT-4o is presented only as '7 out of 10 instruments' with no error bars, confidence intervals, or significance testing; given the small sample sizes and non-normal FAD distributions, this comparison should be qualified or analyzed more carefully.","section":"Section 4.1.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is exploratory and the central research question is worth asking, but the quantitative evaluation is not yet reliable: the FAD methodology appears misunderstood, the tier comparison is confounded, and the environmental-sound result is near chance. I would not recommend reject because the direction is plausible and the issues could in principle be addressed with a revised experimental design. However, the authors should be encouraged to either add control experiments that equalize prompt information across tiers or explicitly reframe the paper as a descriptive study of code-mediated audio generation rather than evidence for latent auditory understanding."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate probe, but the headline claim is ahead of the evidence. The paper does something new—transfers the code-as-intermediary probing idea from vision to audio—and the basic observation is believable: text-only LLMs can produce recognizable single notes by emitting Python synthesis code, and they fall apart on speech. I'd credit that. It also tries to ground FAD scores with human listeners and includes GPT-4o as a multimodal comparison, which is a sensible instinct. The citation pattern is unremarkable; the relevant prior work is present.\n\nThe soft spots are serious. FAD is a distributional metric over sets of embeddings; the paper says it computes FAD between two audio features per file, which is not FAD. The human-calibrated thresholds (10, 15) are then used to label the same FAD values that produced them, so the categorical results like 'moderately similar' are partly circular. No error bars, no code/data release, and no reproducibility artifacts, so the numerical claims cannot be checked. The environmental results are only 22.5% accuracy against a 20% chance level, with no chance baseline or retrieval stability analysis.\n\nThe stress-test note is right, and it is the most important problem. Musical-note prompts include pitch, velocity, amplitude, and timbre descriptions, while environmental and speech prompts give only class names or a phonetic description. The three tiers therefore differ in prompt informativeness as much as acoustic complexity. A model that simply executes explicit synthesis parameters on notes and guesses generic code on class names would produce exactly this observed pattern. That undercuts the abstract's 'latent understanding' claim, especially since the paper's own Section 2.3 discusses code memorization but never tests whether the successful code is novel or retrieved.\n\nThat said, the descriptive result—declining quality from tones to environmental sounds to speech—is plausible and worth knowing. The paper is for people thinking about what text-only models can do through code, and about how to evaluate such probes. It is not a strong empirical contribution as written, but it deserves referee time rather than a desk reject because the question is real and the fixable issues are clear: use a per-file embedding distance with proper sampling, add baselines and uncertainties, release code/data, and temper the 'latent understanding' language to match the evidence.","headline":"Worth reading as a capability probe, but not as a demonstrated result: simple notes work via code, yet the complexity gradient is confounded by prompt information and the metrics are loose.","tokens_in":8102,"tokens_out":2624,"would_cite":false,"duration_ms":34009,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Text-based LLMs can generate recognizable audio by writing Python synthesis code, with quality that drops from musical notes to environmental sounds and disappears for speech.","keywords":["audio generation","large language models","code as intermediary","text-to-audio","Fréchet Audio Distance","CLAP","musical note synthesis","environmental sound generation"],"falsifier":"Run the musical-note prompting with the model forbidden from using common synthesis libraries or with a novel instrument whose acoustic signature is not documented in typical code tutorials; if generation quality collapses, the successes were routine code retrieval, not auditory understanding. Alternatively, if CLAP retrieval on environmental sounds stays near 22.5% while model scale and prompt detail increase, the paper's 'latent understanding' reading loses its support.","tokens_in":7101,"feed_emoji":"🎵","tokens_out":4544,"duration_ms":54123,"temperature":0.7,"pith_summary":"The paper asks whether LLMs trained only on text can produce audio. It uses Python code as an intermediary: the model is prompted to write synthesis code, and the code is compiled into a WAV file. On musical notes from NSynth, both GPT-4 and Llama 3.1-70B produced audible notes, with median FAD scores in the 'moderately similar' range for 6 of 10 instruments. On environmental sounds from FSD50K, only 22.5% of generated samples were retrieved as the target class by CLAP, close to the 20% chance level. No speech samples were generated successfully. The paper reads this declining performance as evidence of a latent auditory understanding that remains rudimentary.","feed_headline":"Text-only LLMs synthesize musical notes by writing code","feed_subtitle":"Musical notes land 'moderately similar'; environmental sounds sit near chance; speech produces nothing.","key_machinery":"The load-bearing mechanism is code as a textual intermediary: instead of asking the model for raw audio, the prompt asks for Python that synthesizes the sound, then executes that code to produce a WAV. Higher-level attributes such as instrument, pitch, velocity, and class description are carried in the prompt, and the model's pretraining on code supplies the synthesis routines. Evaluation is carried by two embedding-based metrics: FAD with VGGish features for musical notes, and CLAP retrieval against five candidate classes for environmental sounds. These metrics convert subjective audio quality into a numerical comparison that the paper maps onto human similarity categories for the musical-note case.","core_discovery":"The core claim is that text-only LLMs can map textual descriptions to acoustic structure through generated code, but the mapping degrades as acoustic complexity rises. For musical notes, the models reproduce quantitative properties such as pitch and amplitude well enough that human evaluators rate FAD scores below 10 as highly similar and 10–15 as moderately similar; percussion and keyboard instruments do best, wind instruments worst. For environmental sounds, CLAP-based retrieval lands at 22.5% against a 20% chance baseline, which the authors interpret as a basic grasp of acoustic properties rather than faithful replication. Human speech fails outright. The conclusion that LLMs 'possess a latent understanding of the auditory world' rests mainly on the musical-note results and on the observation that even incorrectly classified environmental samples contained distinguishable class-related features.","pith_inferences":["The musical-note successes may reflect memorized synthesis routines, such as common numpy and scipy patterns for sine waves, envelopes, and filters, rather than a general internal model of sound; the paper does not test for this.","A direct extension would be to evaluate code novelty: if models reuse verbatim snippets from common tutorials, an obfuscation or novel-instrument probe should degrade performance sharply.","The 22.5% CLAP retrieval being only 2.5 points above chance suggests the 'latent understanding' claim for environmental sounds is overgenerous; a more conservative reading is that text-only models fail to translate class descriptions into recognizable sound events.","The code-as-intermediary approach could be combined with an audio-feedback loop, running the code, comparing the resulting WAV to the target with CLAP, and reprompting the model with the error, to test whether the bottleneck is synthesis knowledge or prompt expressiveness."],"forward_implications":["If text-only LLMs can synthesize recognizable musical notes, code writing can serve as a lightweight, training-free route to basic audio assets for prototyping.","The near-chance CLAP accuracy means environmental sound generation is not yet usable for synthesis tasks, and adding class descriptions does not fix it.","Speech generation is effectively out of reach for this prompting approach with current text-only models.","Because GPT-4o, a multimodal audio-trained model, outperformed text-only models on only 7 of 10 instruments and by small margins, text-only models may already capture much of the code-mediated audio capability."],"supporting_citations":[{"why":"Supplies the NSynth musical-notes dataset used as generation target and evaluation ground truth.","marker":"Tensorflow, 2017"},{"why":"Supplies the FSD50K environmental-sound classes used as generation targets.","marker":"Fonseca et al., 2022"},{"why":"Supplies the Speech Commands target words used in the speech generation attempt.","marker":"Tensorflow, 2023"},{"why":"Defines Frechet Audio Distance, the distributional metric used to score musical-note generations.","marker":"Kilgour et al., 2019"},{"why":"Provides VGGish, the feature extractor used to compute FAD scores.","marker":"Nagrani et al., 2017"},{"why":"Demonstrates the code-as-intermediary prompting strategy for modality transfer that this paper adapts to audio.","marker":"Sharma et al., 2024"},{"why":"Supports the claim that LLM-generated code is not mere reproduction, which underlies the interpretation of the results.","marker":"Honarvar et al., 2023"}],"fun_headline_variants":["LLMs turn text into code that plays notes, but not speech","Text-only LLMs write audio code: notes work, sounds near chance","LLM audio generation via code: notes moderate, speech zero","Code-writing LLMs produce notes, fail at environmental sounds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central interpretation assumes that the Python code the model writes reflects internal auditory knowledge rather than memorized synthesis routines from its training data.","fun_headline_variants_meta":{"raw":{"variants":["LLMs turn text into code that plays notes, but not speech","Text-only LLMs write audio code: notes work, sounds near chance","LLM audio generation via code: notes moderate, speech zero","Code-writing LLMs produce notes, fail at environmental sounds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1464,"prompt_tokens":891,"completion_tokens":573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":500}},"tokens_in":507,"tokens_out":573,"duration_ms":7565,"temperature":1.0,"reasoning_tokens":500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:56:29.006530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the musical-note prompting with the model forbidden from using common synthesis libraries or with a novel instrument whose acoustic signature is not documented in typical code tutorials; if generation quality collapses, the successes were routine code retrieval, not auditory understanding. Alternatively, if CLAP retrieval on environmental sounds stays near 22.5% while model scale and prompt detail increase, the paper's 'latent understanding' reading loses its support.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NSynth musical-notes dataset used as generation target and evaluation ground truth."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Speech Commands target words used in the speech generation attempt."},{"cited_title":"Franke, and Andrew Zisserman","cited_arxiv_id":null,"evidence_quote":"Provides VGGish, the feature extractor used to compute FAD scores."},{"cited_title":"Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code","cited_arxiv_id":"2312.14856","evidence_quote":"Supports the claim that LLM-generated code is not mere reproduction, which underlies the interpretation of the results."}],"review_version":1}