{"id":"139f1213-1a64-40d3-9c65-4d1aadab4054","arxiv_id":"2507.20880","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"JAM is a 530M-parameter flow-matching song generator that adds word- and phoneme-level timing control and duration control, achieving strong lyric fidelity and musicality scores when ground-truth timings are provided.","lead":"A team built JAM, a compact 530M-parameter AI that turns lyrics into full songs while letting musicians control exactly when each word is sung. The model is small, fast, and claims better lyric accuracy, but its headline results rely on providing the exact word timings in advance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 3x WER/PER improvement rests on oracle word timings and ASR-derived ground truth; without oracle timing, JAM's advantage disappears, so the headline outperformance claim is not robust.","rationale":"The reader identified the Parakeet timestamp-as-ground-truth assumption as the weakest link, and the full text supports this. Section 4.1 explicitly relies on Parakeet for yi annotations; Section 4.3 uses HDmucs + Parakeet to build the JAME ground truth; and no human verification is reported. My stress-test sharpens this into a falsifiable protocol flaw: the WER/PER comparison in Table 3 gives JAM oracle timings while baselines receive none, and the scoring ASR is the same model family used to condition JAM. The decisive internal evidence is Table 9, where substituting predicted timings collapses JAM's WER from 0.151 to 0.370 and PER from 0.101 to 0.320—worse than the DiffRhythm baseline. This shows the claimed advantage is not intrinsic to the architecture or training; it is contingent on the evaluation supplying perfect timing information. The paper partially acknowledges this in Section 7 and the disclaimer, but the abstract and introduction present the 3x improvement unconditionally, which overstates what the experiments establish. I also considered the DPO/SongEval overlap: Section 3.7 uses SongEval as both the DPO reward and an evaluation metric, which can inflate reported aesthetic scores. This is a real secondary concern, but it is less load-bearing than the WER/PER issue because the human evaluation in Table 6 partially corroborates JAM's musical preference. The timing-oracle issue, by contrast, is precisely where the strongest headline number is produced, and the paper's own Table 9 demonstrates the fragility. The proposed test—alternative ASR, human-verified transcripts, and non-oracle timings—directly isolates whether the semantic contribution, rather than the evaluation artifact, is what drives the result.","tokens_in":21998,"tokens_out":2320,"duration_ms":29993,"concrete_test":"Re-run the Table 3 comparison under two changes: (1) score all generated songs, including JAM, with a different ASR than Parakeet (e.g., Whisper-large-v3) against human-verified lyrics and word boundaries on a random 50-song subset of JAME; and (2) generate JAM outputs without oracle timings, using the paper's own enhanced GPT-4o duration predictor from Section 6.1, while keeping baselines at their default protocols. If JAM's WER/PER advantage over DiffRhythm falls below 3x, or is no longer statistically significant, the central outperformance claim is conditional on oracle timing and ASR choice.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claim is the >3x improvement in WER/PER over prior song generators (abstract, Section 1, Table 3). This claim depends on two coupled protocol choices. First, in Table 3 JAM is evaluated with oracle word-level timestamps from JAME, while all baselines receive no timing conditioning. This is not an apples-to-apples comparison: JAM is being tested on the exact ground-truth alignment information it was trained to use, whereas the baselines must infer timing internally. Second, the same mechanism that supplies those timestamps—HTDemucs vocal separation followed by Parakeet ASR—is used both to create the training-conditioning annotations yi=(wi,tstart_i,tend_i) in Section 4.1 and to generate the 'ground-truth' lyrics and evaluation transcripts for JAME in Section 4.3. Consequently, JAM is trained to reproduce Parakeet's transcription biases, and its generated vocals are then scored against Parakeet's own output. No human verification of timestamps or transcripts is reported. Section 6 and Table 9 make the risk concrete: when JAM is driven by a non-oracle duration predictor (GPT-Dur), WER rises from 0.151 to 0.370 and PER from 0.101 to 0.320—worse than DiffRhythm (0.348/0.264) and comparable to or worse than other baselines. Thus the headline intelligibility advantage is not a property of the model alone; it is a property of the model plus oracle conditioning, evaluated with the same ASR used to create that conditioning. The more defensible contribution is the controllability mechanism itself, but the paper's strongest comparative claim—'over 3x improvement'—is not supported for the realistic use case where timings must be predicted or provided by users.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JAM, a 530M-parameter rectified-flow song generator based on a 16-layer LLaMA-style DiT backbone, conditioned on lyrics with word- and phoneme-level timings, target duration, and style. Training proceeds through pretraining, full-song supervised fine-tuning, and iterative direct preference optimization (DPO) using SongEval scores as a synthetic reward. The authors also introduce JAME, a public evaluation set of 250 post-training-cutoff songs across five genres, and report objective and subjective comparisons against DiffRhythm, ACE-Step, LeVo, and YuE. The headline claims are that JAM is the first system with word-level timing and duration control, achieves over 3x improvement in WER/PER over prior work, and outperforms baselines on several aesthetic metrics despite being the smallest model.","tokens_in":22358,"tokens_out":3498,"duration_ms":39164,"significance":"If the central claims hold, the paper makes a useful contribution to lyrics-to-song generation: it demonstrates a compact model with fine-grained temporal controllability, introduces a public benchmarking dataset designed to avoid data contamination, and provides detailed ablations and an honest discussion of limitations. The release of the model, code, and evaluation set is a concrete practical benefit. However, the headline quantitative claims rest on two protocol choices that materially weaken them: JAM is evaluated with oracle word-level timestamps while baselines are not, and the same ASR system is used to create both the training annotations and the evaluation ground truth. The paper's own Section 6 results show the oracle dependence directly. The core ideas are defensible, but the evidence as presented does not yet support the strongest claims.","major_comments":[{"comment":"The WER/PER comparison in Table 3 is not apples-to-apples. JAM is evaluated with oracle word-level timestamps from JAME, while the baselines receive only lyrics and style and must infer timing internally. Since word-level timing is the conditioning signal JAM was trained on, the reported >3x WER/PER advantage is partly an artifact of giving JAM privileged information. The paper's own Section 6.3 and Table 9 make this explicit: replacing oracle timestamps with GPT-Dur predicted timestamps raises PER from 0.101 to 0.320, worse than DiffRhythm's 0.264. To support the abstract's 'over 3x lyrical fidelity' claim, the authors should either give the same word-level timing information to the baselines or report JAM without oracle timings as the primary configuration, ideally with a learned duration predictor.","section":"Section 4.3 / Table 3 / Table 9"},{"comment":"The intelligibility metrics are circular with respect to the ASR system. Parakeet is used both to generate the training-conditioning annotations yi=(wi,tstart_i,tend_i) in Section 4.1 and to produce the reference lyrics and evaluation transcripts in Section 4.3. JAM is therefore trained to reproduce Parakeet's transcription biases, and its outputs are then scored against Parakeet's own transcriptions. No human verification of timestamps or transcripts is reported. I recommend human verification on a subset of JAME and/or evaluation with a different ASR system, reporting WER/PER for both, so that the intelligibility numbers can be interpreted as properties of the generated audio rather than of the ASR pipeline.","section":"Section 4.1 vs. Section 4.3"},{"comment":"SongEval serves both as the DPO reward signal during training and as an evaluation metric in Table 4. The steady improvement in SongEval dimensions across DPO rounds is therefore expected from optimization of the reward itself, and the 'aesthetic alignment' claim is partly circular. The authors should include an evaluation signal not used in training—for example, human ratings on the same samples, or a different held-out aesthetic model—to demonstrate that the DPO procedure improves aesthetics independently of the reward used.","section":"Section 3.7 / Table 4"}],"minor_comments":[{"comment":"The loop header reads 'for each (wi, tstart_i, tstart_i)' but should presumably be 'for each (wi, tstart_i, tend_i)'.","section":"Algorithm 1, line 2"},{"comment":"There are typographical inconsistencies: 'HDmucs' should be 'HTDemucs', and 'HDemucs' appears in the metrics paragraph.","section":"Section 4.3"},{"comment":"The abstract and introduction claim 'over 3x' WER/PER improvement, but Table 3 shows JAM's WER is about 2.3x better than DiffRhythm, 2.7x better than ACE-Step, and 3.4x better than YuE. The claim should be qualified as relative to the strongest baseline or rephrased.","section":"Section 1 and Table 3"},{"comment":"The phrase 'using the all-in-one method (Kim and Nam 2023)' is ambiguous; the method name and relevant configuration should be stated explicitly.","section":"Section 6.1"},{"comment":"The DPO-FM loss in Eq. (8) uses u(xw_t, t; θ) without explicitly showing the conditioning c, whereas Eq. (2) uses u(zt, t, c; θ). Aligning the notation would improve readability.","section":"Section 3.7"},{"comment":"The limitation paragraph says the system lacks 'phoneme-level duration control', which seems to contradict the abstract's claim of 'word- and phoneme-level timing and duration control.' The distinction between phoneme-level conditioning and independent phoneme-duration control should be clarified.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"JAM is the first open full-song generator to condition on word- and phoneme-level timing, and it ships a public benchmark and model. That is a real contribution, not a gimmick. The stress-test is right about the headline: the >3x WER/PER advantage in Table 3 rests on oracle timings for JAM while baselines get no timing conditioning. The paper's own Section 6 shows that with a naive duration predictor, JAM's PER rises to 0.32, worse than DiffRhythm. So the 'over 3x' claim is not a claim about the model alone; it is a claim about the model plus privileged conditioning. To the authors' credit, they do not hide this: the Limitations section and the whole GPT-Dur section spell out the dependency. But the abstract and introduction still sell the 3x as a property of JAM, and that framing should change.\n\nThe more defensible novelty is controllability itself. The word-level alignment via upsampled phoneme sequences with filler tokens is simple and works: beat-quantized timestamps (Direct-Quant) degrade only modestly from oracle, which suggests the mechanism is robust to reasonable input precision. The model is small (530M), released, and the JAME benchmark with post-2025 tracks is a useful public resource. The ablations on token-level duration control are clean and convincing.\n\nSoft spots: SongEval is used both as the DPO reward and as a headline evaluation metric. That is circular in the strict sense, though the paper also reports human ratings showing JAM leads on enjoyment and structure clarity, which mitigates the circularity. The bigger worry is the ASR loop: Parakeet generates both the training timestamps and the evaluation transcripts. If Parakeet has systematic biases, JAM may simply be better at mimicking Parakeet's errors. No human verification of timestamps is reported, and that is a genuine gap. Still, the direct comparison in Table 3 would be less problematic if the authors reported a JAM variant with predicted timing or no timing in that same table; as written, the table is apples-to-oranges.\n\nWho is this for? Researchers working on controllable music generation and anyone building practical song tools where a musician can supply word timings. It deserves a serious referee: the paper is substantive, honest in places, and the public resources are valuable. The referee should push for a fair baseline protocol and a human-verified subset of JAME, but the work should not be desk-rejected.","headline":"A real controllability contribution with a released model and benchmark, but the headline 3x intelligibility claim only holds with oracle timing inputs that baselines don't get.","tokens_in":22965,"tokens_out":2208,"would_cite":true,"duration_ms":24811,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"JAM is a 530M-parameter flow-matching song generator that conditions on word- and phoneme-level timestamps, giving musicians control over the timing of every word and the total duration while reducing word and phoneme error rates by more…","keywords":["lyrics-to-song generation","flow matching","word-level timing control","duration control","direct preference optimization","song evaluation benchmark","singing voice synthesis","controllable music generation"],"falsifier":"Manually transcribe a held-out set of generated songs with human-verified word boundaries and compute WER against a human reference instead of the automatic recognizer's transcript; if the gap between JAM and the baselines shrinks substantially, the reported 3x improvement partly reflects recognition bias rather than true lyrical fidelity.","tokens_in":102,"feed_emoji":"🎵","tokens_out":8034,"duration_ms":98287,"temperature":0.7,"pith_summary":"This paper claims that a compact 530M-parameter flow-matching model can give musicians precise control over when each word and phoneme is sung and how long the song lasts, while matching or beating far larger lyrics-to-song systems. The central claim is that conditioning on word- and phoneme-level timestamps, rather than just the lyric text, dramatically improves vocal intelligibility: on the paper's new JAME benchmark, word error rate falls to 0.151 and phoneme error rate to 0.101, a reduction of more than threefold over prior open models. It further claims that iterative preference optimization using automated aesthetic scores, with no human annotations, moves generated songs toward what listeners prefer on enjoyment, musicality, and structural clarity. The paper also argues that a public, genre-diverse, contamination-controlled evaluation set is necessary for fair comparison in this fast-moving area.","feed_headline":"530M model puts word-level timing inside AI song generation","feed_subtitle":"Musicians set each word's timing and total song length; lyric errors drop by over 3x.","key_machinery":"The load-bearing mechanism is the word-level phoneme alignment map: for each word $(w_i, t_i^s, t_i^e)$ the model spreads IPA phonemes evenly across the latent frames between start and end, surrounded by SONG FILLER tokens for silence, instrumental passages, and padding, and VOCAL FILLER tokens for transitions inside a word. This upsampled phoneme sequence is embedded and convolutionally downsampled to a latent-aligned lyric embedding that is injected into the first eight transformer layers along with a learned padding bias that distinguishes content from silence. The training objective is rectified flow matching, and multi-condition classifier-free guidance keeps style and lyric conditioning separable at inference.","core_discovery":"The paper's core discovery is that lyrics-to-song generation becomes far more intelligible and controllable when the generator receives temporally aligned phonemes instead of a bare lyric string. JAM converts each word with its start and end time into IPA phonemes, spreads them evenly across the corresponding latent frames, and marks non-vocal and intra-word gaps with distinct filler tokens; this alignment signal, combined with a global duration embedding and a learnable bias on padded regions, lets a 530M-parameter model generate full 44.1kHz songs up to 3 minutes 50 seconds while halving the word and phoneme error rates of the next-best system. A second discovery is that iterative direct preference optimization, using an automated song-quality scorer as reward and a ground-truth reconstruction term as regularizer, raises aesthetic scores across rounds even as it introduces a mild realism cost measured by Fréchet Audio Distance.","pith_inferences":["If word-level timing is what drives lyric fidelity, then parameter scale is not the primary lever: much smaller, timing-conditioned models may continue to out-sing larger models, and existing large generators could be adapted to accept the same conditioning.","The evaluation loop is partly circular because the same automatic recognizer that created the training timestamps scores the outputs, so the model may be learning to match that recognizer's transcription biases; a human-verified reference transcript would be needed to confirm the 3x gain is genuinely audible.","A learned duration predictor trained jointly with the generator could preserve oracle-level controllability without requiring users to supply accurate word timings, widening the system beyond expert musicians.","The filler-token alignment scheme generalizes beyond songs: any generative task that must place discrete events on a continuous latent timeline--for example, speech with controllable emphasis or instrumental performance with note onsets--could reuse the same mechanism."],"forward_implications":["A musician can specify the start and end time of every word and the overall song length, enabling precise prosody, phrasing, and pacing without any retraining.","The reported word error rate of about 0.15 and phoneme error rate of about 0.10 imply that generated vocals are several times more intelligible than those of previous open models, making them closer to usable production drafts.","JAME offers a public, five-genre benchmark built from songs released after the baseline training cutoffs, so future lyrics-to-song models can be compared on identical prompts and references without training-data leakage.","Each additional round of direct preference optimization improves aesthetic scores, and adding a ground-truth reconstruction term to the DPO loss reduces the accompanying drift in genre accuracy and Fréchet Audio Distance.","Because beat-quantized timestamps degrade quality only mildly while naive predicted timestamps degrade it sharply, practical deployment will depend on a dedicated learned duration predictor rather than a general-purpose language model."],"supporting_citations":[{"why":"Supplies the flow-matching training objective the model regresses against.","marker":"Lipman et al. 2023"},{"why":"Provides rectified-flow trajectories and the logit-normal timestep sampling used in training.","marker":"Esser et al. 2024"},{"why":"Supplies the VAE encoder for the latent space and the global duration-conditioning idea.","marker":"Evans et al. 2024"},{"why":"Supplies the VAE decoder initialization and the prior lyrics-to-song system that JAM is compared against and improves on.","marker":"Ning et al. 2025"},{"why":"Defines the DPO objective adapted for flow-matching in the aesthetic alignment stage.","marker":"Rafailov et al. 2024"},{"why":"Provides the automatic speech recognizer that generates the word-level timestamps used in both training and evaluation.","marker":"NVIDIA 2025"},{"why":"Supplies the automated song-quality scorer used to construct win/loss preference pairs without human labels.","marker":"Yao et al. 2025"},{"why":"Provides the MuQ-MuLan embedding model used to condition style from reference audio or text.","marker":"Zhu et al. 2025"}],"fun_headline_variants":["Word-level timing control for AI song generation with 530M flow model","JAM: 530M flow model slices song lyrics into time-controlled phonemes","Phoneme alignment gives musicians word-level timing in AI songs","Lyric errors halve with word-timed phonemes in 530M song generator","Fine-grained word control joins aesthetic alignment in JAM song model"],"cache_read_input_tokens":24960,"weakest_assumption_plain":"The paper assumes the automatic speech recognizer's word timestamps, extracted from separated vocal tracks, are accurate enough to serve as ground truth for both training and evaluation; if those timestamps carry systematic bias, the model can learn and reproduce that bias, inflating the measured word and phoneme error-rate gains.","fun_headline_variants_meta":{"raw":{"variants":["Word-level timing control for AI song generation with 530M flow model","JAM: 530M flow model slices song lyrics into time-controlled phonemes","Phoneme alignment gives musicians word-level timing in AI songs","Lyric errors halve with word-timed phonemes in 530M song generator","Fine-grained word control joins aesthetic alignment in JAM song model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000559,"raw_usage":{"total_tokens":2651,"prompt_tokens":930,"completion_tokens":1721,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1624}},"tokens_in":546,"tokens_out":1721,"duration_ms":12527,"temperature":1.0,"reasoning_tokens":1624,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:10:02.368590+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually transcribe a held-out set of generated songs with human-verified word boundaries and compute WER against a human reference instead of the automatic recognizer's transcript; if the gap between JAM and the baselines shrinks substantially, the reported 3x improvement partly reflects recognition bias rather than true lyrical fidelity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the automatic speech recognizer that generates the word-level timestamps used in both training and evaluation."}],"review_version":1}