{"id":"49f50425-97f5-4e97-a5e3-ba1f0cc3603a","arxiv_id":"2502.08122","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A deployed AI songwriting assistant in Hookpad supports non-sequential generation and collects a live flywheel of accepted and rejected suggestions.","lead":"Hookpad Aria is a songwriting copilot built into the Hookpad lead-sheet editor. It generates continuations, fills missing spans, and converts melody to harmony or harmony to melody, and since March 2024 it has logged 318k suggestions and 74k accepted edits from 3k users.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed conditioning on meter is not supported by the described input representation: a per-beat click track encodes tempo but no downbeat/meter signal, so the model cannot honor a user-selected meter in sparse contexts.","rationale":"The reader's weakest assumption concerns encoding fidelity and alignment with the user's selected span, key, meter, and tempo. My stress-test converges on the same area but sharpens it to a specific, checkable defect: the described input representation contains no explicit meter signal, only a uniform per-beat click track. This is more concrete than a general worry about encoding loss and is load-bearing because the paper explicitly claims generation is conditioned on meter, key, and tempo. I read the paper in good faith: the system is deployed, the usage statistics are substantial, and the interview findings are real qualitative evidence for a weaker 'assists ideation' claim. However, the central 'copilot' claim and the flywheel's future value depend on suggestions that fit the user's musical context. If meter is not encoded, fill-in-the-middle and continuation suggestions in sparse contexts may align to beats but not to measures, undermining the claimed control. The proposed concrete test is feasible even without releasing the model: checking the input construction or running controlled generations under different meters would settle whether the conditioning claim holds. I do not recommend changing the reader's CONDITIONAL verdict, because the concern is currently unresolved rather than disproven; hence UNCHANGED. If the test showed no meter sensitivity, the verdict should move toward REJECT or at least require the paper to weaken its global-attributes claim.","tokens_in":4901,"tokens_out":5984,"duration_ms":96462,"concrete_test":"Inspect the deployed Aria model input construction or run a controlled API experiment: fix identical local context (same melody, harmony, and note-onset positions relative to beats) under 3/4 vs 4/4 project settings, request many fill-in-the-middle suggestions, and measure whether phrase lengths or downbeat-aligned note onsets differ systematically. Additionally, check the model input code for any meter token, barline marker, or downbeat control note; if none exists, the global-attributes conditioning claim fails by construction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central functional claim is that Hookpad Aria generates suggestions \"conditioned on global attributes (meter, key, and tempo)\" (Section 2). The only time-representation mechanism described is a click track with \"one note per beat\" (Section 2). A uniform per-beat click encodes tempo but carries no downbeat or barline information, and no token, control channel, or embedding for meter is described anywhere in the modeling setup. For left-to-right continuation or fill-in-the-middle with sparse surrounding context, the model therefore has no input signal that distinguishes 3/4 from 4/4, or any other meter, despite the user's project specifying one. This is not merely a missing evaluation; it is a missing input signal relative to a stated capability. The aggregate acceptance statistics (74k accepted out of 318k suggestions) cannot resolve this concern, because acceptance measures user behavior in a specific UI, not alignment to meter, key, or tempo. The paper's own Figure 1 lists \"Alignment, Personalization, Evaluation\" as future work, which is an explicit admission that the alignment claim is not yet validated. The eight user interviews support the weaker claim that Aria assists ideation, but they do not substantiate the stronger claim that outputs respect the user's selected global structure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Hookpad Aria, a generative symbolic music system integrated into the Hookpad web editor for lead-sheet composition. The system is built by fine-tuning the Anticipatory Music Transformer on TheoryTab lead sheets, with a custom encoding between Hookpad's functional-harmony representation and MIDI, plus a click-track pseudo-instrument intended to represent beat timing in absolute time. The paper describes support for left-to-right continuation, fill-in-the-middle, harmony-from-melody, and melody-from-harmony generation, all conditional on the user's project context. It reports deployment statistics (318k suggestions generated for 3k users, 74k accepted) and qualitative findings from eight user interviews, and frames the system as a scalable data flywheel for studying human-AI music co-creation.","tokens_in":5146,"tokens_out":6574,"duration_ms":56041,"significance":"If the described capabilities work as claimed, the paper demonstrates a rare full integration of a state-of-the-art symbolic generative model into a production songwriting editor, with an explicit opt-in data-collection mechanism and unusually large in-the-wild interaction data. The click-track representation is a pragmatically simple way to give an absolute-time model a beat reference, and the interview findings address the under-studied dimensions of agency and ideation in music co-creation. The authors are also honest in Figure 1 that alignment, personalization, and evaluation remain future work. However, the central 'copilot' claim rests on capabilities that are not currently verified: the paper provides no objective evaluation of generation quality, encoding fidelity, or alignment to user-selected global attributes, and the described input representation appears to lack any signal for meter. The contribution is better framed as a deployment and usage report than as a validated generative system.","major_comments":[{"comment":"The claim that generation is 'conditioned ... global attributes (meter, key, and tempo) of their project' is not supported by the described input representation. The only timing mechanism described is a click track with 'one note per beat,' which conveys tempo and beat positions but no downbeat, barline, or meter information; no token, embedding, or control channel for meter is described anywhere in Section 2. In a sparse continuation or fill-in-the-middle context, the model therefore has no way to distinguish 3/4 from 4/4, so the stated meter-conditioning capability cannot be realized. Please either add an explicit meter signal (e.g., barline or beat-number tokens, or a conditioning embedding) or remove/qualify the meter part of the claim.","section":"§2, Figure 1"},{"comment":"As written, Table 1 does not define a missing span to be generated. In the Fill-in-middle row, the events M<te ∪ H<te and controls M≥te ∪ H≥te ∪ C together include every melody and harmony note (since M<te and M≥te partition M, and similarly for H), leaving no notes to fill. The same issue applies to the Harm-to-mel and Mel-to-harm rows, where the indicated events and controls partition all of M and H. The construction should explicitly exclude all notes with start times in [ts,te] from both events and controls, or introduce separate notation for context before ts and after te; otherwise the fine-tuning data do not correspond to infilling.","section":"§2, Table 1"},{"comment":"The paper presents no objective evidence that the generated suggestions are coherent or aligned with the user's selected span, key, meter, or tempo. The acceptance count (74k/318k) is an implicit, UI-specific behavior signal, and the eight interviews are self-selected and qualitative; both support user engagement but not the correctness of the generative model. Since Figure 1 itself lists Alignment and Evaluation as future work, please either add at least basic validation (e.g., reconstruction accuracy on held-out TheoryTab spans under the proposed encoding and partitioning, or a small listening/alignment study) or explicitly recast the paper's contribution as a deployment and usage report rather than a validated copilot.","section":"§2, §3"}],"minor_comments":[{"comment":"The phrase 'symbolic musical scores that describes melody and harmony' has a subject-verb agreement error; it should be 'that describe'.","section":"Abstract, §1"},{"comment":"The brand name 'Github' should be written as 'GitHub'.","section":"§1"},{"comment":"Please specify the total number of fine-tuning examples generated from the 50k lead sheets, the number of fine-tuning steps, and the details of the random time-span sampling; this information is needed to understand the balance among the four capabilities in Table 1.","section":"§2"},{"comment":"The qualitative findings would be substantially strengthened by stating how the eight participants were recruited, their musical background and prior experience with AI tools, and the analysis method used (e.g., thematic coding); the current description does not permit assessment of the representativeness or reliability of the reported themes.","section":"§3"},{"comment":"The 'Global structure' icon is not tied to any described input mechanism; consider replacing it with a block diagram showing how meter, key, and tempo actually enter the model, especially after the meter-conditioning concern is addressed.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop-style late-breaking/demo paper. The deployment statistics and the integration effort are real strengths, but the stated capability of conditioning on meter is not implemented in the described representation, and Table 1's partitioning notation appears inconsistent with infilling. These are fixable within the scope of the manuscript, which is why I recommend major revision rather than rejection. If the journal expects a full research contribution rather than a systems/demo report, the lack of any objective evaluation of generation quality may require the authors to add evaluation or substantially narrow their claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper is a late-breaking demo abstract about a real deployed system, not a full research paper. The genuinely new thing is the integration of the Anticipatory Music Transformer into Hookpad's functional-harmony lead-sheet editor, with fill-in-the-middle, harmony-from-melody, melody-from-harmony, and a click-track pseudo-instrument for beat alignment. The usage statistics—318k suggestions, 74k accepts from 3k users—are a real artifact and the flywheel framing is sensible as a research opportunity, provided it is not treated as validation of musical quality.\n\nThe paper does some things well: it is honest about future work (alignment, personalization, evaluation are listed in Figure 1 as future), the interview findings are plausible and nicely capture agency and ideation, and the system design is described enough to be approximately reproduced.\n\nThe main soft spot is the meter-conditioning claim. The text says users' selections are conditioned on global attributes (meter, key, tempo), but the only time representation described is a click track with one note per beat. That encodes tempo but no downbeat or bar length, and no token or embedding for meter is mentioned. In sparse contexts the model has no way to distinguish 3/4 from 4/4. This is not merely a missing evaluation; it is a missing input signal relative to the stated capability. If the deployed system actually includes meter tokens that the paper omitted, the authors should say so. As written, the claim is unsupported. The acceptance counts cannot fix this because acceptance measures UI behavior, not metrical alignment.\n\nThe second soft spot is the lack of any objective quality evaluation—no listening tests, no alignment metrics, no baselines. The eight self-selected interviews support the weaker claim that Aria helps with ideation, not the stronger claim that outputs respect the user's selected global structure. This is proportionately less severe for a demo abstract, but worth flagging.\n\nThe citation pattern looks fine; the references cover the relevant prior work. There is no mathematical derivation to check, so circularity concerns do not really apply.\n\nFor a serious referee: this deserves review, not desk rejection, for the late-breaking/demo venue it targets. The main fix is to clarify the meter conditioning mechanism or soften the claim. If the paper aspires to a full archival publication, it needs real evaluation. I would send it to review with the expectation of a minor revision.","headline":"Real deployed co-creation system with a genuine usage dataset, but the meter-conditioning claim is unsupported by the described input representation.","tokens_in":5675,"tokens_out":3088,"would_cite":true,"duration_ms":69328,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hookpad Aria embeds a transformer music copilot in a lead-sheet editor and reports 74k accepted suggestions from 3k users.","keywords":["generative music","human-AI co-creation","symbolic music generation","lead sheets","fill-in-the-middle","music transformer","interaction data","songwriting assistant"],"falsifier":"A reconstruction or alignment check would settle it: take a set of lead sheets, encode them into the MIDI-plus-click-track format, run the fine-tuned model to generate a fill or continuation, decode back to lead-sheet form, and measure whether the output respects the requested span boundaries, key, meter, and tempo—for instance, whether notes fall on the beat grid defined by the click track and whether the harmony respects the stated key. A high rate of off-grid, out-of-key, or span-violating outputs would indicate the encoding loses the musical information the copilot claims to honor.","tokens_in":4732,"feed_emoji":"🎵","tokens_out":6214,"duration_ms":46092,"temperature":0.7,"pith_summary":"The paper presents Hookpad Aria, a generative AI system integrated into the Hookpad lead-sheet editor to assist songwriters in composing Western pop songs. It claims that the system supports non-sequential workflows—continuing material left-to-right, filling missing spans, and generating harmony from melody or melody from harmony—while conditioning on the key, meter, and tempo of the user's project. The authors further claim that Aria is a scalable data flywheel for music co-creation, having generated 318k suggestions for 3k users since March 2024, with 74k suggestions accepted into songs. The result would matter because it shows a state-of-the-art symbolic music model working inside a real editor and collecting implicit user feedback at scale, opening the door to alignment and evaluation on interaction data.","feed_headline":"Songwriting copilot: 74k AI suggestions accepted","feed_subtitle":"Transformer model inside Hookpad continues, fills, and reharmonizes lead sheets; users kept 74k of 318k suggestions.","key_machinery":"The central mechanism is the Anticipatory Music Transformer, a transformer language model that partitions a piece into two sequences, events and controls, with controls shifted five seconds forward and interleaved with events, so the model can 'anticipate' future context and learn fill-in-the-middle. To adapt it to Hookpad, the authors design an encoding between Hookpad's proprietary functional-harmony lead-sheet representation and MIDI notes, and represent beats in absolute time by adding a click track with one note per beat so the model can learn to output notes in lockstep with the user's grid. Fine-tuning examples are built by selecting a random time span and capability, then partitioning melody, harmony, and click-track notes into events and controls accordingly; this single setup supports all four generation capabilities.","core_discovery":"Hookpad Aria is a generative AI system designed to assist musicians in writing Western pop songs, integrated into Hookpad, a web-based editor for lead sheets: symbolic scores describing melody and harmony. The system offers left-to-right continuation, fill-in-the-middle, harmony-from-melody, and melody-from-harmony, all conditioned on the project's meter, key, and tempo, and it displays endless alternative generations for a selected span that users can audit and accept. Built by fine-tuning the Anticipatory Music Transformer on 50k lead sheets, Aria has generated 318k suggestions for 3k unique users since its release in March 2024, of which 74k were accepted. Semi-structured interviews with eight users suggest the tool facilitates ideation, preserves a sense of agency through short reusable suggestions and seamless editor integration, and that users want more control, such as genre, emotional tone, instruments, and structural elements.","pith_inferences":["The acceptance rate of roughly 23% (74k of 318k) could serve as a preference signal for training a reward model, though the paper does not analyze acceptance by capability or span type.","The click-track encoding likely constrains the model to the notated meter, so extending it to tempo changes or rubato may require re-encoding; a testable extension is to measure whether generations stay synchronized when the user changes tempo after generation.","The same events/controls reformulation could be applied to other symbolic tasks with explicit timing constraints, such as drum fills or orchestration, wherever a 'fill this span' interface is natural.","A stronger validation would pair the flywheel counts with automated alignment checks (e.g., whether accepted suggestions stay within the selected span in key and meter), which the paper does not report."],"forward_implications":["If the system works as claimed, non-sequential composition—filling in a middle span or swapping melody for harmony—is feasible with a single transformer model inside a commercial editor.","The 318k suggestions and 74k acceptances constitute a real-world interaction dataset that can be mined for implicit feedback on musical quality and style.","The reported user interviews suggest that short, editable, span-specific suggestions preserve songwriter agency better than end-to-end text-to-music generation.","The desire for additional controls (genre, emotion, instruments, structure) points to concrete next steps for conditioning symbolic generation models.","The data flywheel enables A/B testing of generative models in the wild and eventual alignment of the model to user preferences."],"supporting_citations":[{"why":"Supplies the backbone language model with the events/controls anticipation mechanism and fill-in-the-middle capability that Aria fine-tunes.","marker":"[5]"},{"why":"Provides the symbolic music generation baseline and the context of prior transformer-based left-to-right generation that Aria extends to non-sequential control.","marker":"[2]"},{"why":"The GitHub Copilot productivity-and-satisfaction study that motivates the copilot-style integration into an editor.","marker":"[19]"},{"why":"Prior work on AI-steering tools for music co-creation that Aria contrasts with its editor-integrated, non-sequential approach.","marker":"[22]"},{"why":"Prior work on automatic melody harmonization that frames the melody-to-harmony capability Aria includes.","marker":"[26]"}],"fun_headline_variants":["AI copilot for songwriters: 74k suggestions kept","Hookpad Aria: 318k AI ideas, 74k adopted into songs","Aria copilot: fill gaps, continue, reharmonize your song","Songwriting copilot: 74k AI suggestions accepted in Hookpad"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the custom encoding between Hookpad's functional-harmony lead sheets and the MIDI note representation of the Anticipatory Music Transformer, together with the click track used to represent beats in absolute time, preserves enough musical information for the generated suggestions to be coherent and aligned with the user's selected span, key, meter, and tempo.","fun_headline_variants_meta":{"raw":{"variants":["AI copilot for songwriters: 74k suggestions kept","Hookpad Aria: 318k AI ideas, 74k adopted into songs","Aria copilot: fill gaps, continue, reharmonize your song","Songwriting copilot: 74k AI suggestions accepted in Hookpad"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000224,"raw_usage":{"total_tokens":1433,"prompt_tokens":887,"completion_tokens":546,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":464}},"tokens_in":503,"tokens_out":546,"duration_ms":5269,"temperature":1.0,"reasoning_tokens":464,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T10:20:34.627480+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reconstruction or alignment check would settle it: take a set of lead sheets, encode them into the MIDI-plus-click-track format, run the fine-tuned model to generate a fill or continuation, decode back to lead-sheet form, and measure whether the output respects the requested span boundaries, key, meter, and tempo—for instance, whether notes fall on the beat grid defined by the click track and whether the harmony respects the stated key. A high rate of off-grid, out-of-key, or span-violating outputs would indicate the encoding loses the musical information the copilot claims to honor.","supporting_citations":[{"cited_title":"Performance RNN: Generating music with expressive timing and dynamics,","cited_arxiv_id":null,"evidence_quote":"Supplies the backbone language model with the events/controls anticipation mechanism and fill-in-the-middle capability that Aria fine-tunes."},{"cited_title":"LA VE: LLM-powered agent assistance and language augmentation for video editing,","cited_arxiv_id":null,"evidence_quote":"The GitHub Copilot productivity-and-satisfaction study that motivates the copilot-style integration into an editor."},{"cited_title":"On the design of ai-powered code assis- tants for notebooks,","cited_arxiv_id":null,"evidence_quote":"Prior work on AI-steering tools for music co-creation that Aria contrasts with its editor-integrated, non-sequential approach."},{"cited_title":"Cococo: Ai-steering tools for music novices co-creating with generative models","cited_arxiv_id":null,"evidence_quote":"Prior work on automatic melody harmonization that frames the melody-to-harmony capability Aria includes."}],"review_version":1}