{"id":"38a3b909-c5ed-4b25-bf7d-c823b4d9bf17","arxiv_id":"2608.07359","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Two CNN detectors that are near-perfect on clean AI music drop to F1 0.19 to 0.47 on real TV broadcast recordings from the new BAMM dataset.","lead":"This paper introduces BAMM, a 40-hour dataset of real TV recordings containing AI-generated and human-made music, and shows that current CNN detectors score far worse on real broadcasts than on clean or simulated audio. The result matters because broadcast monitoring services and AI transparency rules need reliable automated detection, and current models are far from that bar.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BAMM's AI labels are assigned by an ensemble that includes CNN Clean, the model evaluated in Table 1; the RTB numbers therefore measure a label-selected subset, not independent real-world performance.","rationale":"The reader's weakest assumption is that BAMM's AI labels are trustworthy and independent of the models under test, and I agree this is the most load-bearing point. The paper explicitly discloses that CNN Clean is a member of the labeling ensemble in Section 3.1, so the RTB evaluation cannot be read as a clean measurement of how CNN detectors generalize to real broadcast audio. The bias direction is not obvious: because selection happens on clean tracks and the broadcast transform is nonlinear lossy coding plus acoustic mixing, selected tracks could be either easier or harder than the population mean. This ambiguity is exactly why the concern is load-bearing rather than cosmetic. If the selection makes the task easier, the central claim is conservative but the numerical results are not generalizable; if it makes the task harder, the claim overstates the failure. Either way, the benchmark's value as a reference evaluation setting depends on resolving this issue. The reader's conditional verdict is appropriate. I do not see grounds to reject the paper: the authors are transparent about the pipeline, the CFM and STB results provide internal consistency, and the conclusion is explicitly scoped to CNN-based detectors and Suno v3.5. The missing piece is an independent label check or a sensitivity analysis of the labeling pipeline, so the CONDITIONAL verdict should remain unchanged.","tokens_in":8784,"tokens_out":9874,"duration_ms":101670,"concrete_test":"Reconstruct the RTB AI class without the evaluated models: label clean reference tracks using only the four ensemble detectors that are not CNN Clean, or use provenance metadata from the generative platforms, then retrieve the same broadcast occurrences and recompute Table 1 for CNN Clean and CNN Broadcast. Also evaluate CNN Clean on broadcast clips whose clean reference tracks failed the unanimous ensemble vote. If the F1/AUC values shift materially, for example by more than 0.05, or if the rejected clips score systematically differently from the admitted ones, the reported RTB numbers are label-construction artifacts rather than stable measurements of broadcast difficulty.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The RTB result in Table 1 is not an independent estimate of real-world CNN detector performance because the BAMM AI labels are generated by the same model family being evaluated. Section 3.1 states that a candidate reference track is admitted to the AI class only if it exceeds calibrated thresholds across all five detectors in the detector ensemble, and the first detector listed is CNN Clean, the model whose RTB F1/AUC is reported as 0.186/0.707. Thus the positive class is restricted to tracks that CNN Clean already recognizes under clean foreground conditions. The selection can bias the broadcast evaluation in either direction: clean agreements may pick out robust, easily detectable artifacts, making the test easier, or fragile codec-specific artifacts that the 8 kHz / 40 kbps AAC-LC broadcast chain removes, making it harder. Either way, Table 1 measures performance on a cherry-picked subset, not on a representative sample of AI music in TV. The paper's headline conclusion may survive, since failure on an easy subset is still failure, but the specific F1/AUC magnitudes and BAMM's validity as a reference benchmark are not established. The manuscript also limits the benchmark to Suno v3.5 (Section 4), so generalization to other generators is not addressed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BAMM, a 40-hour dataset of real television broadcasts containing AI-generated and human-made music, and uses it to evaluate two CNN-based detectors: CNN Clean, trained on clean foreground music, and CNN Broadcast, trained on synthetic broadcast mixtures. The models are compared across three scenarios: Clean Foreground Music (CFM), Synthetic TV Broadcast (STB), and Real TV Broadcast (RTB). The authors report near-perfect CFM performance, substantial degradation in STB, and further degradation in RTB, with CNN Broadcast outperforming CNN Clean but still showing limited F1 and AUC. The paper concludes that current CNN training approaches are insufficient for reliable AI-generated music detection in real-world broadcast monitoring.","tokens_in":9068,"tokens_out":2757,"duration_ms":27274,"significance":"If the RTB evaluation were valid, the paper would provide a valuable real-world benchmark and a sobering result: state-of-the-art CNN detectors that perform near-perfectly on clean audio degrade dramatically on actual broadcast recordings. The public release of BAMM, the baseline code, and the evaluation scripts are concrete contributions that could enable future work on robust detection and domain adaptation. The paper's main finding—that synthetic broadcast degradation does not fully capture the difficulty of real broadcast conditions—is plausible and important. However, the RTB evaluation is compromised by the label-selection procedure, and the specific F1/AUC magnitudes in Table 1 cannot be interpreted as unbiased estimates of real-world performance.","major_comments":[{"comment":"The AI-labeled reference tracks in BAMM are selected by requiring unanimous agreement from a five-detector ensemble, and the first detector listed is CNN Clean, the very model whose RTB performance is reported in Table 1. Consequently, the RTB positive class is restricted to tracks that CNN Clean already recognizes under clean foreground conditions. This selection can bias the measured F1/AUC in either direction, but in any case the RTB numbers do not measure performance on a representative sample of AI-generated music in broadcast. The paper should either hold CNN Clean out of the labeling ensemble, construct an independent label set (e.g., via metadata or manual expert annotation), or at minimum report results on a subset of BAMM labeled without CNN Clean's involvement, so that the headline degradation is not an artifact of the evaluation protocol.","section":"Section 3.1, Table 1"},{"comment":"The benchmark explicitly focuses on Suno v3.5, and the paper acknowledges that model-agnostic detection remains open. However, the abstract and conclusions state broadly that 'current training approaches on CNN-based detectors remain insufficient for reliable AI-generated music detection in broadcast monitoring.' Since only a single generator family/version is evaluated, this claim is too strong. The conclusions should be qualified to Suno v3.5, or the evaluation should include at least one additional generator (e.g., a newer Suno version or a different system) to support the general statement.","section":"Section 4, Conclusions"},{"comment":"BAMM is built by fingerprint-matching clean reference tracks to broadcast occurrences and then applying a Deep Music Detector filter. The paper does not report how many candidate tracks failed at each pipeline stage, nor how representative the final 40-hour set is of all AI-generated music in the broadcast archive. Without a dropout analysis, it is unclear whether the RTB results reflect the difficulty of the broadcast domain or the peculiarities of the subset of tracks that have clean reference versions, are findable by fingerprinting, and survive the DMD filter. The paper should provide stage-by-stage counts and discuss potential selection effects on the measured performance.","section":"Section 3.2–3.3, Section 4.2.3"}],"minor_comments":[{"comment":"The table reports F1 and ROC values without confidence intervals or any measure of uncertainty. Given that the paper's main numerical claims rest on differences such as F1 0.186 vs. 0.472, the authors should provide bootstrap confidence intervals or at least a significance test to show that the differences are not due to the specific choice of clips or thresholds.","section":"Section 5, Table 1"},{"comment":"The temporal analysis in Figure 2 is presented as validation of the labeling strategy, but it only shows aggregate score distributions over time. It would be more informative to show that the unanimous-agreement subset is stable under perturbation of the ensemble composition, since the ensemble includes the model under evaluation.","section":"Section 3.1, Figure 2"},{"comment":"The description of CNN Broadcast training states that speech segments are concatenated to span the full music duration and mixed at random SNR between -30 dB and +30 dB. It is unclear whether the speech is mixed at constant gain or whether the SNR is drawn per clip; this detail should be clarified for reproducibility.","section":"Section 4.1.2"},{"comment":"The related work section cites several relevant papers, but the discussion of detection robustness would benefit from a reference to recent work on out-of-distribution generalization for audio deepfake detection, which is closely connected to the domain-gap argument made here.","section":"Section 2, references"}],"recommendation":"major_revision","confidential_remarks":"The main concern is the circularity in the BAMM label construction: the AI class is defined by unanimous agreement of an ensemble that includes the evaluated model. This is a load-bearing issue for the RTB results. The paper's contribution of a real-broadcast dataset is potentially valuable, but the evaluation protocol needs to be revised to avoid self-selection. I would not reject outright, since the STB and CFM results are less affected and the authors may be able to re-analyze BAMM with a held-out labeling scheme. However, the current version of Table 1 and the RTB conclusions should not be published as-is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Name],\n\nYou should know two things about this paper. First, it introduces BAMM, a 40-hour real-TV dataset for AI-generated music detection, with code and data publicly released. That fills a real gap: prior work only used synthetic mixtures. Second, the real-broadcast evaluation numbers are entangled with the way the AI labels were built, so the specific F1/AUC values in Table 1 are not trustworthy as they stand.\n\nThe paper does some things well. The comparison of clean-trained and broadcast-trained CNN variants across CFM, STB, and RTB is clearly described. The score distributions and foreground/background breakdown are useful. The authors are upfront about the scope: only Suno v3.5, only CNN-based detectors. Shipping the dataset and code is a concrete contribution that others can build on.\n\nThe soft spot is load-bearing. Section 3.1 says a reference track is labeled AI only if all five detectors in the ensemble agree, and the first detector listed is CNN Clean, the same model evaluated in Table 1. That means the AI class in BAMM is restricted to tracks that CNN Clean already recognizes under clean foreground conditions. The RTB evaluation then measures performance on that selected subset, not on a representative sample of AI music in TV. The direction of the bias is unclear—it could make the test easier or harder—but either way the headline numbers are not an independent estimate. The qualitative conclusion may survive, since failing even on an easy subset is still failure, but the magnitudes are not established.\n\nSecondary concerns: no confidence intervals, and the benchmark is limited to one generator. That last one is acknowledged, so it is a scope limit, not an oversight.\n\nBottom line: this is a paper worth engaging with. The dataset is a real contribution, and the evaluation design problem is instructive. It deserves peer review, but a referee should insist the authors either re-run RTB on labels produced without the evaluated models, or quantify the selection effect. I would cite BAMM as a benchmark even with that caveat.","headline":"Useful real-world benchmark dataset, but the RTB detection numbers are entangled with the label-generation pipeline, so the headline degradation is plausible but the specific magnitudes are not established.","tokens_in":9586,"tokens_out":1699,"would_cite":true,"duration_ms":15319,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces BAMM, a 40-hour real-TV dataset, and finds that current CNN detectors for AI-generated music drop from near-perfect $F_1$ scores on clean tracks to 0.186 (clean-trained) and 0.472 (broadcast-trained) on real…","keywords":["AI-generated music detection","broadcast monitoring","BAMM dataset","CNN detector","domain gap","audio fingerprinting","real-world evaluation","AI music artifacts"],"falsifier":"Independently re-label a random sample of BAMM clips using official release provenance, artist metadata, or expert listening, and compare with the ensemble labels; if independent labels disagree often on the AI class, the reported RTB $F_1$ drop is inflated by selection bias, while strong agreement would confirm that current detectors genuinely fail on real broadcasts.","tokens_in":8583,"feed_emoji":"📺","tokens_out":7826,"duration_ms":63933,"temperature":0.7,"pith_summary":"This paper argues that current CNN-based AI-generated-music detectors, which look close to perfect on isolated clean tracks, are not reliable when deployed on real television broadcasts. To test this, it introduces BAMM, a 40-hour dataset of actual TV recordings labeled as AI-generated or human-made through audio fingerprinting and a strict five-model consensus. On this dataset, the clean-trained CNN drops from $F_1$ 0.992 under controlled conditions to $F_1$ 0.186, and the broadcast-trained variant drops to $F_1$ 0.472, with both models showing substantial score overlap between AI and human music. The finding matters because broadcast monitoring is the setting where AI-music detection would actually be used, and synthetic broadcast benchmarks substantially overstate how ready these tools are.","feed_headline":"Real TV cuts AI-music detector F1 from 0.99 to 0.19","feed_subtitle":"New 40-hour BAMM benchmark shows clean- and broadcast-trained CNNs still confuse AI and human music on real broadcasts.","key_machinery":"The load-bearing machinery is the BAMM curation pipeline plus a fixed CNN architecture. BAMM is built by taking clean reference tracks, labeling the AI class only when five different detectors—including the model later evaluated as CNN Clean—agree unanimously, finding their occurrences in a global TV archive through landmark audio fingerprinting, and filtering retrieved clips with a deep music detector that keeps only foreground or background music. The evaluated models are six-layer CNNs on mel-spectrograms that are identical except for training data: CNN Clean uses clean foreground music only, while CNN Broadcast mixes the same music with speech at random signal-to-noise ratios and encodes it to 8 kHz mono at 40 kbps to emulate broadcast degradation. Because the architecture is fixed, the comparison isolates the effect of training domain, and BAMM supplies the first real-TV test set against which the synthetic broadcast benchmark can be checked.","core_discovery":"The central discovery is that the domain gap between clean foreground music and real TV broadcast audio is large enough to break current artifact-based CNN detectors. In the paper's three-scenario evaluation, both the clean-trained and broadcast-trained variants achieve $F_1$ above 0.99 on clean foreground music (CFM), but on synthetic broadcast mixtures (STB) they fall to $F_1$ 0.342 and 0.661, and on the real broadcast clips of BAMM (RTB) they fall to $F_1$ 0.186 and 0.472, with ROC-AUC 0.707 and 0.775 respectively. The score distributions show the failure mode: human-made clips are usually scored near zero, but many AI-generated clips also receive low scores, so the models miss AI music far more often than they false-alarm on human music. The authors interpret this as evidence that broadcast masking, low-bitrate encoding, and background placement hide the decoding artifacts these models were trained on, making current training approaches insufficient for reliable detection in broadcast monitoring.","pith_inferences":["Because the AI label requires unanimous agreement among five clean-trained detectors, BAMM's AI class likely contains only tracks that are easy to flag in clean conditions; if so, recall on arbitrary AI broadcast music could be even lower than the reported $F_1$ values.","A natural next experiment, not run here, is to fine-tune or augment the detector on BAMM clips and test on held-out channels and dates; large gains would show the gap is distributional, while small gains would point to an intrinsic loss of detectable artifacts in 8 kHz/40 kbps audio.","The 8 kHz mono, AAC-LC proxy stream removes much of the high-frequency spectrum where decoding artifacts live, so detectors using different front-ends—for example source-separation or artifact fingerprints—may behave quite differently from these mel-spectrogram CNNs on the same BAMM clips."],"forward_implications":["Neither clean-trained nor broadcast-trained CNNs can currently serve as a reliable AI-generated-music monitor on real TV: best-case $F_1$ is 0.472 with AUC 0.775 on RTB.","Adding speech mixtures and low-bitrate encoding to training helps—CNN Broadcast beats CNN Clean on every broadcast scenario—but the remaining gap shows that training-domain matching alone does not solve the problem.","Synthetic broadcast benchmarks overstate real-world readiness: STB $F_1$ scores (0.342 and 0.661) are consistently higher than RTB $F_1$ scores (0.186 and 0.472).","Background music is harder than foreground music for both models, with foreground AUC values of 0.782 and 0.858 versus background AUC values of 0.667 and 0.745 on BAMM.","The dominant failure is missed AI content rather than false alarms, since human clips concentrate near zero while many AI clips are also scored low."],"supporting_citations":[{"why":"Supplies the CNN architecture and the decoding-artifact detection approach that CNN Clean and CNN Broadcast are built on.","marker":"[6]"},{"why":"Provides AI-OpenBMAT, the matched clean and synthetic broadcast material used for the CFM and STB scenarios, and reports the synthetic degradation this work extends.","marker":"[7]"},{"why":"Provides the SONICS Suno v3.5 dataset used as the AI-generated training class and supplies one of the five detectors in the labeling ensemble.","marker":"[13]"},{"why":"Supplies the verified human-composed tracks used to calibrate the ensemble to zero false positives on the human baseline.","marker":"[20]"},{"why":"Describes the landmark audio fingerprinting method used to locate reference tracks inside the TV broadcast archive.","marker":"[21]"},{"why":"Supplies the speech material used to construct CNN Broadcast's mixed music-plus-speech training samples.","marker":"[22]"},{"why":"Defines the deep music detector used to filter BAMM clips down to foreground and background music segments.","marker":"[16]"}],"fun_headline_variants":["Real TV wrecks AI-music detection: F1 crashes to 0.19","Broadcast audio blinds AI-music CNNs: F1 0.19 on real TV","New BAMM benchmark: Real TV audio kills AI-music F1 (0.19)","AI-music detectors flop on real TV: F1 drops to 0.19"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All BAMM AI labels come from unanimous agreement among five clean-trained detectors, so the reported real-broadcast degradation could partly reflect that particular labeling rule rather than the intrinsic difficulty of AI-generated music in TV audio.","fun_headline_variants_meta":{"raw":{"variants":["Real TV wrecks AI-music detection: F1 crashes to 0.19","Broadcast audio blinds AI-music CNNs: F1 0.19 on real TV","New BAMM benchmark: Real TV audio kills AI-music F1 (0.19)","AI-music detectors flop on real TV: F1 drops to 0.19"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000986,"raw_usage":{"total_tokens":4185,"prompt_tokens":954,"completion_tokens":3231,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":3135}},"tokens_in":570,"tokens_out":3231,"duration_ms":21469,"temperature":1.0,"reasoning_tokens":3135,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T05:36:03.512834+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently re-label a random sample of BAMM clips using official release provenance, artist metadata, or expert listening, and compare with the ensemble labels; if independent labels disagree often on the AI class, the reported RTB $F_1$ drop is inflated by selection bias, while strong agreement would confirm that current detectors genuinely fail on real broadcasts.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CNN architecture and the decoding-artifact detection approach that CNN Clean and CNN Broadcast are built on."},{"cited_title":"Ai music tops 50% of daily uploads on deezer,","cited_arxiv_id":null,"evidence_quote":"Provides the SONICS Suno v3.5 dataset used as the AI-generated training class and supplies one of the five detectors in the labeling ensemble."},{"cited_title":"High-fidelity audio compression with im- proved RVQGAN,","cited_arxiv_id":null,"evidence_quote":"Supplies the verified human-composed tracks used to calibrate the ensemble to zero false positives on the human baseline."},{"cited_title":"A Fourier Explanation of AI-music Artifacts,","cited_arxiv_id":null,"evidence_quote":"Describes the landmark audio fingerprinting method used to locate reference tracks inside the TV broadcast archive."},{"cited_title":"Sonics: Synthetic or not - identify- ing counterfeit songs,","cited_arxiv_id":null,"evidence_quote":"Supplies the speech material used to construct CNN Broadcast's mixed music-plus-speech training samples."},{"cited_title":"AI-generated music detection in broadcast monitoring,","cited_arxiv_id":null,"evidence_quote":"Defines the deep music detector used to filter BAMM clips down to foreground and background music segments."}],"review_version":1}