{"id":"0817274d-06af-40c9-bae1-bd8132024bfc","arxiv_id":"1908.08609","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A random forest classifier trained on Spotify features plus Billboard chart-derived fields reports 87.7 percent test accuracy, but the chart-derived fields leak the target and invalidate the predictive claim.","lead":"This paper applies standard machine learning classifiers to Spotify audio features and reports 88 percent accuracy for predicting Billboard Hot 100 hits. The headline result is undermined because the feature set includes the chart position and chart-derived scores, so the models are largely reading the target variable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 88% accuracy is likely an artifact of target leakage: rank, weeks, score, and popularity are all defined after or by the Billboard outcome the model claims to predict.","rationale":"The reader identified the same load-bearing concern: Section 2.1 lists rank, weeks, score, and popularity as features, all of which are unavailable at release time and largely determined by the Billboard outcome being predicted. My reading of the manuscript confirms this: the dataset construction merges Spotify and Billboard data, and no attempt is made to restrict features to pre-release information. The random split in Section 2.1 and Figure 2 does not address leakage because the leaked columns are present in every split. The RF result of 87.7% test accuracy is therefore not a valid demonstration of hit prediction skill. I agree with the reader's verdict of REJECT, and my concern does not require changing that verdict. I also note the SVM precision of 0.995 reported in Table 1 is suspiciously high, consistent with the model exploiting a near-deterministic leak rather than learning a general pattern, which further supports the leakage interpretation. No other concern is more load-bearing than this one.","tokens_in":4216,"tokens_out":1451,"duration_ms":17606,"concrete_test":"Retrain the random forest with the exact pipeline but exclude weeks, rank, score, and popularity from the feature set, and also split the data temporally so that all training songs have release dates before all test songs. If accuracy falls to near chance, or even materially below the reported 0.877, the headline result depends on outcome leakage and is not evidence of prospective hit prediction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a random forest using the described features can predict Billboard success 88% of the time. That claim requires the features to be knowable before the outcome. In Section 2.1, the feature list includes 'weeks', 'rank', and 'score', which are direct functions of Billboard chart appearance, and 'popularity', which Spotify defines from cumulative and recent play counts. For a hit song, these features are largely determined by the very chart outcome used to create the label 'billboard_hit'; for a non-hit, 'weeks' and 'rank' are fixed at their never-charted sentinel values. A model trained on these features can trivially separate hits from non-hits by reading the label through the features, and the random train/validation/test split does not remove this leakage because every split contains the same contaminated feature columns. The 88% figure therefore does not measure predictive skill for future releases; it measures the model's ability to reconstruct the label from outcome-derived inputs. This is the single most load-bearing concern because the paper's stated purpose, solving Hit Song Science, and its practical framing (labels investing in likely hits) both require prospective prediction. The manuscript itself provides no evidence that any accuracy remains when the leaky features are removed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses Hit Song Science: predicting whether a song becomes a Billboard Hot 100 hit. The authors built a dataset of about 1.8 million Spotify tracks merged with Billboard Hot 100 chart data (1985–2018), balanced it to about 12,000 hits and 12,000 non-hits, and trained logistic regression, a neural network, random forest, and support vector machine classifiers on 27 features. The random forest achieves 87.7% test accuracy, which the authors present as evidence that audio features plus an artist past-performance feature can predict Billboard success. The stated goal is prospective prediction that could guide labels' investment decisions.","tokens_in":4468,"tokens_out":1775,"duration_ms":19315,"significance":"If the accuracy figure reflected true prospective predictive skill, the paper would be a useful large-scale contribution to Hit Song Science, one of the largest datasets in this area and with a clear comparison of four standard classifiers. The authors also appropriately report precision and recall, not just accuracy, and they make an explicit effort to include non-hit songs rather than only charted tracks. However, the central claim is undermined by target leakage: several features are direct functions of the Billboard outcome the model is supposed to predict, so the reported accuracy is not evidence of predictive skill for unseen future songs.","major_comments":[{"comment":"The features 'weeks', 'rank', and 'score' are direct outputs of Billboard chart appearance: 'weeks' is the total number of weeks on the Hot 100, 'rank' is the chart position (0 for never-charted), and 'score' is a custom weighted rank derived from the same chart data. Because the label 'billboard_hit' is defined as having appeared on the Hot 100, these features encode the label itself. For non-hits they are constant sentinel values (weeks=0, rank=0, score=0.0), so a classifier can separate classes almost trivially. The 87.7% RF test accuracy therefore does not measure the ability to predict whether a future song will become a hit; it measures the model's ability to recover the label from outcome-derived inputs.","section":"§2.1"},{"comment":"The 'popularity' feature, described as based on total and recent play counts, is a post-release measure that is not available before the Billboard outcome is known. For hits, popularity can reflect the very success the model claims to predict; for non-hits, it reflects the absence of such success. This creates a second leakage path independent of the direct chart features. A random train/validation/test split over tracks does not remove this leakage, because all splits contain the same contaminated feature columns. The paper provides no experiment with these features removed, so its reported accuracy cannot be interpreted as prospective predictive performance.","section":"§2.1, §3"},{"comment":"The problem framing requires temporal prediction: a label would want to know before release, or shortly after release, whether a song will chart. The paper uses a random split over tracks, not a temporal split by release date, and does not report performance on a held-out period (e.g., songs released after the training period). Without such an evaluation, even the non-leaky features (audio features, artist past performance) have no demonstrated prospective utility. This is load-bearing because the introduction explicitly motivates the work through label investment decisions.","section":"§2.1, §3"},{"comment":"The paper claims in the abstract and conclusion that random forest was able to 'predict Billboard song success with 88% accuracy.' Given the leakage in the feature set, this statement is unsupported. At best, the model classifies already-charted versus never-charted songs when the chart-derived features are supplied as inputs. The conclusion's statement that audio features 'combined with artist past-performance has managed to explain a majority of the variance' is also unsubstantiated, because no ablation or feature-importance analysis separates the contribution of the audio and past-performance features from the contribution of the leaky chart features.","section":"§2.1, §4"}],"minor_comments":[{"comment":"The bullet list of 27 features omits 'artist past-performance', although the paragraph below the list describes it as an additional feature that was mined; the list should include it or the count should be adjusted.","section":"§2.1"},{"comment":"The captions contain typos: 'years with few than 5k songs' should be 'fewer than 5k songs'.","section":"Figure 1"},{"comment":"The logistic regression section says 'RMSEprop' to back-propagate gradients; this appears to be a typo for RMSprop.","section":"§2.3"},{"comment":"The conclusion refers to 'Figure 1' for model results, but the model results are shown in Figure 4; the reference is incorrect.","section":"§4"},{"comment":"Reference [4] cites a blog post for RMSprop rather than the original source (Hinton's lecture notes); this is a minor citation clarity issue.","section":"References"}],"recommendation":"reject","confidential_remarks":"The target-leakage problem is severe enough that the paper's central quantitative claim cannot be fixed by local revisions; an entirely new feature set and evaluation design would be needed. I would also note that the authors do not share code or data, so reproducibility of the dataset construction and the leakage analysis is limited. This manuscript should not be accepted in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou need to know two things about this paper: the headline 88% accuracy is an artifact of target leakage, and the dataset itself is the only contribution that might survive the fix.\n\nWhat is new: they built a ~24k balanced dataset from 1.8M Spotify tracks, added an artist past-performance feature, and ran four off-the-shelf classifiers. That is a real data-collection effort, and the feature descriptions are clear.\n\nThe problem is fatal to the central claim. Section 2.1 lists `weeks`, `rank`, and `score` as features, all direct functions of the Billboard chart outcome that defines `billboard_hit`. `popularity` is also post-release, since Spotify defines it from cumulative and recent plays. For a hit, these features are essentially the label itself; for a non-hit, they take sentinel values. A random forest cannot miss. The random split does not matter because the contamination lives in the columns, not the rows. The paper calls this prediction, but no temporal split is used, and no evidence exists that accuracy survives removing those four features.\n\nThe rest is minor: no dataset link, only three prior works cited, no feature-importance analysis to at least diagnose the leakage. The results tables are reported honestly, but the interpretation is wrong.\n\nI would desk-reject the current version: the main result is not a result. But I would invite a resubmission if the authors (a) release the dataset, (b) drop chart-derived and post-release features, and (c) use a temporal split (e.g., train pre-2010, test post-2010). An audio-features-only baseline would be useful for Hit Song Science. As it stands, this is a cautionary example of leakage, not a scientific contribution.\n\nMy honest editorial call: reject, but don't burn the bridge.","headline":"The 88% hit-prediction accuracy is a leakage artifact; only the dataset effort survives.","tokens_in":4975,"tokens_out":3415,"would_cite":false,"duration_ms":35356,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's random forest classifier distinguishes Billboard Hot 100 hits from non-hits with 87.7% test accuracy using Spotify audio features, track metadata, and artist past performance.","keywords":["hit song science","machine learning","classification","random forest","Spotify audio features","Billboard Hot 100","data mining","artist past performance"],"falsifier":"Retrain the random forest on the same train/test split after removing rank, weeks, score, and Spotify popularity from the feature set. If accuracy falls substantially below 88% or approaches chance, the reported result depends on post-outcome information; if accuracy stays high, the audio and artist features carry the signal.","tokens_in":1288,"feed_emoji":"🎵","tokens_out":1723,"duration_ms":57686,"temperature":0.7,"pith_summary":"This paper attempts to settle the Hit Song Science question, whether chart success can be predicted from measurable song and artist features, by building a balanced dataset of roughly 12,000 Billboard Hot 100 hits and 12,000 Spotify non-hits. It reports that a random forest model reaches 87.7% accuracy on a held-out test set and 88.7% on validation, with 0.86 precision and 0.90 recall. The claim matters because a working pre-release hit predictor would let labels allocate promotion budgets, artists choose between mainstream and niche directions, and streaming platforms surface songs more effectively. The paper presents its large dataset and the addition of artist past-performance as the key advances over earlier, smaller hit-prediction studies.","feed_headline":"Spotify data predicts Billboard hits with 88% accuracy","feed_subtitle":"A random-forest model trained on audio features and artist history separates hits from non-hits on a 24,000-song test.","key_machinery":"The central object is the SpotifyBillboard dataset: roughly 24,000 tracks, balanced between Billboard Hot 100 hits and sampled non-hits, each described by 27 features. The features that carry the argument are Spotify's audio descriptors (danceability, energy, loudness, valence, tempo, acousticness, instrumentalness, liveness, speechiness, key, mode, and time signature), track duration, the explicit flag, album type and release date, a mined artist past-performance count (prior Billboard hits before the track's release), and four fields the paper treats as predictors: Spotify popularity, Billboard rank, weeks on chart, and a custom chart score. The classifier that converts these features into the reported result is a random forest with 80 estimators, a maximum of eight features per split, and the Gini criterion, tuned through randomized and grid search.","core_discovery":"The central claim is that the SpotifyBillboard dataset, a balanced collection of about 24,000 tracks with 27 features, can train a random forest that predicts Billboard Hot 100 membership at 87.7% test accuracy and 88.7% validation accuracy. The paper further reports that random forest beats logistic regression, a one-hidden-layer neural network, and a support vector machine, and that the SVM is the most conservative positive predictor, achieving 99.5% validation precision at the cost of only about 70% recall. Interpreted as the paper intends, the discovery is that Spotify audio descriptors, track metadata, and a mined artist past-performance feature explain most of the variation in chart success.","pith_inferences":["A necessary check the paper does not perform is to retrain the models with rank, weeks, score, and Spotify popularity removed; if accuracy drops sharply, the reported score partly reflects post-release information rather than pre-release prediction.","A genuinely pre-release test would require feature snapshots taken at or before a track's release date, before any chart data or play-count data accumulates.","The high accuracy on this balanced dataset likely overstates real-world performance, where hits are far rarer than non-hits and base rates are heavily skewed.","One testable extension is to compare an audio-only feature set against the full feature set on the same split, which would isolate how much of the signal comes from Spotify's acoustic descriptors versus the chart and popularity fields."],"forward_implications":["A music label using the random forest model could identify likely Billboard hits with roughly 90% recall among actual hits, at 86% precision.","The SVM variant offers a near-zero false-positive alternative, so a risk-averse label could use it to select only the most certain hits while accepting many misses.","Artist past performance and track duration, alongside standard Spotify audio features, are presented as the features that push accuracy beyond earlier hit prediction work.","If future work adds social context features such as label influence and social media presence, the paper expects accuracy to improve beyond the reported 88%.","The trained models, especially random forest, could be applied to new Spotify tracks as a practical screening tool for chart potential."],"supporting_citations":[{"why":"Provides the prior hit-prediction approach using Spotify data that this paper extends with a larger dataset and additional features.","marker":"[1]"},{"why":"Defines the Spotify popularity feature used as a predictor in the model.","marker":"[2]"},{"why":"Supplies earlier audio-feature-based hit song prediction that motivates the choice of Spotify audio descriptors.","marker":"[3]"},{"why":"Describes the RMSprop optimizer used to train the neural network model reported in the comparison.","marker":"[4]"}],"fun_headline_variants":["Random forest hits 88% accuracy on Billboard prediction","Spotify features forecast Billboard hits with 88% accuracy","88% accuracy: Random forest predicts chart success from Spotify data","Random forest and Spotify data predict Billboard hits at 88%","Billboard hit prediction: random forest scores 88% accuracy"],"cache_read_input_tokens":7168,"weakest_assumption_plain":"The load-bearing premise is that rank, weeks on the chart, score, and Spotify popularity are known at prediction time, even though three of those are defined by Billboard chart history and popularity reflects post-release plays; if that timing assumption fails, the accuracy number does not measure prediction.","fun_headline_variants_meta":{"raw":{"variants":["Random forest hits 88% accuracy on Billboard prediction","Spotify features forecast Billboard hits with 88% accuracy","88% accuracy: Random forest predicts chart success from Spotify data","Random forest and Spotify data predict Billboard hits at 88%","Billboard hit prediction: random forest scores 88% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000368,"raw_usage":{"total_tokens":1872,"prompt_tokens":738,"completion_tokens":1134,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":354,"completion_tokens_details":{"reasoning_tokens":1051}},"tokens_in":354,"tokens_out":1134,"duration_ms":7361,"temperature":1.0,"reasoning_tokens":1051,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:34:23.977930+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the random forest on the same train/test split after removing rank, weeks, score, and Spotify popularity from the feature set. If accuracy falls substantially below 88% or approaches chance, the reported result depends on post-outcome information; if accuracy stays high, the audio and artist features carry the signal.","supporting_citations":[{"cited_title":"Hitpredict: Predicting hit songs using spotify data","cited_arxiv_id":null,"evidence_quote":"Provides the prior hit-prediction approach using Spotify data that this paper extends with a larger dataset and additional features."},{"cited_title":"Song popularity, 2019","cited_arxiv_id":null,"evidence_quote":"Defines the Spotify popularity feature used as a predictor in the model."},{"cited_title":"Dance hit song prediction","cited_arxiv_id":null,"evidence_quote":"Supplies earlier audio-feature-based hit song prediction that motivates the choice of Spotify audio descriptors."},{"cited_title":"Understanding rmsprop: faster neural network learning, 9 2018","cited_arxiv_id":null,"evidence_quote":"Describes the RMSprop optimizer used to train the neural network model reported in the comparison."}],"review_version":1}