REVIEW 4 major objections 5 minor 4 references
Song Hit Prediction: Predicting Billboard Hits Using Spotify Data
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper's random forest classifier distinguishes Billboard Hot 100 hits from non-hits with 87.7% test accuracy using Spotify audio features, track metadata, and artist past performance.
desk verdict The 88% hit-prediction accuracy is a leakage artifact; only the dataset effort survives. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the SpotifyBillboard dataset: roughly 24,000 tracks, balanced between Billboard Hot 100 hits and sampled non-hits, each described by 27 features. The features that carry the argument are Spotify's audio descriptors (danceability, energy, loudness, valence, tempo, acousticness, instrumentalness, liveness, speechiness, key, mode, and time signature), track duration, the explicit flag, album type and release date, a mined artist past-performance count (prior Billboard hits before the track's release), and four fields the paper treats as predictors: Spotify popularity, Billboard rank, weeks on chart, and a custom chart score. The classifier that converts these features into the reported result is a random forest with 80 estimators, a maximum of eight features per split, and the Gini criterion, tuned through randomized and grid search.
What would settle it
Retrain the random forest on the same train/test split after removing rank, weeks, score, and Spotify popularity from the feature set. If accuracy falls substantially below 88% or approaches chance, the reported result depends on post-outcome information; if accuracy stays high, the audio and artist features carry the signal.
Extended reading notes
Core claim
The central claim is that the SpotifyBillboard dataset, a balanced collection of about 24,000 tracks with 27 features, can train a random forest that predicts Billboard Hot 100 membership at 87.7% test accuracy and 88.7% validation accuracy. The paper further reports that random forest beats logistic regression, a one-hidden-layer neural network, and a support vector machine, and that the SVM is the most conservative positive predictor, achieving 99.5% validation precision at the cost of only about 70% recall. Interpreted as the paper intends, the discovery is that Spotify audio descriptors, track metadata, and a mined artist past-performance feature explain most of the variation in chart success.
Load-bearing premise
The load-bearing premise is that rank, weeks on the chart, score, and Spotify popularity are known at prediction time, even though three of those are defined by Billboard chart history and popularity reflects post-release plays; if that timing assumption fails, the accuracy number does not measure prediction.
Editorial extensions
If this is right
- A music label using the random forest model could identify likely Billboard hits with roughly 90% recall among actual hits, at 86% precision.
- The SVM variant offers a near-zero false-positive alternative, so a risk-averse label could use it to select only the most certain hits while accepting many misses.
- Artist past performance and track duration, alongside standard Spotify audio features, are presented as the features that push accuracy beyond earlier hit prediction work.
- If future work adds social context features such as label influence and social media presence, the paper expects accuracy to improve beyond the reported 88%.
- The trained models, especially random forest, could be applied to new Spotify tracks as a practical screening tool for chart potential.
Reading between the lines
- A necessary check the paper does not perform is to retrain the models with rank, weeks, score, and Spotify popularity removed; if accuracy drops sharply, the reported score partly reflects post-release information rather than pre-release prediction.
- A genuinely pre-release test would require feature snapshots taken at or before a track's release date, before any chart data or play-count data accumulates.
- The high accuracy on this balanced dataset likely overstates real-world performance, where hits are far rarer than non-hits and base rates are heavily skewed.
- One testable extension is to compare an audio-only feature set against the full feature set on the same split, which would isolate how much of the signal comes from Spotify's acoustic descriptors versus the chart and popularity fields.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses Hit Song Science: predicting whether a song becomes a Billboard Hot 100 hit. The authors built a dataset of about 1.8 million Spotify tracks merged with Billboard Hot 100 chart data (1985–2018), balanced it to about 12,000 hits and 12,000 non-hits, and trained logistic regression, a neural network, random forest, and support vector machine classifiers on 27 features. The random forest achieves 87.7% test accuracy, which the authors present as evidence that audio features plus an artist past-performance feature can predict Billboard success. The stated goal is prospective prediction that could guide labels' investment decisions.
Significance. If the accuracy figure reflected true prospective predictive skill, the paper would be a useful large-scale contribution to Hit Song Science, one of the largest datasets in this area and with a clear comparison of four standard classifiers. The authors also appropriately report precision and recall, not just accuracy, and they make an explicit effort to include non-hit songs rather than only charted tracks. However, the central claim is undermined by target leakage: several features are direct functions of the Billboard outcome the model is supposed to predict, so the reported accuracy is not evidence of predictive skill for unseen future songs.
major comments (4)
- [§2.1] The features 'weeks', 'rank', and 'score' are direct outputs of Billboard chart appearance: 'weeks' is the total number of weeks on the Hot 100, 'rank' is the chart position (0 for never-charted), and 'score' is a custom weighted rank derived from the same chart data. Because the label 'billboard_hit' is defined as having appeared on the Hot 100, these features encode the label itself. For non-hits they are constant sentinel values (weeks=0, rank=0, score=0.0), so a classifier can separate classes almost trivially. The 87.7% RF test accuracy therefore does not measure the ability to predict whether a future song will become a hit; it measures the model's ability to recover the label from outcome-derived inputs.
- [§2.1, §3] The 'popularity' feature, described as based on total and recent play counts, is a post-release measure that is not available before the Billboard outcome is known. For hits, popularity can reflect the very success the model claims to predict; for non-hits, it reflects the absence of such success. This creates a second leakage path independent of the direct chart features. A random train/validation/test split over tracks does not remove this leakage, because all splits contain the same contaminated feature columns. The paper provides no experiment with these features removed, so its reported accuracy cannot be interpreted as prospective predictive performance.
- [§2.1, §3] The problem framing requires temporal prediction: a label would want to know before release, or shortly after release, whether a song will chart. The paper uses a random split over tracks, not a temporal split by release date, and does not report performance on a held-out period (e.g., songs released after the training period). Without such an evaluation, even the non-leaky features (audio features, artist past performance) have no demonstrated prospective utility. This is load-bearing because the introduction explicitly motivates the work through label investment decisions.
- [§2.1, §4] The paper claims in the abstract and conclusion that random forest was able to 'predict Billboard song success with 88% accuracy.' Given the leakage in the feature set, this statement is unsupported. At best, the model classifies already-charted versus never-charted songs when the chart-derived features are supplied as inputs. The conclusion's statement that audio features 'combined with artist past-performance has managed to explain a majority of the variance' is also unsubstantiated, because no ablation or feature-importance analysis separates the contribution of the audio and past-performance features from the contribution of the leaky chart features.
minor comments (5)
- [§2.1] The bullet list of 27 features omits 'artist past-performance', although the paragraph below the list describes it as an additional feature that was mined; the list should include it or the count should be adjusted.
- [Figure 1] The captions contain typos: 'years with few than 5k songs' should be 'fewer than 5k songs'.
- [§2.3] The logistic regression section says 'RMSEprop' to back-propagate gradients; this appears to be a typo for RMSprop.
- [§4] The conclusion refers to 'Figure 1' for model results, but the model results are shown in Figure 4; the reference is incorrect.
- [References] Reference [4] cites a blog post for RMSprop rather than the original source (Hinton's lecture notes); this is a minor citation clarity issue.
Circularity Check
The 88% accuracy is produced by target leakage: the feature set includes rank, weeks, and score, which are direct Billboard Hot 100 chart outcomes, and popularity, which reflects post-release play counts, so the model reads the label through the features.
-
self definitional
[Section 2.1, Dataset and Features (feature definitions for weeks, rank, score, and billboard_hit)]
"weeks: a value indicating the total number of weeks the track was on the Billboard Hot 100 chart. rank: a value between 0 and 100 indicating a track's position on the Billboard Hot 100 chart. A value of 0 indicates that the track never appeared on the Billboard Hot 100 chart. score: a weighted rank value from 0.0 to 1.0 indicating the popularity of a track. ... A value of 0.0 indicates the track never appeared on the chart. ... billboard_hit: indicates whether the track appeared on the Billboard Hot 100 chart (1=hit, 0=non-hit)."
The target billboard_hit is defined as whether a track appeared on the Billboard Hot 100 chart, and three of the inputs (weeks, rank, score) are defined directly from that same chart appearance. For every hit, rank, weeks, and score are nonzero chart-derived values; for every non-hit, they are sentinel zeros. A classifier can therefore separate the classes almost perfectly by thresholding these columns, and the reported 87.7% test accuracy is an estimate of how well the model reconstructs the label from the label's own chart outcome, not of prospective skill in predicting future hits. The random train/validation/test split does not remove the contamination because every split contains the same leaky feature columns.
-
other
[Section 2.1, Dataset and Features (feature definition for popularity)]
"popularity: a value between 0 and 100, with 100 being the most popular. Popularity is calculated by Spotify, and is based, "in the most part, on the total number of plays the track has had and how recent those plays are" [2]."
The popularity feature is derived from cumulative and recent play counts, which are post-release success measures rather than pre-release song attributes. Using popularity as an input to predict whether a song will become a Billboard hit amounts to using the downstream listening behavior that constitutes the outcome being predicted. This reinforces the central circularity: the 88% figure does not demonstrate Hit Song Science, because the model never has to infer future success from the audio features; it can rely on outcome-derived inputs.
full rationale
The paper's central claim is that a random forest predicts Billboard song success with 88% accuracy. That claim requires the input features to be knowable before the Billboard outcome occurs. The paper's own feature definitions show otherwise: weeks, rank, and score are explicit functions of Billboard Hot 100 chart appearance, and billboard_hit is defined by that same appearance; popularity is a function of total and recent play counts. Thus the model's inputs contain the target's own value, making the high accuracy a label-reconstruction artifact rather than evidence of prediction. The discussion of train/validation/test separation addresses only track identity leakage and does not cure column-level leakage. No self-citation or external-benchmark defense is offered, and no ablation with the leaky features removed is reported. Therefore the main result reduces by construction to its own outcome definition, warranting a circularity score of 8.
Assumptions & free parameters
free parameters (5)
- score weighting scheme =
not specified
- balanced sample size =
12,000 hits and 12,000 non-hits
- RF hyperparameters =
max_features=8, n_estimators=80, min_samples_split=2
- SVM hyperparameters =
gamma=0.1, C=10
- NN hyperparameters =
10 hidden units, batch size 32, 1000 epochs
assumptions (4)
- domain assumption The chart-derived fields weeks, rank, score and Spotify popularity are treated as features available before the Billboard outcome.
- domain assumption Random train/validation/test split preserves the i.i.d. assumption for future song prediction.
- domain assumption Matching Spotify tracks to Billboard tracks by title and artist is correct and unique.
- domain assumption Non-hits from Spotify form a fair counterfactual population of songs that could have been hits.
invented entities (1)
-
score (custom weighted Billboard rank)
Cite this review
Pith. "Pith review of Song Hit Prediction: Predicting Billboard Hits Using Spotify Data." pith.science (2026). https://pith.science/paper/MUKVBROC
@misc{pith2026190808609,
author = {Pith},
title = {Pith review of: Song Hit Prediction: Predicting Billboard Hits Using Spotify Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/MUKVBROC}},
note = {Machine review of arXiv:1908.08609}
}
read the original abstract
In this work, we attempt to solve the Hit Song Science problem, which aims to predict which songs will become chart-topping hits. We constructed a dataset with approximately 1.8 million hit and non-hit songs and extracted their audio features using the Spotify Web API. We test four models on our dataset. Our best model was random forest, which was able to predict Billboard song success with 88% accuracy.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Hitpredict: Predicting hit songs using spotify data
Elena Georgieva, Marcella Suta, and Nicholas Burton. Hitpredict: Predicting hit songs using spotify data. 2018
work page 2018
- [2]
-
[3]
Dorien Herremans, David Martens, and Kenneth Sörensen. Dance hit song prediction. Journal of New Music Research, 43(3):291–302, 2014
work page 2014
-
[4]
Understanding rmsprop: faster neural network learning, 9 2018
Vitaly Bushaev. Understanding rmsprop: faster neural network learning, 9 2018. Referenced on May 10th, 2019. 6
work page 2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.