{"id":"6d65c8eb-621b-4ce8-9979-933c8ef737d3","arxiv_id":"1908.08769","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A combined bag-of-words lyrics network and spectrogram CNN classify Lukthung songs from other Thai genres with an F1 of 0.86 on a private 10,547-song dataset.","lead":"Neural networks trained on song lyrics and audio clips can tell Lukthung, a Thai country-music genre, apart from other Thai music with an F1 score of 0.86. The method could help Thai streaming services personalize recommendations by filtering Lukthung songs for listeners who do not enjoy them.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random song-level split without an artist-disjoint partition can inflate the reported F1 of 0.86, undermining the claim of classifying songs from never-seen-before artists.","rationale":"The artist-leakage issue is the most load-bearing problem, and it matches the reader's weakest_assumption. The paper explicitly frames the model as handling unknown artists, but the evaluation protocol tests random songs, not random artists. Artist is a powerful nuisance variable: songs by one artist share lyrics, vocal timbre, instrumentation, and production style. In a random split those cues are available at training time, so the model may learn artist-dependent shortcuts. This is especially plausible for the BoW-MLP because Lukthung lyrics have characteristic vocabulary; an artist who consistently uses certain words will have those words in both training and test. The audio model similarly can memorize producer and studio characteristics. Without an artist-disjoint experiment or per-artist statistics, the 0.86 F1 cannot be interpreted as genre-level generalization. Other weaknesses, such as missing error bars and absent baselines, are secondary: they would not invalidate the headline claim if artist-disjoint performance held. Therefore the conditional verdict is appropriate, pending an artist-disjoint evaluation.","tokens_in":7977,"tokens_out":3694,"duration_ms":36631,"concrete_test":"Create an artist-disjoint split by assigning all songs by the same artist to the same fold. Retrain BoW-MLP, Spectro-CNN, and the combined model on 55% of artists, validate on 20%, and test on the remaining 25%. Report F1, precision, recall, and the number of unique artists with song counts. If the combined F1 drops by more than about 0.05 from 0.856, or if the artist count is small, the claimed generalization to never-seen-before artists is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the combined model classifies Lukthung and generalizes to songs from 'unknown or never-seen-before artists' (Section IV-A). The only evaluation in Section VI is a random split of songs into 55/20/25 with no artist-disjoint partition. Removing artist names from lyrics does not remove artist identity: lyrics vocabularies, songwriting habits, and audio production characteristics are artist-specific. With roughly 10.5k songs from six genres, the number of artists is likely small enough that random splits place multiple songs by the same artist in both training and test. The BoW-MLP can then memorize artist-level word patterns rather than genre-level Lukthung properties, and the audio model can similarly exploit artist-specific timbral or production cues. The paper's own error analysis supports this risk: false positives include 'Lukthung songs sung with Pop/Rock artists,' indicating that artist identity is a confound. Therefore the reported F1 of 0.856 in Table II may be inflated and does not substantiate the never-seen-before-artist generalization claim. No artist-disjoint results, per-artist statistics, or error bars are provided, so this is currently unresolved rather than refuted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a binary classification system for distinguishing Lukthung, a Thai country music genre, from five other genres using lyrics and audio. A bag-of-words representation fed to a two-layer MLP is used for lyrics; a spectrogram-based CNN with vertical and horizontal filters is used for audio; and a combined model concatenates the last-layer features of both. The dataset consists of 10,547 expert-tagged Thai songs from 1985 to 2019. On a random 55/20/25 split, the combined model achieves F1 = 0.8561, outperforming the audio-only baselines (RF, LR, SVM) and the individual proposed models.","tokens_in":8228,"tokens_out":5813,"duration_ms":51415,"significance":"If the reported performance holds under rigorous evaluation, the paper would be a useful demonstration of multimodal genre classification for a non-Western music genre, with practical implications for music streaming services. The authors make several commendable choices: a sizable expert-annotated corpus, explicit architectural adaptations to Lukthung-specific traits such as vibrato and traditional instruments, and an analysis of false positives and false negatives. However, the evaluation protocol currently does not support the paper's generalization claim to never-before-seen artists, and the statistical significance of the gains is not established.","major_comments":[{"comment":"The paper claims in Section IV-A that removing artist names allows classification of 'songs from unknown or never-seen-before artists,' but Section VI evaluates only a random song-level split (55/20/25) with no artist-disjoint partition. Because lyrics vocabulary, songwriting habits, and audio production are artist-specific, a random split can place songs by the same artist in both training and test, allowing the models to memorize artist identity rather than genre. The error analysis in Section VI reports false positives including 'Lukthung songs sung with Pop/Rock artists,' which indicates that artist identity is a confound. The authors should either provide an artist-disjoint split with per-artist statistics or retract the generalization claim.","section":"IV-A, VI"},{"comment":"The F1 differences between the combined model (0.8561), BoW-MLP (0.7905), and Spectro-CNN (0.7730) are reported without confidence intervals, significance tests, or multiple runs. A single random split cannot establish that the observed gains are not due to chance; the authors should report mean and standard deviation over several seeds, or at least a paired significance test, to support the claim that combining features substantially improves performance.","section":"VI, Table II"},{"comment":"The comparison baselines are limited to audio-only classifiers (RF, LR, SVM) with default Scikit-learn parameters and MFCC features; no lyrics-only baselines (e.g., TF-IDF with SVM) or standard multimodal baselines are included. Consequently, the abstract's claim that the proposed models 'outperform all of the standard classifiers' rests on a narrow set of weak baselines. Adding a lyrics baseline and a late-fusion baseline would make the comparison more informative and the claim more accurate.","section":"VI, Table II"}],"minor_comments":[{"comment":"In Section V-B, the phrase 'We used the chorus part solely not only for computational reasons but also because...' is ungrammatical; please rephrase to 'not only... but also...'.","section":"V-B"},{"comment":"Reference [16] misspells 'October' as 'Octoboer' and Reference [11] misspells 'International' as 'Internataional'.","section":"References"},{"comment":"Table II is garbled for the combined model row: the Input column reads 'Lyrics & Combined model' and 'Audios' appears after the F1 value. Please reformat the table.","section":"VI, Table II"},{"comment":"The normalization in Eq. (1) divides by $\\max \\log c_i$, which is undefined when the maximum word count of a song is 1; please clarify how such cases are handled.","section":"V-A, Eq. (1)"},{"comment":"The claim that the best scores are highlighted in bold should be verified; the bold formatting is not visible in the manuscript text, and the final version should ensure consistent highlighting.","section":"VI"},{"comment":"In Section III, 'There are few songs before 2000s' should be 'before the 2000s' or 'before 2000'.","section":"III"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal and the topic is of potential interest. The evaluation must be strengthened with an artist-disjoint split and statistical tests before I can recommend acceptance. If the authors are unable to provide an artist-disjoint evaluation, the claims should be downgraded to classification on a random split of the collected dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is that the paper opens a new dataset and task: it is the first to train neural classifiers for Lukthung, a major Thai genre, using both lyrics and audio. The audio model adapts the Pons et al. filter design to preserve vibrato by using small frequency-band horizontal filters before pooling, which is a sensible domain adaptation. The combined model, which concatenates the last-layer features from the BoW-MLP and the Spectro-CNN, does beat both single-modality models on the reported test set (F1 0.856 vs 0.773 for audio and 0.791 for lyrics). That result, if reliable, matters for Thai streaming services.\n\nThe soft spot is the evaluation. The paper says the goal is to classify songs from unknown or never-seen-before artists, but the only split is a random 55/20/25 split of songs. With roughly 10.5k songs across six genres, the number of artists is small enough that the same artists almost certainly appear in train and test. The BoW model can memorize artist-specific vocabulary, and the audio model can exploit production and vocal signatures that are artist-specific rather than genre-specific. The paper's own error analysis supports this: it explicitly notes false positives that are \"Lukthung songs sung with Pop/Rock artists,\" which is a direct sign that artist identity is a confound. So the central improvement claim is plausible but not established for the stated generalization scenario.\n\nOther issues are more minor: no confidence intervals or significance tests; the baselines are audio-only with default scikit-learn parameters, so the \"combined beats traditional\" comparison is not against a strong multimodal baseline; and there is no lyrics baseline from the traditional-methods side. The chorus start is approximated at 30% of the song, which could add noise. The paper does not release code or data, but for a Tencent-related project that is not unusual.\n\nMy overall take: the work is a reasonable first application, not a methodological advance. The central flaw is fixable: add an artist-disjoint split, report variance or per-artist statistics, and include a standard lyrics baseline. I would not desk-reject it, but I would send it to review with the expectation of major revision, primarily to make the evaluation match the claim.\n\nWho is this for? Researchers working on non-Western music classification, and to a lesser extent anyone who wants a concrete example of the artist-leakage problem in music tagging. The paper is worth reading for that cautionary lesson, and worth a serious referee.","headline":"A genuinely new Lukthung benchmark that fails to test its own never-seen-artist claim because the data is split at song level, not artist level, so the reported F1 is likely optimistic.","tokens_in":8752,"tokens_out":1551,"would_cite":false,"duration_ms":17135,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Combining lyrics and audio embeddings classifies Thai Lukthung songs with F1 0.86.","keywords":["Lukthung classification","Thai music genre","lyrics classification","audio classification","spectrogram CNN","bag-of-words","feature fusion","music information retrieval"],"falsifier":"Compute the same F1 on an artist-disjoint split, where all songs by one artist go into the same partition; a large drop would show that the 0.86 F1 is inflated by artist-specific cues rather than genre-level recognition.","tokens_in":7803,"feed_emoji":"🎵","tokens_out":4624,"duration_ms":46927,"temperature":0.7,"pith_summary":"This paper asks whether the Thai genre Lukthung can be reliably separated from other Thai popular music using only lyrics, only audio, or both. It claims that a simple bag-of-words lyric representation already outperforms standard audio-only classifiers, that a spectrogram CNN using specially shaped filters captures the timbral and rhythmic signature of Lukthung, and that the best result comes from concatenating the learned features of both models into a final classifier, reaching an F1 of about 0.86. The motivation is practical: streaming services that recommend by genre can use the model to avoid recommending Lukthung to listeners who never hear it. The result matters because it is a demonstration that genre classification can work on a non-Western, under-resourced music style using a relatively small dataset.","feed_headline":"Fusing lyrics and audio classifies Lukthung at F1 0.86","feed_subtitle":"A two-stream neural model combines bag-of-words lyrics and spectrogram features, beating separate audio and lyrics classifiers for Thai…","key_machinery":"The machinery is a two-stream feature-extraction architecture: a bag-of-words MLP for lyrics and a spectrogram CNN with parallel vertical and horizontal filters for audio, whose last-layer representations are concatenated and fed to a small feed-forward classifier. The vertical filters are designed to capture timbral features such as overtone stacks and vibrato, and the horizontal filters capture tempo and rhythm; fusing these learned representations is what carries the combined model's improvement.","core_discovery":"The paper reports that classifying Lukthung from other Thai genres is best done by combining lyrics and audio: the combined model reaches an F1 of 0.8561, beating the lyrics-only BoW-MLP (0.7905) and the audio-only Spectro-CNN (0.7730), and far exceeding classical MFCC baselines, including a random forest that achieves high precision but only 0.1313 recall. The authors also find that the learned features separate the classes in t-SNE, and that false negatives are mainly modern Lukthung songs resembling Pop or Rock and songs whose labels appear incorrect, while false positives are nearby genres such as Life and Lukkrung or non-Lukthung songs sung by Lukthung artists.","pith_inferences":["The paper removes artist names from the lyrics but still evaluates on a random song-level split; a fair test of genre-level generalization would hold out whole artists, and without that split the 0.86 F1 could partially reflect artist identity rather than genre.","The same two-stream fusion could be tested on neighboring Thai genres such as Lukkrung and Mor-lam to see whether the learned features distinguish among country-like genres or only separate Lukthung from the rest.","Because the audio model was adapted from an architecture designed for western music tagging, its filter shapes may be implicitly tuned to the spectrogram statistics of that music; retuning them for Thai vocal vibrato could be a direct next step.","The paper's claim about complementary lyrics and audio features could be tested adversarially by ablating each stream in the combined model and measuring how much the F1 drops."],"forward_implications":["If the reported F1 holds, Thai streaming services can automatically tag Lukthung songs in large catalogs and exclude them from recommendations for non-Lukthung listeners.","The lyrics-only bag-of-words model demonstrates that a small dataset and simple text features can compete with deeper lyric models, avoiding the need for large pretrained word embeddings.","The error analysis suggests that isolating singing voice and modeling vocal accent or vibrato could improve classification of modern Lukthung songs that otherwise resemble Pop or Rock.","The learned feature representations could be reused to study how Lukthung changes across eras, since the features separate songs by genre and reveal boundary cases.","The combined architecture is a template for genre classification in other regional or under-resourced music traditions where both lyrics and audio are available."],"supporting_citations":[{"why":"Supplies the spectrogram-CNN architecture with vertical and horizontal filters that the audio model adapts to Thai timbral and temporal features.","marker":"[5]"},{"why":"Provides the lyrics-feature categorization and lyrics classification baseline that motivates the simpler bag-of-words approach.","marker":"[7]"},{"why":"Shows that ensembles of audio and lyrics features can improve genre classification, grounding the combined-model design.","marker":"[8]"},{"why":"Extends the audio-plus-lyrics combination idea to another regional music genre, supporting the paper's transfer to Lukthung.","marker":"[9]"},{"why":"Represents the large-corpus hierarchical attention approach that the paper cites as impractical for its smaller dataset.","marker":"[17]"},{"why":"Supplies the residual network blocks used in the final classification module of the audio model.","marker":"[18]"},{"why":"Defines the hand-crafted feature and classifier baseline that the proposed neural models are compared against.","marker":"[3]"}],"fun_headline_variants":["Fusing lyrics and audio hits F1 0.86 on Lukthung","Lyric-audio model beats single-modality Lukthung classifiers","Two-stream lyrics+audio classifier achieves 0.86 F1 on Lukthung","Combined lyrics+audio achieves F1 0.86 for Lukthung"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the random song-level split into training and test sets measures genre-level generalization, even though no artist-disjoint split is reported and artists may appear in both sets.","fun_headline_variants_meta":{"raw":{"variants":["Fusing lyrics and audio hits F1 0.86 on Lukthung","Lyric-audio model beats single-modality Lukthung classifiers","Two-stream lyrics+audio classifier achieves 0.86 F1 on Lukthung","Combined lyrics+audio achieves F1 0.86 for Lukthung"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001229,"raw_usage":{"total_tokens":5037,"prompt_tokens":916,"completion_tokens":4121,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":4036}},"tokens_in":532,"tokens_out":4121,"duration_ms":28492,"temperature":1.0,"reasoning_tokens":4036,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:29:23.227200+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the same F1 on an artist-disjoint split, where all songs by one artist go into the same partition; a large drop would show that the 0.86 F1 is inflated by artist-specific cues rather than genre-level recognition.","supporting_citations":[{"cited_title":"Lyrics-based analysis and classification of music,","cited_arxiv_id":null,"evidence_quote":"Provides the lyrics-feature categorization and lyrics classification baseline that motivates the simpler bag-of-words approach."},{"cited_title":"Music genre classification by ensembles of audio and lyrics features,","cited_arxiv_id":null,"evidence_quote":"Shows that ensembles of audio and lyrics features can improve genre classification, grounding the combined-model design."},{"cited_title":"Turkish music genre classification using audio and lyrics features,","cited_arxiv_id":null,"evidence_quote":"Extends the audio-plus-lyrics combination idea to another regional music genre, supporting the paper's transfer to Lukthung."},{"cited_title":"Lyrics-based music genre classification using a hier- archical attention network,","cited_arxiv_id":null,"evidence_quote":"Represents the large-corpus hierarchical attention approach that the paper cites as impractical for its smaller dataset."},{"cited_title":"Musical genre classification of audio signals,","cited_arxiv_id":null,"evidence_quote":"Defines the hand-crafted feature and classifier baseline that the proposed neural models are compared against."}],"review_version":1}