{"id":"c954d7e3-0ea1-4a60-ae24-810efa71cb90","arxiv_id":"2507.03875","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A benchmark reports that ChatGPT outperforms other LLMs and traditional classifiers on multi-label movie genre prediction, but the result is undermined by likely pretraining contamination and weak baselines.","lead":"This paper tests whether ChatGPT and other large language models can predict a movie's genres from its trailer subtitles and poster, using the MovieLens-100K dataset. It reports that ChatGPT beats other models and that fine-tuning helps, but the evaluation does not address whether the model is simply recalling movies it already knows.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot scores may reflect memorized movie-genre associations rather than subtitle-based inference; Section VI's own admission that MovieLens labels are incomplete compounds this, so 'masters genre recognition' is not established.","rationale":"I agree with the reader's identified weakest assumption: the evaluation does not establish that ChatGPT is inferring genres from trailer subtitles, because the movies are famous, the labels are noisy, and no contamination control exists. This is the single most load-bearing concern because it undermines the paper's main contribution—the zero-shot and few-shot F1 tables and the comparative claim that ChatGPT 'outperformed other LLMs.' If the model is retrieving known movie/genre associations, the numbers may still describe predictive accuracy on MovieLens, but they do not describe genre recognition from subtitles, and the title's promise of 'how it masters genre recognition' is unsupported. The concern is not merely theoretical: the paper itself, in Section VI, provides evidence that the ground-truth labels are incomplete or wrong, which directly affects every metric in Tables I-VI. The VLM integration claim is also weak (Table VI shows macro F1 unchanged at 0.64 and micro F1 only +0.01), but that is secondary to the contamination/label-validity issue because even the base LLM comparison is not yet interpretable. A single well-designed unseen-movie or entity-scrubbed control would settle the issue, but without it the central claim is not established. The correct verdict is therefore REJECT, consistent with the reader's moderate-confidence rejection.","tokens_in":12665,"tokens_out":5530,"duration_ms":68314,"concrete_test":"Build a validation set of roughly 100 films released after gpt-3.5-turbo's training cutoff (or otherwise absent from its pretraining), with trailer transcripts and independently human-verified genre labels from a source not used in training. Rerun the exact zero-shot prompt from Section III-A with the same temperature, parsing, and genre list, and compute macro-F1. If macro-F1 on this never-seen set falls to near the majority-class baseline or markedly below the reported 0.64, the headline zero-shot performance is attributable to memorized movie-genre associations rather than subtitle-based genre inference. A complementary check within the original data is to rerun the prompt on subtitles with all proper nouns, character names, and title references replaced by placeholders; a large F1 drop would corroborate reliance on film identity rather than on genre-relevant textual content.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim that ChatGPT 'masters genre recognition' from trailer subtitles rests on treating MovieLens-100K F1 scores as direct measures of subtitle-based inference. That requires at least two conditions: (i) the model cannot simply retrieve the known movie-to-genre mapping from pretraining, and (ii) the MovieLens labels used as ground truth are correct. Neither is established. The 1,682 movies are well-known and predate GPT-3.5's training data; trailer subtitles frequently contain dialogue, character names, and other cues that identify the specific film. If the model recognizes 'Toy Story' from its transcript and recalls its listed genres, then the zero-shot F1 values in Tables I and II (e.g., micro 0.67, macro 0.64) do not measure genre recognition from language at all. The paper provides no control for this: no held-out or post-cutoff movies, no scrambled or entity-scrubbed subtitles, and no analysis of which input features drove predictions. Section VI then explicitly admits that MovieLens-100K genre labels are 'limited or, in some instances, inappropriate,' giving the example that Adventure is a valid genre for Toy Story but is counted as a false positive. Since the same labels are used in every table, the reported precision, recall, and F1 numbers are partly artifacts of label noise, and the relative comparison between ChatGPT and the davinci models is not necessarily a comparison of genre-recognition skill. This is a load-bearing construct-validity problem: the headline capability, and the claimed zero-shot superiority, would collapse if the model is largely recalling metadata or if the ground truth is unreliable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks three LLMs (text-davinci-002, text-davinci-003, and gpt-3.5-turbo, plus a fine-tuned gpt-3.5-turbo) on multi-label movie genre prediction using the MovieLens-100K dataset. Trailer subtitles are used in zero-shot and few-shot prompts, SBERT-based classifiers are compared, a cost analysis is performed, and LLaVA-extracted poster descriptions are added to the prompts. The central claims are that ChatGPT outperforms the other LLMs, that fine-tuning improves performance, and that VLM information further enhances genre prediction.","tokens_in":12981,"tokens_out":5032,"duration_ms":56234,"significance":"If the results were valid, the paper would offer a useful comparison of LLMs on a practical content-understanding task, together with a rare cost-benefit analysis. The authors should be credited for extending the dataset with trailer subtitles and posters, for evaluating multiple prompting and fine-tuning settings, and for candidly acknowledging label noise in the case study. However, the evaluation lacks any control for pretraining contamination, relies on ground-truth labels that the paper itself describes as incomplete or inappropriate, and the VLM integration results are far weaker than the abstract suggests. The central capability claim is therefore not established by the current experiments.","major_comments":[{"comment":"The zero-shot and few-shot F1 scores cannot be interpreted as measuring genre recognition from subtitles. The 1,682 movies predate GPT-3.5's training data, and trailer subtitles often contain dialogue, character names, and other identifying content, so the model may be retrieving known movie-to-genre associations from pretraining rather than reasoning from the provided text. The paper provides no control: there are no post-cutoff movies, no entity-scrubbed or scrambled subtitles, and no analysis of which input features drove predictions. This also affects RQ2's comparison with SBERT-based classifiers, which have no such prior knowledge. A concrete fix would be to evaluate on movies released after the model's training cutoff or on subtitles with identifying named entities removed.","section":"Section IV, Tables I-II and IV.C"},{"comment":"The ground-truth labels are admitted to be 'limited or, in some instances, inappropriate,' yet these same labels are used to compute every precision, recall, and F1 score in the paper. The Toy Story example shows that a prediction can be correct in reality but counted as a false positive against MovieLens labels. This is not merely a VLM-specific issue; it undermines the absolute scores and the model rankings throughout the paper. The authors should either correct and release a revised label set or report results separately on labels verified against external sources.","section":"Section VI, Tables I-VI"},{"comment":"The claim that VLM integration 'further enhances' genre prediction is not supported by the reported numbers. Table VI shows micro, weighted, and samples F1 improvements of only 0.01-0.02, while macro F1 is unchanged at 0.64, and Table V shows precision declining on most individual genres. Section V states that 'F1-score has improved in all the average metrics,' which is contradicted by the unchanged macro average. The abstract should be revised to reflect the marginal and mixed nature of the VLM contribution.","section":"Section V, Tables V-VI and Abstract"},{"comment":"The fine-tuning split is arithmetically inconsistent with the stated dataset size. The paper says MovieLens-100K contains 1,682 movies and then reports an 80:20 split as 1,205 training and 302 test movies, but these two numbers sum to 1,507. The discrepancy of 175 movies is unexplained and matters because every subsequent result depends on which movies were used for training and testing.","section":"Section IV.A.1"}],"minor_comments":[{"comment":"'few show settings' should be 'few-shot settings'.","section":"Section III.A.2"},{"comment":"The percentage annotation '0.85⇓⇓15%)15%)' is malformed and should be fixed.","section":"Table V, Animation row"},{"comment":"'Get Shortly' should be 'Get Shorty'.","section":"Section VI"},{"comment":"'idential' should be 'identical'.","section":"Section IV.B"},{"comment":"The sentence 'Detailed descriptions of these LLMs can be found here' contains no link or citation; please add a proper reference.","section":"Section III.B"},{"comment":"The cost analysis is hard to follow because 'Cost chatgpt', 'ScalingF actor', and the scaling factor X are not clearly defined; please define X and the training-cost multiplier explicitly.","section":"Section IV.D, Eq. (4)"},{"comment":"The paper alternates between 'MovieLens' and 'Movielens'; please standardize the spelling.","section":"Entire manuscript"},{"comment":"No code or data release is mentioned; providing the prompts, subtitle texts, and poster descriptions would substantially improve reproducibility.","section":"Entire manuscript"}],"recommendation":"reject","confidential_remarks":"The paper's core empirical claim is undercut by the absence of contamination controls and by reliance on labels that the authors themselves acknowledge to be noisy. The VLM result is internally inconsistent with the abstract. Because the contribution is an empirical benchmark without code or data release, the issues cannot be resolved by local edits; the evaluation design would need to be reworked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a decent applied benchmark with a useful dataset extension, but the title and abstract oversell it. The paper adds trailer subtitles and IMDb posters to MovieLens-100K and runs zero-shot, few-shot, and fine-tuned LLM genre classification. That is genuinely useful for people building content-tagging pipelines. The internal numbers are consistent, the cost analysis is sensible, and Section VI deserves credit for admitting the MovieLens labels are incomplete—though that admission cuts against their own evaluation.\n\nThe load-bearing problem is construct validity. The zero-shot F1 scores are computed against MovieLens labels for 1,682 well-known movies, and gpt-3.5-turbo was almost certainly trained on web data that includes movie metadata. Trailer subtitles often contain character names and dialogue that identify the film, so the model may be recalling the movie-to-genre mapping rather than inferring genres from the text. There is no control: no post-cutoff movies, no entity-scrubbed or shuffled subtitles, no analysis of which cues drove predictions. So the headline \"masters genre recognition\" is not established. At best this is \"ChatGPT can tag known movies from subtitles when it has likely seen them before.\"\n\nSecond issue: the VLM integration is described as enhancing findings, but Table VI shows macro F1 unchanged at 0.64 and precision dropping on most genres. That is a null-to-marginal result, and the abstract and conclusion present it as a win. The paper's own Section VI partially explains why precision drops—labels are wrong—but that same label noise undermines every F1 table.\n\nThe traditional baselines are weak (SBERT embeddings into KNN/logistic regression/SVM), so \"ChatGPT beats traditional classifiers\" is not a strong claim. The few-shot results are plausible, and the cost analysis is fine.\n\nWho is this for? Practitioners building recommendation or content tagging features might get a useful benchmark and a dataset seed. Anyone reading it as evidence about LLM capabilities should be careful. With a contamination-control experiment and honest treatment of the VLM result, this could be a solid applied paper. As is, it is an okay workshop-level benchmark with an overstated headline.\n\nMy recommendation: don't desk reject out of hand—it deserves a serious referee who can push for controls—but the central claim needs to be downgraded or re-argued.","headline":"A useful applied benchmark and dataset extension, but the 'masters genre recognition' claim fails because the evaluation cannot rule out memorization of well-known movie metadata.","tokens_in":13523,"tokens_out":2339,"would_cite":false,"duration_ms":25339,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that ChatGPT, without fine-tuning, outperforms other large language models at multi-label movie genre prediction from trailer subtitles, and that fine-tuning it yields the best overall performance.","keywords":["Genre prediction","Large language models","Multi-label classification","MovieLens-100K","Zero-shot prompting","Few-shot prompting","Vision-language model","Trailer subtitles"],"falsifier":"Take a set of movies whose trailers were released after ChatGPT's training cutoff, run the same zero-shot prompts, and check whether the macro F1 stays near the reported 0.64; if it falls to chance, the reported scores largely reflect memorized genre knowledge rather than subtitle-based recognition.","tokens_in":12486,"feed_emoji":"🎬","tokens_out":5693,"duration_ms":53779,"temperature":0.7,"pith_summary":"This paper tests whether ChatGPT can predict a movie's genres from the subtitles of its trailer, using the MovieLens-100K dataset and its 18 genre labels. The authors report that ChatGPT (gpt-3.5-turbo), with no fine-tuning, outperforms two earlier text-davinci models, and that fine-tuning it pushes F1 higher still. They also show that a vision-language model reading the movie poster can lift recall for visually suggestive genres like action and war, though precision often falls. The result matters because it suggests general-purpose LLMs can take over content-classification work that previously required task-specific training.","feed_headline":"Untuned ChatGPT beats rival LLMs at movie genre guessing","feed_subtitle":"Fine-tuning lifts F1 higher; poster cues help recall, not precision.","key_machinery":"The central objects are the carefully constructed prompts: a zero-shot prompt that gives the model the closed genre set G and a trailer subtitle S; a few-shot prompt that adds two example subtitle-to-genre pairs; and a fine-tuning instruction in a conversational chat format. The VLM extension uses llava to convert a movie poster into descriptive text (title, colors, imagery, actors, taglines) that is appended to the subtitle prompt. These prompts convert genre prediction into a constrained text-generation problem, and the paper's argument rides on the model's ability to map subtitle language to genre labels without task-specific training.","core_discovery":"On the MovieLens-100K benchmark, a multi-label genre prediction task with 18 genres and up to several genres per movie, the paper claims that ChatGPT without any fine-tuning is the strongest zero-shot predictor among the tested LLMs, beating text-davinci-002 and text-davinci-003 on all 54 metric instances (precision, recall, F1 across 18 genres), and that fine-tuned ChatGPT performs best overall. Under few-shot prompting with two examples, fine-tuned ChatGPT again leads, and the optimal number of shots is two. The paper further claims that a VLM (llava) that reads IMDb posters and adds poster-derived text to the subtitle prompt improves recall across average metrics while reducing precision in most genres, and that all tested LLMs exceed traditional classifiers (KNN, logistic regression, SVM) trained on SBERT subtitle embeddings, even at zero training examples.","pith_inferences":["If the model has seen these well-known movies in pretraining, part of the reported accuracy may reflect memorized genre metadata rather than inference from subtitle content; a test on trailer subtitles from post-cutoff movies would separate the two.","The case study suggests MovieLens genre labels are incomplete — Toy Story's Adventure label is a false positive only because the dataset omits it — so the true precision and recall of all systems may be better than the reported numbers.","The same prompt design could transfer to other content-classification domains (books, music, games) where transcripts or subtitles are available, with visual cues as an optional second channel.","The pattern of recall rising while precision falls when poster text is added suggests posters flag additional plausible genres but also distract from the subtitle signal; a learned fusion weight might capture both signals."],"forward_implications":["Zero-shot ChatGPT can be used as a drop-in genre tagger for movie content, with no training data, at a macro F1 around 0.64.","Fine-tuning ChatGPT on roughly 80 percent of MovieLens-100K raises macro F1 to about 0.81, a 26.5 percent relative improvement, at six times the token training cost.","Few-shot prompting with two examples offers only a 4.6 percent F1 gain over zero-shot while tripling input tokens, so zero-shot is cost-preferred.","Adding VLM poster information improves recall (micro recall rises by 7 percent) but lowers precision in most genres; fine-tuning the VLM could yield more task-specific gains.","LLMs outperform traditional classifiers trained on SBERT embeddings, maintaining decent performance even with no training examples."],"supporting_citations":[{"why":"Supplies the SBERT embeddings that feed the traditional classifiers compared against the LLMs.","marker":"[27]"},{"why":"Supplies the llava vision-language model used to extract poster information for the VLM+LLM integration.","marker":"[22]"},{"why":"Shows that the number of prompt examples affects in-context learning, framing the analysis of shot count.","marker":"[28]"},{"why":"Connects prompting LLMs to recommender systems, motivating the application context.","marker":"[29]"},{"why":"Defines the KNN classifier used as one traditional baseline.","marker":"[24]"},{"why":"Defines logistic regression, another traditional baseline.","marker":"[25]"},{"why":"Defines the SVM baseline.","marker":"[26]"}],"fun_headline_variants":["ChatGPT tops genre guessing without fine-tuning","Untuned ChatGPT outguesses rival LLMs on genres","Poster visuals boost ChatGPT's recall, not precision","Why ChatGPT crushes genre prediction: no tuning needed","ChatGPT's genre mastery: zero-shot beats tuned rivals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes ChatGPT's genre guesses come from the provided trailer subtitles rather than from genre information the model memorized during pretraining, and that MovieLens-100K genre labels are complete and accurate.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT tops genre guessing without fine-tuning","Untuned ChatGPT outguesses rival LLMs on genres","Poster visuals boost ChatGPT's recall, not precision","Why ChatGPT crushes genre prediction: no tuning needed","ChatGPT's genre mastery: zero-shot beats tuned rivals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1559,"prompt_tokens":917,"completion_tokens":642,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":580}},"tokens_in":533,"tokens_out":642,"duration_ms":6964,"temperature":1.0,"reasoning_tokens":580,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:00:20.302238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of movies whose trailers were released after ChatGPT's training cutoff, run the same zero-shot prompts, and check whether the macro F1 stays near the reported 0.64; if it falls to chance, the reported scores largely reflect memorized genre knowledge rather than subtitle-based recognition.","supporting_citations":[{"cited_title":"Per- sonalized prompt for sequential recommendation,","cited_arxiv_id":null,"evidence_quote":"Shows that the number of prompt examples affects in-context learning, framing the analysis of shot count."},{"cited_title":"A brief review of nearest neighbor algorithm for learning and classification,","cited_arxiv_id":null,"evidence_quote":"Defines the KNN classifier used as one traditional baseline."},{"cited_title":"Logistic regression,","cited_arxiv_id":null,"evidence_quote":"Defines logistic regression, another traditional baseline."},{"cited_title":"What is a support vector machine?","cited_arxiv_id":null,"evidence_quote":"Defines the SVM baseline."}],"review_version":1}