{"id":"d1c7474c-df44-4fa6-aa72-259d84068e34","arxiv_id":"1908.07819","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A script-only model with attention, genre, and emotion features predicts MPAA ratings with a weighted F1 of 78.03%, beating a threshold baseline, an SVM, and a CNN.","lead":"The authors trained a neural network to read movie scripts and predict the MPAA age rating (G, PG, PG-13, R, NC-17), reaching a weighted F1 score of about 78%. It is one of the first attempts to automate film suitability ratings from text alone, and it comes with a new dataset of roughly 7,000 movies.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 78.03% weighted F1 is driven by the majority R class; the model has zero recall on G and NC-17, so the central suitability claim rests on an unvalidated metric.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing concern: the evaluation metric hides complete failure on the two safety-critical classes. The paper's own error analysis (Section 7, Figure 4) confirms zero correct predictions for G and NC-17, and the class counts in Table 1 show that weighted F1 is dominated by R and PG-13. This is not a dispute about the arithmetic of the reported score; it is a question of whether the score supports the central claim about predicting suitability for children. The concern is concrete and testable: reporting per-class metrics, macro-F1, and repeated CV would settle whether the zero-recall pattern is a small-sample artifact or a systematic limitation. I agree with the reader and would keep the CONDITIONAL verdict; the numerical result may stand for the majority classes, but the suitability claim should not be accepted without the additional evaluation. The absence of released code and data is a secondary reproducibility issue, not the main load-bearing problem.","tokens_in":9679,"tokens_out":5153,"duration_ms":51845,"concrete_test":"Run the best model under repeated stratified 10-fold cross-validation with multiple seeds and report per-class precision, recall, and F1 for G and NC-17, plus macro-F1 with 95% confidence intervals. If per-class recall for G and NC-17 remains at or near 0, or if macro-F1 is not significantly above the SVM baseline, the 78.03% weighted F1 does not support the paper's suitability-prediction claim for the extreme classes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result (Table 3) is a weighted F1 of 78.03%, but the dataset is severely imbalanced (Table 1: G=92, NC-17=14 out of 7,040 movies), and the confusion matrix in Figure 4 shows zero true positives for both G and NC-17. Weighted F1 gives these two classes almost no influence on the score, so the 78.03% figure mostly reflects performance on R and PG-13. The stated purpose of the work is to predict children's suitability, for which the extreme classes are exactly the most decision-relevant: G means all ages admitted and NC-17 means no one under 17 admitted. The paper itself acknowledges in Section 7 that 'our model is not able to correctly predict any instances of G and NC-17' and attributes this to the low number of instances, but it provides no repeated splits, confidence intervals, or per-class metrics to distinguish a small-sample artifact from a systematic inability to detect child-appropriate or adult-only content. Without such evidence, the 78.03% weighted F1 does not establish the central suitability-prediction claim, even though it may be a valid benchmark result for the majority classes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new task, automatic prediction of MPAA ratings from movie scripts, together with a corpus of about 7K movies (Table 1) and a neural architecture. The proposed model is an LSTM with attention over dialogue, combined with a movie-level NRC emotion vector and a multi-hot genre vector (Section 4). On a single stratified 80/10/10 split, the full model achieves a weighted F1 of 78.03%, which is 6.1 points above an SVM baseline and 3.29 points above a CNN baseline (Table 3). The paper also presents threshold, SVM, and CNN baselines, per-genre results, an emotion analysis, an attention-weight analysis, and a discussion of MPAA ratings' practical effect on under-18 audiences.","tokens_in":9953,"tokens_out":3488,"duration_ms":32207,"significance":"The dataset release and the systematic comparison of threshold, SVM, CNN, and LSTM-attention models are genuinely useful first steps for a task that appears not to have been studied before. The ablation results (Table 3) give credible evidence that emotion and genre features each contribute to weighted F1 on this corpus, and the reported 78.03% is internally consistent with the described pipeline. However, the significance of the headline claim is currently limited by the evaluation metric: because the test set contains 4,030 R-rated movies out of 7,040, weighted F1 is dominated by the majority classes, and the model identifies no G or NC-17 movies correctly (Figure 4). For a system whose stated purpose is predicting children's suitability, that is not a mere presentation issue; it affects whether the 78.03% figure supports the central claim. With per-class results, repeated splits, or a reframed claim about predicting restricted ratings, the contribution could become solid.","major_comments":[{"comment":"The model achieves zero true-positive predictions for G and NC-17, as the confusion matrix in Figure 4 shows and as the paper acknowledges in Section 7, while weighted F1 (78.03%) is dominated by R and PG-13 because of the class distribution in Table 1. Since the paper's stated goal is to predict suitability for children, the G and NC-17 classes are the most decision-relevant ones: G is the only rating with unrestricted admission for all ages, and NC-17 is the only rating that excludes all viewers under 17. The single weighted-F1 number therefore does not, by itself, support the suitability-prediction claim; the authors should report per-class precision/recall/F1 and a macro-F1 or balanced evaluation, and temper the abstract and conclusion accordingly.","section":"§7, Fig. 4; §3, Table 1; §6, Table 3"},{"comment":"All results are based on one stratified split, with no repeated runs, confidence intervals, or significance tests; the gap between the best model and the CNN baseline is 3.29 F1 points, which could plausibly be within run-to-run variance for LSTM training. The authors should report results over multiple seeds (and ideally multiple splits) with standard deviations and a paired significance test, or at least show that the ranking of models is stable across restarts.","section":"§5–§6, Table 3"},{"comment":"The development set is used both for hyperparameter selection (learning rate, LSTM hidden units, dropout) and for selecting the 'best weighted-F1' model over 200 iterations, with no statement of how many configurations were tried; this makes the validation-based model selection part of the reported test result. The paper should state the number of configurations, confirm that the test set was used only once, and ideally include a small study of sensitivity to the reported hyperparameters.","section":"§5.2"}],"minor_comments":[{"comment":"The phrase 'spatial dependency of the words' should be 'sequential dependency' for an LSTM.","section":"§4.2"},{"comment":"In the threshold baseline description, 'NRC-17' should be 'NC-17'.","section":"§5.1"},{"comment":"The text says 'MPPA rating' in the paragraph after Table 4; should be 'MPAA'.","section":"§6"},{"comment":"The term 'miss-classified' should be 'misclassified' throughout Section 7.","section":"§7.1"},{"comment":"'It should bee noted' contains a typo; should be 'be noted'.","section":"§3"},{"comment":"The column header 'Rate' is ambiguous; use 'Rating' and format the example sentences so their source rating is clear.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the paper overstates the suitability-prediction claim relative to what the evaluation shows. The strengths are the new dataset and the baseline comparisons; I think the manuscript can be revised to a solid contribution by reframing the claim or adding the missing per-class and statistical analysis. No concerns about citation or scope beyond the metric issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is the first paper to frame MPAA rating prediction from movie scripts as a task, and it contributes a ~7k-script dataset with ratings. Second: the 78.03% weighted F1 headline is mostly a measure of how well the model predicts R-rated movies, because the test set is dominated by R (4,030 out of 7,040) and the model predicts zero correct for G and NC-17. The authors show this in the confusion matrix and admit it, so there is no deception, but the central claim that they are predicting children's suitability is undercut by total failure on the two classes that matter most for that claim.\n\nWhat's genuinely good: the dataset is new and, once released, will be a resource. The task is well motivated. The ablation design (adding genre and emotion to a plain LSTM with attention) is clean and shows both features help. The error analysis is more honest than usual—they examine attention weights, bad word ratios, and emotion patterns, and they avoid overclaiming in the text. The baseline set includes a threshold model, SVM, and CNN, which is fair. The related work is appropriate.\n\nThe soft spots are real. The evaluation is a single stratified split with no repeated runs, no confidence intervals, and no significance tests. The weighting of F1 masks the extreme classes, and the paper offers only \"low instance count\" as the explanation. That is plausible, but per-class precision/recall and at least a few random seeds would be needed to know whether the failure is a small-sample artifact or a systematic inability to detect G or NC-17 content. The paper also only promises data availability; no code or data is shipped. Minor point: the threshold baseline with a hand-compiled bad-word list is a strawman, though it serves the rhetorical purpose of showing bad words alone don't suffice.\n\nOverall, I'd send this to peer review. The dataset and task are worth publishing, but the evaluation needs to be strengthened before the suitability claim can be taken at face value. If I were working on content moderation or movie analytics, I'd want to cite the dataset. I'd bring it to a reading group as a case study in metric choice on imbalanced data.","headline":"New dataset and task for MPAA rating prediction from scripts, but the 78% weighted F1 hides zero correct predictions on G and NC-17, undercutting the suitability claim.","tokens_in":10466,"tokens_out":2408,"would_cite":true,"duration_ms":22462,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A neural model reads movie scripts and predicts the MPAA rating with 78 percent weighted F1, beating an SVM baseline by 6.1 percent.","keywords":["MPAA rating prediction","movie scripts","text classification","RNN with attention","emotion analysis","genre modeling","content suitability","parental guidance"],"falsifier":"Run the same architecture on a class-balanced test set or report per-class recall for G and NC-17; if the model cannot identify a single all-ages or adult-only film even when those classes are oversampled, the claim that it predicts suitability collapses. Alternatively, have parents rate a held-out set of unrated scripts and compare their judgments to the model's output; if script-based predictions disagree with human suitability judgments more often than human raters disagree with each other, the task framing fails.","tokens_in":9509,"feed_emoji":"🎬","tokens_out":4555,"duration_ms":43565,"temperature":0.7,"pith_summary":"Can a movie's age-suitability rating be predicted before production, just from its script? This paper argues yes: a recurrent network with attention that reads character dialogue and folds in the movie's genre and emotional tone assigns MPAA ratings (G, PG, PG-13, R, NC-17) with a weighted F1 of 78.03 percent, beating an SVM baseline by 6.1 percent. The authors contribute the first script corpus paired with MPAA ratings, about 7,000 movies across 24 genres, and show that context-sensitive modeling beats counting bad words. If the approach holds, parents could screen unrated films, streaming services could auto-apply age filters, and producers could adjust content before costly post-production changes. The paper also reports that its model predicts zero G and zero NC-17 films correctly, so the headline score does not yet translate into a complete suitability checker.","feed_headline":"Scripts alone predict MPAA ratings at 78 percent F1","feed_subtitle":"An RNN with attention plus emotion and genre cues beats bad-word lists and SVM baselines, but misses G and NC-17 films.","key_machinery":"The load-bearing mechanism is an attention-augmented recurrent reader: a 300-dimensional GloVe embedding layer feeds an LSTM whose hidden states are combined by Bahdanau-style attention, so the model learns which words in a conversation matter for the rating. Two context vectors are concatenated to the attention output before the dense and softmax layers: a normalized emotion vector (the per-movie percentage of words in each of eight NRC emotion categories plus two sentiments) and a 24-way multi-hot genre vector. The emotion vector supplies the dominant affective tone; the genre vector supplies the interpretive frame that decides whether, for example, a violent phrase is routine in an action movie or alarming in an animated family film. The ablation comparisons (L&A, +genre, +emotion, +emotion+genre) are what attribute the measured gain to each signal.","core_discovery":"The central claim is that the MPAA rating of a movie is predictable from the movie's script alone, provided the model sees not just words but also the genre and the emotional content of conversations. Jointly modeling genre and emotion improves a plain LSTM-with-attention reader: emotion alone adds about 1.9 points and genre alone about 1.0 point of weighted F1, and their combination reaches 78.03 percent, outperforming the threshold-based bad-word baseline by 14.13 percent, an SVM with lexical and sentiment features by 6.1 percent, and a CNN analog by 3.29 percent. The system uses 300-dimensional GloVe embeddings, an LSTM layer, Bahdanau attention, a normalized emotion vector built from the NRC lexicon, and a 24-way multi-hot genre vector before a softmax over five ratings. The error analysis is interpreted as evidence that context, not raw profanity counts, drives suitability distinctions: R-rated 'disgust' words denote violent acts while PG-rated ones denote milder discomfort. However, the confusion matrix shows no correct predictions for G or NC-17, and most R-rated errors fall into the adjacent PG-13 class.","pith_inferences":["A practical parental-control deployment would need the model to recover G and NC-17, which it currently never does; this implies either cost-sensitive training, oversampling, or a two-tier cascade rather than the reported five-way classifier.","Because the emotion signal is computed from a lexicon's word counts rather than a learned representation, the reported emotion gains may understate what a trained affect model could achieve; testing a supervised emotion encoder on the same architecture is a natural next experiment.","The paper's under-18-audience analysis suggests MPAA ratings shape viewing behavior; if automated ratings become widespread, they could shift what children watch, making the reliability of extreme-class predictions a policy matter rather than just a benchmark number.","The 6-point gap over the SVM is measured on a single split; an adversarial check would compare against a stronger contextual encoder to see whether the recurrent-attention advantage survives."],"forward_implications":["If script-only prediction is reliable, producers can estimate a film's rating before shooting wraps and adjust content to hit a target audience.","Streaming services and parental-control tools can assign provisional age filters to the large back-catalog of unrated online films.","Emotion and genre features are measurable gains, indicating that suitability is an affective-context phenomenon rather than a profanity-counting problem.","The same architecture transfers as a baseline to other content-suitability axes, such as violence intensity or substance-use depictions.","The error profile suggests that for practical use, a two-stage design (first separate R and PG-13 from family-friendly films, then refine) may serve better than five-way classification."],"supporting_citations":[{"why":"Supplies the movie-script corpus (about 15K scripts with metadata) that the paper extends with crawled MPAA ratings to form its 7,040-movie dataset.","marker":"(Shafaei et al., 2019)"},{"why":"Provides the NRC emotion lexicon from which the per-movie normalized emotion vectors (eight emotions plus two sentiments) are computed.","marker":"(Mohammad, 2011)"},{"why":"Supplies the attention mechanism used to weight LSTM hidden states and aggregate them into a single contextual vector.","marker":"(Bahdanau et al., 2014)"},{"why":"The closest prior work, predicting violence from movie scripts; the paper uses its finding that genre affects how dialogue is interpreted to justify the genre vector.","marker":"(Martinez et al., 2019)"},{"why":"Establishes that swearing impact depends on communication context, motivating the paper's context-sensitive architecture over bad-word lists.","marker":"(Jay and Janschewitz, 2008)"},{"why":"Supports the claim that MPAA ratings combine multiple content aspects (violence being only one), justifying prediction beyond violence detection.","marker":"(Jenkins et al., 2005)"}],"fun_headline_variants":["MPAA rating predicted from script alone at 78% F1","Scripts + emotion + genre predict movie ratings 78% F1","Joint emotion and genre models beat bad-word lists for MPAA ratings","RNN with attention predicts MPAA ratings, but misses G and NC-17","Context beats profanity counts for MPAA rating prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single weighted-F1 score on an imbalanced test split measures whether a model can actually judge children's suitability; with 4,030 of 7,040 movies rated R and zero correct predictions for G and NC-17, the 78 percent score says little about the extremes that matter most for child safety.","fun_headline_variants_meta":{"raw":{"variants":["MPAA rating predicted from script alone at 78% F1","Scripts + emotion + genre predict movie ratings 78% F1","Joint emotion and genre models beat bad-word lists for MPAA ratings","RNN with attention predicts MPAA ratings, but misses G and NC-17","Context beats profanity counts for MPAA rating prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000876,"raw_usage":{"total_tokens":3774,"prompt_tokens":914,"completion_tokens":2860,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":2768}},"tokens_in":530,"tokens_out":2860,"duration_ms":19528,"temperature":1.0,"reasoning_tokens":2768,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:53:22.270373+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same architecture on a class-balanced test set or report per-class recall for G and NC-17; if the model cannot identify a single all-ages or adult-only film even when those classes are oversampled, the claim that it predicts suitability collapses. Alternatively, have parents rate a held-out set of unrated scripts and compare their judgments to the model's output; if script-based predictions disagree with human suitability judgments more often than human raters disagree with each other, the task framing fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the movie-script corpus (about 15K scripts with metadata) that the paper extends with crawled MPAA ratings to form its 7,040-movie dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The closest prior work, predicting violence from movie scripts; the paper uses its finding that genre affects how dialogue is interpreted to justify the genre vector."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that swearing impact depends on communication context, motivating the paper's context-sensitive architecture over bad-word lists."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the claim that MPAA ratings combine multiple content aspects (violence being only one), justifying prediction beyond violence detection."}],"review_version":1}