{"id":"5e16f72d-db78-4539-8008-a93e9a1a9ae1","arxiv_id":"1908.09083","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multi-view hierarchical attention model with multiple instance learning predicts movie story tags from synopses plus reviews, and its attention scores extract open-vocabulary tags from reviews without direct supervision.","lead":"This paper trains a neural model to tag movies with story attributes like 'violence' or 'romantic' by reading both the plot synopsis and user reviews. It reports that reviews add about 4% F1 over synopses alone, and the same attention mechanism can mine new descriptive tags from reviews with no extra annotation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 74% human-relevance result for open-vocabulary tags depends on a lenient 'at least one' criterion, while Appendix F states tags count as relevant only with at least two votes; re-analysis could cut the figure to ~42%.","rationale":"The reader's weakest assumption concerns whether attention weights reliably indicate semantic relevance for open-vocabulary extraction. My concern is more specific: the paper's own Appendix F contradicts the main text's reporting of the human-evaluation criterion, and this directly undermines the only quantitative evidence for the open-vocabulary contribution. However, the other central claim, the 4% F1 improvement from using reviews, is supported by the quantitative table and is not affected by this issue. Since the reader's verdict was already CONDITIONAL, and this concern reinforces the conditionality rather than overturning the paper's main contribution, the verdict should remain UNCHANGED. The check is cheap and should be run before relying on Q3.","tokens_in":25694,"tokens_out":4593,"duration_ms":50979,"concrete_test":"Recompute the relevance rate for the 114/141 YC tags in Appendix F using the appendix's stated at-least-two-votes criterion. If the rate is ~42% rather than 74%, the Section 6.1 claim should be revised and the open-vocabulary contribution should be treated as unvalidated at the reported confidence. Additionally, report the number of distinct tags and votes so the calculation is reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central unsupervised contribution (Section 4.2) is the extraction of an open-vocabulary tagset YC from review attention, and the only quantitative support for its validity is the human evaluation in Section 6.1. The main text reports that 24% of YC tags were accepted by all three judges, 18% by two, and 32% by one, and concludes ~74% were relevant 'by at least one judge.' However, Appendix F states the evaluation protocol differently: 'We consider a tag as relevant if it has at least two votes.' Under that criterion, the relevant fraction is 24% + 18% = 42%, not 74%. This is an internal inconsistency in the evidence for the paper's headline claim that reviews yield complementary story attributes without supervision. The discrepancy matters because if the stricter criterion is the intended one, the open-vocabulary extraction is only validated on fewer than half of the extracted tags, and the abstract's claim is correspondingly weaker. The concern is not that the authors are dishonest; it is that the reported 74% figure conflates one-judge agreement with the appendix's stated standard, and the reader cannot determine which criterion was actually used from the paper alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses automatic story characterization by predicting a closed set of 71 story tags from movie plot synopses and user reviews. It extends the MPST corpus with review summaries, proposes a hierarchical attention encoder with multiple-instance learning and gated fusion of synopsis/review representations, and reports that gated fusion achieves 41.84 micro-F1 versus 37.94 for the best synopsis-only system. It further proposes extracting an open-vocabulary tagset from review attention weights, validated by a human evaluation on 21 movies. The dataset and code are released.","tokens_in":25882,"tokens_out":3558,"duration_ms":34775,"significance":"If the results hold, the paper makes a useful empirical contribution: it provides evidence that user reviews contain complementary signal for story tagging, and it sketches a weakly supervised route to open-vocabulary attributes. The supervised comparison is consistent across Table 2, the subset analysis, and the human evaluation, and the dataset/resource contribution is concrete. The qualitative generalization checks for 2019 movies and non-movie narratives are a valuable addition. However, the unsupervised extraction contribution rests on a human evaluation whose reported 74% relevance figure conflicts with the appendix's stated two-vote criterion, and the main review-versus-synopsis gain is reported without significance testing or variance estimates.","major_comments":[{"comment":"Section 6.1 reports that 24% + 18% + 32% = 74% of the open-vocabulary tags YC were 'marked as relevant by at least one judge,' but Appendix F explicitly states the evaluation protocol as 'We consider a tag as relevant if it has at least two votes.' Under the appendix's stated standard, the relevant fraction is 24% + 18% = 42%, not 74%. This is not merely a presentational difference: the 74% figure is the headline quantitative evidence for the unsupervised extraction contribution. The paper must adopt one standard consistently, report both criteria, and adjust the claims in the abstract and Section 6.1 accordingly. Please also provide the per-tag vote distributions underlying Figure 7(b) so the reader can verify both numbers.","section":"Section 6.1 and Appendix F"},{"comment":"The central claim that adding reviews improves tag prediction (37.94 to 41.84 in top-3 micro-F1) is not accompanied by statistical significance testing or variance estimates. The t-test footnote applies only to the comparison among gated fusion, merge-texts, and concatenation; it does not compare the review-based models with the synopsis-only systems. Because the reported gains are small in absolute terms and each configuration appears to be trained once, I ask for either multiple seeds with standard deviations or a significance test for the synopsis-only versus synopsis+reviews comparison.","section":"Section 5, Table 2"},{"comment":"The importance score gamma_ij = alpha_Wij * alpha_si * |si| treats word-level attention as a direct measure of semantic relevance, and the stopping threshold of 5e-3 is set 'based on our observations on the validation set.' This makes the extraction loop partly self-referential: the same attention weights trained for closed-set tag prediction define both the candidate tags and, via validation-based thresholding, their stopping criterion. The human evaluation is an external check, but it only covers 21 movies. To strengthen this contribution, please provide an additional validation of attention-based importance, such as a comparison with held-out review-side annotations or a sensitivity analysis over the threshold.","section":"Section 4.2, Eq. (1)"},{"comment":"The human evaluation uses three judges on 21 movies and 141 tags, but no inter-annotator agreement statistic is reported. Given that the central quantitative claim for YC depends on judge votes, please report agreement (e.g., Fleiss' kappa) and clarify how disagreements were adjudicated.","section":"Section 6.1"}],"minor_comments":[{"comment":"'demonstrate how can we take advantage' should be 'demonstrate how we can take advantage.'","section":"Abstract"},{"comment":"'Figure-3' should be 'Figure 3.'","section":"Section 4.2"},{"comment":"The footnote explaining the t-test should state explicitly which pairwise comparisons were tested; the current single asterisk is ambiguous.","section":"Table 2"},{"comment":"The emoticon legend is useful, but the appendix's relevance criterion is inconsistent with the main text; please align the two or explicitly report both standards.","section":"Appendix F"},{"comment":"The claim that 'most of these tags except neo noir were found in IMDB' is informal; consider reporting the number of matched tags per movie.","section":"Section 6.3, Table 3"},{"comment":"Capitalization of IMDb/IMDB and MPST is inconsistent; please standardize.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the inconsistency between Section 6.1 and Appendix F is the main obstacle. If the intended criterion is the appendix's two-vote standard, the claimed 74% relevance for open-vocabulary tags drops to about 42%, and the paper's unsupervised contribution must be reframed accordingly. The supervised contribution is likely publishable once significance testing or multiple runs are reported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth engaging, but the paper's most distinctive claim—unsupervised open-vocabulary tags from reviews—rests on a human evaluation reported two different ways, and the stricter reading cuts the headline number almost in half.\n\nWhat's genuinely new: the paper combines synopses and reviews in a hierarchical attention model under MIL, with gated fusion, and shows a consistent ~4 micro-F1 gain over synopsis-only systems on MPST. That result is credible: it appears on the full test set, on the subset where every movie has a review, and in the validation table. The attention-based extraction of complementary tags is a clever trick, and the authors do the right thing by checking it with human judges rather than just asserting it works. They also release reviews, code, and data, which makes the work reproducible. Credit where due.\n\nThe soft spots matter. First, the human evaluation inconsistency: Section 6.1 says 24% of tags got three votes, 18% got two, 32% got one, and then calls ~74% \"relevant by at least one judge.\" Appendix F says \"We consider a tag as relevant if it has at least two votes.\" Under the appendix's own standard, relevance drops to 24+18=42%. The stress-test note is right. This is not a minor wording issue; the open-vocabulary extraction is the paper's main novelty, and 42% validated tags is a much weaker story than 74%. The authors need to pick one criterion, justify it, and re-report. The per-tag vote data in the appendix is detailed enough that the reader could redo this themselves, which is to the paper's credit.\n\nSecond, the main F1 gain is not significance-tested against the synopsis-only baseline; the t-test is only among fusion variants. The direction is consistent, but without multiple seeds or error bars, the 4% could be smaller than it looks. Minor-to-moderate.\n\nThird, the extraction threshold (5e-3) is tuned on validation, and attention-as-importance is partly self-referential. The human evaluation is the external check, so this is addressable rather than fatal.\n\nOverall: a solid conditional accept. The central modeling claim (reviews help) holds up; the unsupervised tag claim is promising but overstated as written. Send it to serious peer review—a good referee can get the authors to fix the reporting, run multiple seeds, and re-analyze the human data. Desk rejection would be wrong.","headline":"A useful multi-view model with a real F1 gain and a neat extraction idea, but the paper's headline 74% for open-vocabulary tags is internally inconsistent with the appendix's stricter criterion and should be re-reported.","tokens_in":26476,"tokens_out":2629,"would_cite":false,"duration_ms":27041,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"User reviews lift automatic movie tagging by about 4 F1 points.","keywords":["multi-label tag prediction","movie plot synopses","user reviews","hierarchical attention","multiple instance learning","gated fusion","open-vocabulary tag extraction"],"falsifier":"Run the extraction on a larger held-out sample and ask annotators to mark every review span that names a story attribute; if the model's top-ranked attention words consistently miss those spans, or if relevance rates on the larger sample fall well below the reported 74%, the unsupervised tagset is an artifact of attention rather than a story signal.","tokens_in":25457,"feed_emoji":"🎬","tokens_out":6136,"duration_ms":58656,"temperature":0.7,"pith_summary":"This paper tries to show that user reviews of movies are a genuinely useful second source for automatic story characterization, not just a noisy complement to plot synopses. It builds a multi-view neural model that reads a synopsis and a condensed review summary through separate hierarchical attention encoders, combines them with gated fusion, and predicts a predefined set of 71 story tags. On the test set the multi-view model reaches 41.84 micro-F1 for the top three tags, compared with 37.94 for the authors' best synopsis-only system, and roughly 31-40 reviews are enough to see a dependable gain. The same review encoder then acts as an unsupervised tag miner: ranking words by attention-derived importance produces an open-vocabulary set of story attributes beyond the 71 predefined tags, and human judges rated about 74% of those extracted tags as relevant by at least one vote. A sympathetic reader would take away that user opinions encode story content worth extracting, and that attention weights can stand in for expensive word-level annotation.","feed_headline":"User reviews lift automatic movie tagging by about 4 F1 points","feed_subtitle":"A multi-view model fuses plot synopses with review summaries and mines new story tags with no extra annotation.","key_machinery":"The load-bearing mechanism is a pair of hierarchical attention encoders—BiLSTMs with word-level attention feeding sentence representations, then sentence-level attention to form a document vector—one for the synopsis and one for a text-rank summary of up to 100 user reviews. The two document vectors are merged by gated fusion, $z = \\sigma(W_z[d^{PS}_h,d^R_h])$, with $h = z*h^{PS} + (1-z)*h^R$, and sentence-level tag predictions are aggregated as in multiple instance learning. After training, the review encoder's attention weights define an importance score for each word, $\\gamma_{ij} = \\alpha^R_{W_{ij}} \\alpha^R_{s_i} |s_i|$, and words whose rank curve flattens below a slope threshold become candidate open-vocabulary tags.","core_discovery":"The paper argues that movie reviews carry story attributes that synopses miss, and that a model trained on both can tag a film better than one trained on synopses alone. On the test set, its gated-fusion multi-view model reaches 41.84 micro-F1 (top-3 tags) using synopses and reviews, against 37.94 for the best synopsis-only variant in the same paper. The same trained review encoder supplies word- and sentence-level attention scores, and ranking review words by those scores yields an open-vocabulary tagset beyond the predefined 71 labels without any tag-level supervision; in a human evaluation on 21 films, about 74% of 141 extracted tags were judged relevant by at least one of three raters.","pith_inferences":["A natural extension would be to calibrate the slope threshold per movie instead of fixing it on validation, since movie length and review style vary; an adaptive threshold may improve extraction for longer reviews.","The same attention-as-supervision trick could be applied to books, TV episodes, or video game narratives, where plot summaries and reader/viewer reviews are both available; the paper's out-of-domain examples hint at this but are not systematic.","The gating vector $z$ could be interpreted per tag as the model's source preference; a failure analysis comparing gate activations with review helpfulness ratings might reveal when reviews mislead rather than help.","Because the open-vocabulary tagset is derived only from attention, a direct comparison of gamma-ranked words against human-annotated relevant spans would be the clean test of whether attention is truly tracking story content."],"forward_implications":["A practical tagging system can start from synopses alone and accept reviews as they arrive; the same trained model handles both settings with no reconfiguration.","At least roughly 31-40 reviews per movie are enough to see a reliable gain in tag F1, so systems can decide when to switch from synopsis-only to multi-view mode.","Tags like non-fiction, inspiring, haunting, and pornographic are better recovered from reviews than synopses, suggesting review text is not merely redundant with plot.","Attention-derived open-vocabulary tags extend the closed 71-label tagset, so story description is no longer bounded by what appears in training and development sets.","The 74% human-relevance rate on extracted tags, if stable, means user reviews can be mined for new story attributes without annotating reviews word by word."],"supporting_citations":[{"why":"Provides the MPST corpus of 71 movie tags and the dataset the paper extends with reviews.","marker":"Kar et al. (2018a)"},{"why":"Supplies the CNN-EF emotion-flow system used as the main supervised baseline on synopsis-only tag prediction.","marker":"Kar et al. (2018b)"},{"why":"Supplies the hierarchical attention network used as the base encoder for both views.","marker":"Yang et al. (2016)"},{"why":"Provides the multiple-instance-learning approach to training and applying instance-level attention from bag labels.","marker":"Angelidis and Lapata (2018)"},{"why":"Supplies the gated fusion mechanism that combines synopsis and review representations.","marker":"Arevalo et al. (2017)"},{"why":"TextRank summarizes each movie's many reviews into one review document fed to the review encoder.","marker":"Mihalcea and Tarau (2004)"},{"why":"Sentence-BERT serves as the strong pretrained-language-model baseline for synopsis representation.","marker":"Reimers and Gurevych (2019)"},{"why":"GloVe vectors initialize the word embeddings in both encoders.","marker":"Pennington et al. (2014)"}],"fun_headline_variants":["Reviews add 4 F1 to movie story tagging","Fusing reviews with synopses gains 4 F1 for movie tags","Unsupervised story tags mined from movie reviews","Multi-view model: synopses + reviews beat synopses alone","Movie reviews: a free source for extra story tags"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything about the unsupervised part rests on one premise: attention weights trained to predict the 71 predefined tags also mark words that name new story attributes, and the only check is human rating of 141 tags from 21 movies.","fun_headline_variants_meta":{"raw":{"variants":["Reviews add 4 F1 to movie story tagging","Fusing reviews with synopses gains 4 F1 for movie tags","Unsupervised story tags mined from movie reviews","Multi-view model: synopses + reviews beat synopses alone","Movie reviews: a free source for extra story tags"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1406,"prompt_tokens":810,"completion_tokens":596,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":512}},"tokens_in":426,"tokens_out":596,"duration_ms":5786,"temperature":1.0,"reasoning_tokens":512,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:22:15.445812+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the extraction on a larger held-out sample and ask annotators to mark every review span that names a story attribute; if the model's top-ranked attention words consistently miss those spans, or if relevance rates on the larger sample fall well below the reported 74%, the unsupervised tagset is an artifact of attention rather than a story signal.","supporting_citations":[],"review_version":1}