Pith. sign in

REVIEW 4 major objections 6 minor 14 references

Multi-view Story Characterization from Movie Plot Synopses and Reviews

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read User reviews lift automatic movie tagging by about 4 F1 points.

desk verdict A useful multi-view model with a real F1 gain and a neat extraction idea, but the paper's headline 74% for open-vocabulary tags is internally inconsistent with the appendix's stricter criterion and should be re-reported. read the letter →

arxiv 1908.09083 v2 pith:YDBI6L5N submitted 2019-08-24 cs.CL

Sudipta Kar , Gustavo Aguilar , Mirella Lapata , Thamar Solorio This is my paper · ORCID
classification cs.CL
keywords multi-labeltagpredictionmovieplotsynopsesuserreviewshierarchicalattentionmultipleinstancelearninggatedfusionopen-vocabularyextraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that user reviews of movies are a genuinely useful second source for automatic story characterization, not just a noisy complement to plot synopses. It builds a multi-view neural model that reads a synopsis and a condensed review summary through separate hierarchical attention encoders, combines them with gated fusion, and predicts a predefined set of 71 story tags. On the test set the multi-view model reaches 41.84 micro-F1 for the top three tags, compared with 37.94 for the authors' best synopsis-only system, and roughly 31-40 reviews are enough to see a dependable gain. The same review encoder then acts as an unsupervised tag miner: ranking words by attention-derived importance produces an open-vocabulary set of story attributes beyond the 71 predefined tags, and human judges rated about 74% of those extracted tags as relevant by at least one vote. A sympathetic reader would take away that user opinions encode story content worth extracting, and that attention weights can stand in for expensive word-level annotation.

What carries the argument

The load-bearing mechanism is a pair of hierarchical attention encoders—BiLSTMs with word-level attention feeding sentence representations, then sentence-level attention to form a document vector—one for the synopsis and one for a text-rank summary of up to 100 user reviews. The two document vectors are merged by gated fusion, $z = \sigma(W_z[d^{PS}_h,d^R_h])$, with $h = z*h^{PS} + (1-z)*h^R$, and sentence-level tag predictions are aggregated as in multiple instance learning. After training, the review encoder's attention weights define an importance score for each word, $\gamma_{ij} = \alpha^R_{W_{ij}} \alpha^R_{s_i} |s_i|$, and words whose rank curve flattens below a slope threshold become candidate open-vocabulary tags.

What would settle it

Run the extraction on a larger held-out sample and ask annotators to mark every review span that names a story attribute; if the model's top-ranked attention words consistently miss those spans, or if relevance rates on the larger sample fall well below the reported 74%, the unsupervised tagset is an artifact of attention rather than a story signal.

Watch

Extended reading notes

Core claim

The paper argues that movie reviews carry story attributes that synopses miss, and that a model trained on both can tag a film better than one trained on synopses alone. On the test set, its gated-fusion multi-view model reaches 41.84 micro-F1 (top-3 tags) using synopses and reviews, against 37.94 for the best synopsis-only variant in the same paper. The same trained review encoder supplies word- and sentence-level attention scores, and ranking review words by those scores yields an open-vocabulary tagset beyond the predefined 71 labels without any tag-level supervision; in a human evaluation on 21 films, about 74% of 141 extracted tags were judged relevant by at least one of three raters.

Load-bearing premise

Everything about the unsupervised part rests on one premise: attention weights trained to predict the 71 predefined tags also mark words that name new story attributes, and the only check is human rating of 141 tags from 21 movies.

Editorial extensions

If this is right

  • A practical tagging system can start from synopses alone and accept reviews as they arrive; the same trained model handles both settings with no reconfiguration.
  • At least roughly 31-40 reviews per movie are enough to see a reliable gain in tag F1, so systems can decide when to switch from synopsis-only to multi-view mode.
  • Tags like non-fiction, inspiring, haunting, and pornographic are better recovered from reviews than synopses, suggesting review text is not merely redundant with plot.
  • Attention-derived open-vocabulary tags extend the closed 71-label tagset, so story description is no longer bounded by what appears in training and development sets.
  • The 74% human-relevance rate on extracted tags, if stable, means user reviews can be mined for new story attributes without annotating reviews word by word.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to calibrate the slope threshold per movie instead of fixing it on validation, since movie length and review style vary; an adaptive threshold may improve extraction for longer reviews.
  • The same attention-as-supervision trick could be applied to books, TV episodes, or video game narratives, where plot summaries and reader/viewer reviews are both available; the paper's out-of-domain examples hint at this but are not systematic.
  • The gating vector $z$ could be interpreted per tag as the model's source preference; a failure analysis comparing gate activations with review helpfulness ratings might reveal when reviews mislead rather than help.
  • Because the open-vocabulary tagset is derived only from attention, a direct comparison of gamma-ranked words against human-annotated relevant spans would be the clean test of whether attention is truly tracking story content.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper addresses automatic story characterization by predicting a closed set of 71 story tags from movie plot synopses and user reviews. It extends the MPST corpus with review summaries, proposes a hierarchical attention encoder with multiple-instance learning and gated fusion of synopsis/review representations, and reports that gated fusion achieves 41.84 micro-F1 versus 37.94 for the best synopsis-only system. It further proposes extracting an open-vocabulary tagset from review attention weights, validated by a human evaluation on 21 movies. The dataset and code are released.

Significance. If the results hold, the paper makes a useful empirical contribution: it provides evidence that user reviews contain complementary signal for story tagging, and it sketches a weakly supervised route to open-vocabulary attributes. The supervised comparison is consistent across Table 2, the subset analysis, and the human evaluation, and the dataset/resource contribution is concrete. The qualitative generalization checks for 2019 movies and non-movie narratives are a valuable addition. However, the unsupervised extraction contribution rests on a human evaluation whose reported 74% relevance figure conflicts with the appendix's stated two-vote criterion, and the main review-versus-synopsis gain is reported without significance testing or variance estimates.

major comments (4)
  1. [Section 6.1 and Appendix F] Section 6.1 reports that 24% + 18% + 32% = 74% of the open-vocabulary tags YC were 'marked as relevant by at least one judge,' but Appendix F explicitly states the evaluation protocol as 'We consider a tag as relevant if it has at least two votes.' Under the appendix's stated standard, the relevant fraction is 24% + 18% = 42%, not 74%. This is not merely a presentational difference: the 74% figure is the headline quantitative evidence for the unsupervised extraction contribution. The paper must adopt one standard consistently, report both criteria, and adjust the claims in the abstract and Section 6.1 accordingly. Please also provide the per-tag vote distributions underlying Figure 7(b) so the reader can verify both numbers.
  2. [Section 5, Table 2] The central claim that adding reviews improves tag prediction (37.94 to 41.84 in top-3 micro-F1) is not accompanied by statistical significance testing or variance estimates. The t-test footnote applies only to the comparison among gated fusion, merge-texts, and concatenation; it does not compare the review-based models with the synopsis-only systems. Because the reported gains are small in absolute terms and each configuration appears to be trained once, I ask for either multiple seeds with standard deviations or a significance test for the synopsis-only versus synopsis+reviews comparison.
  3. [Section 4.2, Eq. (1)] The importance score gamma_ij = alpha_Wij * alpha_si * |si| treats word-level attention as a direct measure of semantic relevance, and the stopping threshold of 5e-3 is set 'based on our observations on the validation set.' This makes the extraction loop partly self-referential: the same attention weights trained for closed-set tag prediction define both the candidate tags and, via validation-based thresholding, their stopping criterion. The human evaluation is an external check, but it only covers 21 movies. To strengthen this contribution, please provide an additional validation of attention-based importance, such as a comparison with held-out review-side annotations or a sensitivity analysis over the threshold.
  4. [Section 6.1] The human evaluation uses three judges on 21 movies and 141 tags, but no inter-annotator agreement statistic is reported. Given that the central quantitative claim for YC depends on judge votes, please report agreement (e.g., Fleiss' kappa) and clarify how disagreements were adjudicated.
minor comments (6)
  1. [Abstract] 'demonstrate how can we take advantage' should be 'demonstrate how we can take advantage.'
  2. [Section 4.2] 'Figure-3' should be 'Figure 3.'
  3. [Table 2] The footnote explaining the t-test should state explicitly which pairwise comparisons were tested; the current single asterisk is ambiguous.
  4. [Appendix F] The emoticon legend is useful, but the appendix's relevance criterion is inconsistent with the main text; please align the two or explicitly report both standards.
  5. [Section 6.3, Table 3] The claim that 'most of these tags except neo noir were found in IMDB' is informal; consider reporting the number of matched tags per movie.
  6. [Throughout] Capitalization of IMDb/IMDB and MPST is inconsistent; please standardize.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: core F1 gains are tested on held-out labels and open-vocabulary tags are externally human-evaluated.

full rationale

The paper's central quantitative claim (Table 2: gated fusion 41.84 vs synopsis-only HN(A)+MIL 37.94 micro-F1) is an empirical head-to-head comparison on a held-out test set against gold labels from the MPST corpus; no prediction is recomputed from a fitted parameter or from the model's own output. The open-vocabulary extraction (Section 4.2, Eq. 1) defines candidate tags from the trained review encoder's attention weights, but the paper does not present this extraction as a supervised prediction whose validity is established by the training objective. Its validity is checked externally through human judges on 21 movies in Section 6.1, and the attention-to-relevance link is an explicit domain assumption rather than an equation that reduces to the model's loss. The stopping threshold of 5e-3 is a validation-tuned hyperparameter, not a fitted label or a renamed prediction. The self-citations (Kar et al. 2018a for the MPST dataset, Kar et al. 2018b for the CNN-EF baseline) point to public resources and prior published systems; they are not invoked as a uniqueness theorem or as an unverified ansatz, and the paper also includes independent baselines such as SBERT. The reported discrepancy between the main-text summary that about 74% of open-vocabulary tags were relevant 'by at least one judge' and Appendix F's stated criterion of 'at least two votes' is an internal-consistency/correctness concern, not a circular-derivation concern. Overall, I find no step where a claimed result is equivalent by construction to its input.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a small number of domain assumptions about the data and the model: that MPST user tags are a valid gold standard, that TextRank summaries preserve review information, and that attention trained for the closed tagset transfers to open-vocabulary relevance. The only fitted constant that directly shapes a reported output is the 5e-3 threshold for YC extraction; all other hyperparameters are standard and affect overall accuracy rather than the novelty of the method. No invented entities are introduced.

free parameters (1)
  • Open-vocabulary tag extraction threshold = 5e-3
    In Section 4.2, the derivative of sorted word importance scores is thresholded at 5e-3 to stop tag selection; the authors state this is based on observations on the validation set.
assumptions (4)
  • domain assumption User-assigned tags in the MPST corpus are a valid gold standard for story attributes
    The 71-label tagset from IMDB/MovieLens users is used as supervision for both synopsis and review encoders; no annotation quality analysis is provided.
  • domain assumption TextRank summaries of the collected reviews preserve the opinion-rich content needed for tag prediction
    Section 3 compresses up to 100 reviews into one summary; if salient content is lost, the reported review-view gains would be specific to this summarization rather than to reviews in general.
  • domain assumption Attention weights learned under bag-level supervision indicate word-level relevance for story attributes
    Equation 1 in Section 4.2 defines word importance solely from attention; the open-vocabulary extraction has no direct supervision and relies on this transfer.
  • domain assumption Treating each sentence as an instance in a multiple instance learning framework is valid for tag prediction
    The MIL formulation assumes that movie-level tags can be explained by sentence-level evidence in the synopsis and review summary (Section 4.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-view Story Characterization from Movie Plot Synopses and Reviews." pith.science (2026). https://pith.science/paper/YDBI6L5N

@misc{pith2026190809083,
  author       = {Pith},
  title        = {Pith review of: Multi-view Story Characterization from Movie Plot Synopses and Reviews},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YDBI6L5N}},
  note         = {Machine review of arXiv:1908.09083}
}
read the original abstract

This paper considers the problem of characterizing stories by inferring properties such as theme and style using written synopses and reviews of movies. We experiment with a multi-label dataset of movie synopses and a tagset representing various attributes of stories (e.g., genre, type of events). Our proposed multi-view model encodes the synopses and reviews using hierarchical attention and shows improvement over methods that only use synopses. Finally, we demonstrate how can we take advantage of such a model to extract a complementary set of story-attributes from reviews without direct supervision. We have made our dataset and source code publicly available at https://ritual.uh.edu/ multiview-tag-2020.

Figures

Figures reproduced from arXiv: 1908.09083 by the authors.

Figure 1
Figure 1. Example snippets from plot synopsis and review of The Godfather and tags that can be generated from these. tions, loyalty, greed, and mafia, whereas the gold standard tags from the plot are violence, murder, atmospheric, action, and revenge. In this paper, we show that such information in reviews can sig￾nificantly strengthen a supervised synopses to tag prediction system, hence alleviating the first limita￾tion. To… view at source ↗
Figure 2
Figure 2. In the center, we show an overview of the model that takes a plot synopsis and a review summary as input, uses two separate encoders to construct the high-level representations and uses them to compute P(YP ). (a) illustrates an enhanced view of the synopsis encoder. It uses a BiLSTM with attention to compute a representation shP i for the ith sentence in the synopsis. Additionally, a matrix of word-level attention … view at source ↗
Figure 3
Figure 3. All words in the shaded area under the solid black curve are selected as candidate tags for YC . XR, we compute an importance score γij as: γij = αWij × αsi × |si | (1) Here, αWij is the attention weight of word wij and αsi is the attention weight of the i th sentence. |si | indicates the number of words in the sentence, and helps overcome the fact that word-level atten￾tion scores are higher in shorter sentences. W… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Average change in F1 with respect to the number of reviews after combining review summaries with synopses. grindhouse film, and allegory. It shows that our model is successfully capturing different story at￾tributes from reviews that are possibly difficult to find in s…
Figure 5
Figure 5. Figure 5: Example sentences from the synopsis of the movie Rush Hour 3 with one of the most relevant tags from the sentence-level predictions. Importance of particular sentences and words for predicting tags is indicated by the highlight intensity of the sentence ids and words. …
Figure 6
Figure 6. Figure 6: Example sentences from the review of the movie August Rush with sentence ids and words highlighted based on their importance in tag prediction. Ground truth tags are thought-provoking, romantic, inspiring, flashback. tiveness of our proposed system by visualizing the a…
Figure 8
Figure 8. Figure 8: Percentage of gates activated (z > 0.5) for syn￾opses and reviews. More active gates indicate more impor￾tance of the source for certain tags. whether to watch the movie or not. At the same time, in 75% of the cases, complementary tags were also deemed relevant. 6.2 In…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages

  1. [10]

    Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. Ashish Vaswani, Noam Shazeer, N...

    work page 2019

  2. [11]

    In 2014 IEEE International Con- ference on Data Mining, pages 1037–1042

    Scalable multi- instance learning. In 2014 IEEE International Con- ference on Data Mining, pages 1037–1042. Joseph Worsham and Jugal Kalita

    work page 2014

  3. [12]

    In Proceedings of the 27th Inter- national Conference on Computational Linguistics , pages 1963–1973

    Genre Iden- tification and the Compositional Effect of Genre in Literature. In Proceedings of the 27th Inter- national Conference on Computational Linguistics , pages 1963–1973. Association for Computational Linguistics. Yumo Xu and Mirella Lapata

    work page 1963

  4. [2004]

    In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing

    Textrank: Bring- ing order into text. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing. Jeffrey Pennington, Richard Socher, and Christopher Manning

    work page 2004

  5. [2006]

    In Proceedings of the 15th ACM International Conference on In- formation and Knowledge Management, CIKM ’06, pages 43–50, New York, NY , USA

    Movie Review Mining and Summarization. In Proceedings of the 15th ACM International Conference on In- formation and Knowledge Management, CIKM ’06, pages 43–50, New York, NY , USA. ACM. Appendix A Data Pre-processing and Input Representation We tokenize the synopses and reviews using spaCy10 NLP library. To remove rare words and other noise, we retain the...

    work page

  6. [2008]

    In Proceedings of the ECML/PKDD 2008 Discovery Challenge

    Multilabel text classification for automated tag suggestion. In Proceedings of the ECML/PKDD 2008 Discovery Challenge. Jim Keeler and David E. Rumelhart

    work page 2008

  7. [2010]

    Coling 2010 Organizing Committee

    , pages 653–661. Coling 2010 Organizing Committee. E. Lin, S. Fang, and J. Wang

    work page 2010

  8. [2013]

    In 2013 27th International Conference on Advanced In- formation Networking and Applications Workshops, pages 179–184

    Mining on- line book reviews for sentimental clustering. In 2013 27th International Conference on Advanced In- formation Networking and Applications Workshops, pages 179–184. Bing Liu

    work page 2013

Show all 14 references
  1. [2014]

    In Proceedings of the 2014 conference on empirical methods in natural language process- ing (EMNLP), pages 1532–1543

    Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methods in natural language process- ing (EMNLP), pages 1532–1543. Philipp Petrenz

    2014

  2. [2015]

    In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

    Neural machine translation by jointly learning to align and translate. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. Douglas Biber

    2015

  3. [2016]

    Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies , pages 1480–1489, San Diego, California. Associa- tion for Computati...

    2016

  4. [2017]

    In 5th Inter- national Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings

    Gated mul- timodal units for information fusion. In 5th Inter- national Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio

    2017

  5. [2018]

    What’s This Movie About? A Joint Neural Network Archi- tecture for Movie Content Analysis. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa- pers), pages 1770–...

    2018

  6. [2019]

    BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171–...

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.