{"id":"89b9a136-104e-4714-ae59-7d70b900a7d3","arxiv_id":"1908.10328","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A context-aware neural model identifies the five traditional turning points in movie plot synopses better than position-based baselines, but its projection onto screenplay scenes is still far below human agreement.","lead":"This paper introduces a dataset and neural models for finding the five key turning points in movie plots, using Wikipedia plot summaries and screenplays. It reports that a context-aware model beats position-only baselines for finding these moments in summaries, but projecting them onto screenplay scenes remains noisy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"End-to-end screenplay projection is numerically below the Theory baseline in Table 6, so the abstract's 'outperforms' claim is unsupported for the central use case.","rationale":"The reader's weakest assumption was the low inter-annotator agreement of the screenplay goldstandard. That is a real threat to evaluation reliability, and I agree it should be addressed. But the more direct threat to the central claim is that the fully end-to-end row in Table 6 is numerically below the Theory baseline on both TA and PA. This does not depend on adjudicating annotation noise; it is a mismatch between the abstract's 'outperforms' claim and the reported numbers for the exact system the abstract describes. The strongest contributions survive: TRIPOD is a new public dataset, the synopsis-level model beats position baselines and the AMT ranking test supports it, and the goldstandard-supervised TAM shows a plausible advantage in screenplay projection. Those facts support an accept-shaped verdict with conditions. The condition should be that the authors either revise the abstract to separate the synopsis outperformance from the screenplay projection result, or provide a significant end-to-end advantage over the theory baseline and a robustness analysis for the neighborhood-size choice. The absence of significance tests is especially important here because the end-to-end deficit is small (0.79 TA, 1.34 PA), so sampling variability could change the conclusion; but as reported, the evidence does not support the central claim. For these reasons I keep the reader's CONDITIONAL verdict and partially agree with the stated weakest assumption.","tokens_in":17522,"tokens_out":5314,"duration_ms":52216,"concrete_test":"Run a paired bootstrap or Wilcoxon signed-rank test on the per-movie scores (five-fold or per-title) comparing TAM End2end against the Theory baseline for TA and PA, using the released TRIPOD evaluation code; report the effect size and confidence intervals. If the end-to-end model is not significantly better than the theory baseline (or is worse), the abstract must be revised to restrict the outperformance claim to synopsis-level prediction or to the goldstandard-supervised TAM variant.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract claims an end-to-end model that identifies turning points in plot synopses and projects them onto screenplay scenes 'outperforms strong baselines.' The only row in Table 6 that matches this description is TAM End2end; it attains TA 7.87 and PA 9.33, while the Theory baseline attains TA 8.66 and PA 10.67. The end-to-end system is therefore numerically worse than the theory baseline on both headline metrics, and no significance test is reported. The better screenplay numbers in Table 6 (e.g., TAM TA 10.63, PA 13.33) are obtained using goldstandard synopsis TP labels, which is not the end-to-end setting described in Section 4.3. This is an internal evidence/claim mismatch, independent of the low inter-annotator agreement: even if the screenplay gold labels are noisy, the comparison against the theory baseline would need to show a significant advantage for the end-to-end pipeline, and Table 6 does not. The low IAA (TA 35.48, PA 56.67) compounds the problem, but it is secondary to the missing comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new task, turning point (TP) identification in movies, and argues that the five canonical TPs of screenwriting theory (opportunity, change of plans, point of no return, major setback, climax) provide a plot segmentation useful for downstream summarization and question answering. The authors introduce TRIPOD, a dataset of 99 movies with sentence-level TP annotations for Wikipedia plot synopses and scene-level gold annotations for 15 screenplays. They develop a series of models — a BiLSTM sentence-contextualized classifier (CAM), a topic-aware variant (TAM) with a context-interaction layer that compares each sentence with its left and right context, TP-specific multi-view encoders, and entity-enriched inputs — and an end-to-end pipeline (Section 4.3) that feeds predicted synopsis TPs into a scene-projection model with position-window constraints. Experiments show that the synopsis-level models clearly beat position-based baselines (Table 4b), while the screenplay-level results (Table 6) are much weaker: the best system reaches TA 14.18% against a human agreement of 35.48%, and the end-to-end system (TAM End2end) is numerically below the Theory baseline. The paper also includes an AMT evaluation of synopsis highlights and a per-TP error analysis.","tokens_in":17717,"tokens_out":10560,"duration_ms":103989,"significance":"The synopsis-level contribution is genuine and well evidenced: TAM with TP-specific views reaches TA 38.57% versus 22.00% for the Theory and 28.00% for the Distribution baseline (Table 4b; +entities 41.33%), the per-TP distance analysis (Table 5) shows that the model tracks the relative difficulty pattern of human annotation, and the AMT ranking experiment provides independent, if modest, support. The paper also releases the TRIPOD dataset publicly, which is a real asset for follow-up work, and the task definition is clearly grounded in screenwriting theory rather than an ad hoc artifact. The weaknesses identified below concern claim calibration and evaluation rigor, not the absence of a useful contribution; the narrative-structure analysis task itself and the synopsis-level results are likely to be of interest to the computational literary analysis community.","major_comments":[{"comment":"The abstract and the contributions list claim that the end-to-end model that 'identifies turning points in plot synopses and projects them onto scenes in screenplays' 'outperforms strong baselines based on ... the expected position of turning points.' In Table 6, however, the only row realizing that description, TAM End2end, attains TA 7.87 and PA 9.33, both below the Theory baseline (TA 8.66, PA 10.67); TAM End2end is better than the Theory baseline only on the distance metric D (10.16 vs 10.45). No significance test is reported, so the Section 6 statement that the end-to-end model 'still remains competitive against the baselines' is the strongest defensible characterization, and even that is questionable given that the Theory baseline is numerically better on both headline accuracy metrics. The synopsis-level results in Table 4b do support an outperformance claim (TAM + TP views: TA 38.57 vs 22.00 Theory and 28.00 Distribution), so the authors should either re-scope the abstract and Section 1 to the synopsis-level task, or provide significance tests demonstrating an end-to-end advantage.","section":"Abstract; §4.3; §6, Table 6"},{"comment":"The paragraph introducing Table 6 states that 'The best performing model is TAM' and uses that as support for the segmentation hypothesis, but the table shows CAM at TA 11.11 / PA 16.00 (and CAM + entities at 14.18 / 17.33), both above TAM's 10.63 / 13.33; TAM is better only on D (8.94 vs 10.23 and 10.15). If D is the primary evaluation metric, the paper should say so explicitly and justify that choice; otherwise the claim and the conclusion drawn from the table need to be corrected.","section":"§6, Table 6"},{"comment":"The screenplay-level gold standard has inter-annotator TA of 35.48% and PA of 56.67% over the six doubly annotated movies, and the best system in Table 6 reaches only 14.18% TA and 17.33% PA, far below the level at which two human annotators agree with each other. Because the test set contains only 15 movies evaluated with five-fold cross-validation, and no significance tests or per-fold breakdowns are reported, differences among systems in Table 6 (e.g., CAM vs TAM, TAM vs TAM End2end) cannot be reliably distinguished from annotation noise. The authors should add statistical testing or fold-level results and should explicitly discuss the ceiling imposed by the label noise when interpreting the comparisons in Table 6.","section":"§3, Eq. (2); §6, Table 6"}],"minor_comments":[{"comment":"The sentence 'we also experimented 3 with' contains a stray literal '3' and should read 'we also experimented with'.","section":"§6"},{"comment":"The phrase 'An other such example' should read 'Another such example'.","section":"Appendix C"},{"comment":"The abbreviation TA is used for two different quantities: the exact-sentence agreement in the synopsis annotation and the Jaccard-based scene-set agreement of Eq. (2); giving these distinct names would prevent confusion when comparing Tables 4 and 6.","section":"§3"},{"comment":"The AMT evaluation reports no number of participants and no significance test for the ranking differences (goldstandard 1.87, TAM 1.98, distribution baseline 2.16); the claim that the human evaluation validates the model should be softened accordingly.","section":"§6, Fig. 4"},{"comment":"TextTiling is mentioned as performing poorly, but no result is reported; since the paper has an appendix with available space, reporting the number would be preferable to an unreported comparison.","section":"§6"},{"comment":"The context window size l is a key hyperparameter of the context interaction layer (two sentences for task 1, 20% of screenplay length for task 2), yet no sensitivity analysis is reported; a brief analysis would strengthen the claim that topic-boundary information, rather than the specific window choice, drives TAM's gains.","section":"Appendix B"},{"comment":"The data augmentation section states that multiple annotations were used 'whenever these were available and considered reliable,' but the reliability criterion is never specified; this should be clarified for reproducibility.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"Recommendation: major_revision. The public dataset and the synopsis-level results are solid, but the abstract and Section 6 overclaim relative to Table 6, and the screenplay-level evaluation is under-powered given the low inter-annotator agreement. The paper should be given the opportunity to re-scope its claims, add significance testing, and clarify the primary screenplay metric. No concerns about novelty disclosure or citation practice; the issue is claim calibration, not methodology integrity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this paper earns a revise-and-resubmit if the authors fix the abstract and add some robustness checks, not because the core resource is flawed. TRIPOD is a real contribution: 99 movies, synopsis-level turning-point annotations with 64% exact-sentence agreement, and a per-TP error analysis that aligns with intuitions about which TPs are hard. The synopsis model (TAM + TP views) beats the position baselines by about ten points of TA (38.6 vs 28.0 distribution, 22.0 theory), and the human evaluation supports that ordering. That is a legitimate result.\n\nThe soft spots are real but concentrated. The abstract promises an end-to-end model that identifies TPs in plot synopses and projects them onto screenplay scenes and \"outperforms strong baselines.\" Table 6 shows the opposite for the end-to-end system: TAM End2end gets TA 7.87/PA 9.33, while the Theory baseline gets 8.66/10.67. The paper text softens this to \"competitive,\" but competitive is not outperforming. The better screenplay numbers in that table come from feeding goldstandard synopsis labels into the scene projector, which is a different setting. That mismatch needs to be fixed, either by claiming less or by showing the end-to-end gap is not significant.\n\nA second issue: the neighborhood size for TP scenes (three) is set from the goldstandard scene annotations, which is a mild form of evaluation leakage. With 15 movies, there is no held-out tuning set, so this is understandable, but it should be stated and ideally tested for sensitivity. Third, no significance tests are reported anywhere; with 15 movies, the differences between 8.66 and 7.87 are plausibly noise.\n\nThe low scene-level IAA (TA 35.48, PA 56.67) is a concern, but the paper is transparent about it, and the partial-agreement metric is reasonable. I would not treat that as disqualifying; the synopsis-level agreement is decent and the dataset's main value is there.\n\nBottom line: the dataset and synopsis experiments are worth serious referee time. The abstract overstates the end-to-end result, and a revision should either restrict the claim to synopsis TP identification or present significance tests that justify it. If I were the editor I would send it out, with the expectation that the authors can fix the framing. I'd cite the dataset if I worked on narrative; the model itself I'd treat cautiously.\n\nBest,\n[Name]","headline":"Genuinely useful dataset and solid synopsis-level results, but the abstract's end-to-end claim is contradicted by the paper's own Table 6.","tokens_in":18266,"tokens_out":2888,"would_cite":true,"duration_ms":30074,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An end-to-end neural network identifies turning points in movie plots and maps them onto screenplay scenes, beating strong baselines.","keywords":["turning point identification","narrative structure","screenplays","plot synopses","neural networks","text segmentation","movie plot analysis","distant supervision"],"falsifier":"On a freshly annotated test set, if the position-only distribution baseline matches or beats the topic-aware model on Partial Agreement, the claimed advantage of content-aware turning point projection fails. Concretely, the end-to-end model's Total Agreement would need to stay significantly above the 8.66% theory baseline on an independent set of 15 movies.","tokens_in":17292,"feed_emoji":"🎬","tokens_out":7058,"duration_ms":66669,"temperature":0.7,"pith_summary":"This paper proposes turning point identification as a way to analyze movie narrative structure: every screenplay is assumed to contain five turning points—Opportunity, Change of Plans, Point of No Return, Major Setback, and Climax—that segment the story into thematic units. The authors build TRIPOD, a dataset of 99 movies with turning point annotations on Wikipedia plot synopses and scene-level annotations on a subset of screenplays. They then train neural models that first choose the turning point sentences in a synopsis and then project those onto scenes in the corresponding screenplay. The paper's central claim is that these models outperform strong baselines on both tasks, and that turning-point segmentation can support downstream tasks such as summarization and question answering over long narratives.","feed_headline":"Neural net finds turning points in movie plots, beating baselines","feed_subtitle":"It maps synopsis beats to screenplay scenes, giving automatic story segmentation.","key_machinery":"The load-bearing mechanism is a context interaction layer that treats a turning point as a segment boundary. For each sentence or scene, the model computes a left-context vector and a right-context vector by averaging the representations of neighboring segments, then enriches the segment's representation with the element-wise product, cosine similarity, and normalized dot product between the segment and each context. This makes the classifier sensitive to whether the segment sits at a topic shift. Around this, the architecture adds a bidirectional LSTM encoder for the whole narrative, one encoder per turning point type, and an entity-aware encoder that weighs words by attention after co-reference resolution. Training also uses distant supervision: the average position of each turning point in training synopses defines windows that constrain both labeling and scene projection.","core_discovery":"The central claim is that turning points are identifiable automatically and that modeling them as topic boundaries helps. The paper's best synopsis model, a topic-aware model with per-turning-point views, reaches 41.33% Total Agreement on the test set versus 22.00% for the distribution baseline and 64.00% human agreement. On screenplays, the topic-aware model reaches 10.63% Total Agreement and 13.33% Partial Agreement using goldstandard synopsis labels, and the fully automatic end-to-end version reaches 7.87% and 9.33%, compared with 8.66% and 10.67% for the theory baseline. The intended reading is that lexical content, context, and expected position together, rather than position alone, locate the narrative beats.","pith_inferences":["An evaluation centered on downstream summarization or question answering would test whether the segmentation carries practical value beyond the reported scene-level metrics.","The expected-position prior could be made genre- or length-conditioned, since the paper reports variance in turning point positions and uses a single global window.","The same architecture transfers naturally to other narrative forms, such as novels or TV episodes, that have act-like boundaries."],"forward_implications":["Automatic annotations can be generated for screenplays and synopses that lack them, giving a standard five-part segmentation of setup, complications, and aftermath.","Summarization and question answering systems can first select turning-point segments instead of scanning the whole screenplay, narrowing the search space.","The TRIPOD dataset provides a benchmark for other narrative-structure models to compare against.","The same annotation scheme could be extended to more movies and to multimodal video, since turning points are defined at the plot level, not the text level."],"supporting_citations":[{"why":"Defines the five turning points and six-stage scheme that the task is built on.","marker":"(Hague, 2017)"},{"why":"Provides the screenwriting-theory account of turning points as junctions between acts.","marker":"(Thompson, 1999)"},{"why":"Supplies the Scriptbase screenplays from which the TRIPOD movies are selected.","marker":"Gorinski and Lapata (2015)"},{"why":"The pre-trained sentence encoder used to initialize all segment representations.","marker":"Cer et al. 2018"},{"why":"Text segmentation via similarity of context windows, which motivates the context interaction layer.","marker":"(Hearst, 1997)"},{"why":"The LSTM architecture used to contextualize synopsis sentences and screenplay scenes.","marker":"Hochreiter and Schmidhuber 1997"},{"why":"Shows why long narratives resist passage-retrieval approaches, motivating turning-point segmentation for QA.","marker":"Kočiský et al. (2018)"},{"why":"Used for co-reference resolution in the entity-specific encoder variant.","marker":"(Manning et al., 2014)"}],"fun_headline_variants":["Neural net locates turning points in movie plots, tops baselines","AI identifies movie story beats, beats baseline models","Turning point detector for films outperforms theory baseline","Neural model finds movie plot beats, outperforms baselines","Movie narrative structure: neural net beats baselines on beats"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on treating human-annotated turning point scenes as the correct answer, even though two human annotators pick the exact same scenes only about 35% of the time; if those labels are too noisy, the reported scene-level results do not measure how well the model finds true turning points.","fun_headline_variants_meta":{"raw":{"variants":["Neural net locates turning points in movie plots, tops baselines","AI identifies movie story beats, beats baseline models","Turning point detector for films outperforms theory baseline","Neural model finds movie plot beats, outperforms baselines","Movie narrative structure: neural net beats baselines on beats"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1457,"prompt_tokens":832,"completion_tokens":625,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":544}},"tokens_in":448,"tokens_out":625,"duration_ms":6144,"temperature":1.0,"reasoning_tokens":544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:46:14.899779+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a freshly annotated test set, if the position-only distribution baseline matches or beats the topic-aware model on Partial Agreement, the claimed advantage of content-aware turning point projection fails. Concretely, the end-to-end model's Total Agreement would need to stay significantly above the 8.66% theory baseline on an independent set of 15 movies.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the five turning points and six-stage scheme that the task is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the screenwriting-theory account of turning points as junctions between acts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Text segmentation via similarity of context windows, which motivates the context interaction layer."}],"review_version":1}