{"id":"87375093-46e1-4b4f-8140-ef57a2ce3738","arxiv_id":"2412.02897","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"MLD-EA fine-tunes Llama-3 to locate and fill a missing sentence in a five-sentence story using character emotions and actions, reporting strong F1 on the constructed task.","lead":"This paper introduces MLD-EA, a system that uses a large language model to find where a story is missing a sentence and then writes a fitting sentence to fill the gap. A general reader should care because it tests whether LLMs can check and repair narrative logic, not just generate text.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's ground truth is the Appendix A emotion-change heuristic: the missing sentence is chosen by Eq. 7 as the maximum emotion-discontinuity point, and the logic checker is evaluated against exactly that choice.","rationale":"The paper proposes a novel task, narrative logic detection, and claims that MLD-EA can identify narrative gaps by modeling character actions and emotions. The reader's weakest assumption identifies the central operationalization: missing sentences are chosen in Appendix A by the emotion-change heuristic (Eqs. 6-7), and the model is trained and evaluated on exactly these choices. I agree that this is the load-bearing weakness. The reported 76.04 average F1 in Table 3 shows that the model can predict the maximum emotion-discontinuity position, but it provides no evidence that this position corresponds to a genuine narrative logic gap as a human reader would perceive it. The paper contains no independent validation: no human annotation of logical gaps, no alternative dataset constructed with different criteria, and no analysis relating the emotion-change rule to causal or commonsense narrative structure. The ablation study (Table 6) does not resolve this because the ground truth is unchanged. The generation evaluation (Section 6.3, Table 4) uses the original deleted sentence as the reference, so it inherits the same operationalization; the BERTScore results are also mixed, with MLD-EA below some prompting baselines. The paper does include some useful artifacts, such as detailed prompt templates and a clear sub-task decomposition, and it honestly states the five-sentence limitation. However, those strengths do not support the central claim as phrased. The proposed concrete test, a scripted baseline that directly computes Eq. 6-7 from the model's predicted emotion labels, would settle whether the high F1 is merely heuristic replication. Unless such a test shows substantially different behavior, the current benchmark cannot distinguish MLD-EA's logic checker from a deterministic emotion-distance scorer, so the verdict of REJECT remains appropriate.","tokens_in":18111,"tokens_out":7503,"duration_ms":72828,"concrete_test":"On the released test split, implement a non-learned scorer that, given the same predicted emotion labels fed to MLD-EA's logic checker (from Section 6.1 and Appendix E), applies Eq. 6-7 to the 4-sentence story: for each candidate insertion position p, score the aggregate emotion-change at p using the supplied emotion tags, and output the argmax p (with -1 for complete stories where the score falls below a threshold). Compare the micro-F1 of this scripted baseline to MLD-EA's 76.04 in Table 3. If the script attains comparable F1, the model's \"logic checker\" is just the Appendix A heuristic in learned form, and the central claim of narrative-gap detection is unsupported. If the script's F1 is substantially lower, then MLD-EA captures signal beyond the emotion-change rule and the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix A constructs the dataset's positive examples by computing, for each character c, D(esi, esj, c) as the emotion distance between sentences si and sj divided by |si-sj|, then selecting si_max = argmax_{si} Σ_c Σ_sj D(...) (Eqs. 6-7) as the sentence to delete. Thus every \"logic gap\" in the test set is, by construction, the position of maximum aggregate emotional discontinuity. Section 5.3 and the prompts in Appendix E then provide the model with character emotions and actions, and Section 4.3's logic checker is trained (and baselines are prompted) to output this same index. The 76.04 average F1 in Table 3 therefore quantifies how accurately MLD-EA can reproduce the emotion-change rule from the input annotations; it does not measure detection of narrative gaps in any independent sense. This is load-bearing because the abstract and introduction claim LLMs can \"identify narrative gaps\" and \"check logical coherence\". Nothing in the paper validates that the emotion-change rule corresponds to human judgments of narrative incoherence: there is no human annotation study, no alternative test set, and no analysis of whether deleted sentences are causally or physically necessary. The ablation (Table 6) showing that actions help (w/o a drops 10.93 F1) does not rescue the claim: the model may be using actions to infer emotions or to approximate the same emotion-discontinuity scoring, since the ground truth is still defined only by emotions. The case study in Table 7 is a single hand-picked example. Consequently, the headline experimental result is consistent with a much weaker conclusion: MLD-EA learns to locate the sentence whose removal maximizes emotional distance, under the particular emotion taxonomy used.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MLD-EA, an LLM-based model for detecting \"narrative logic gaps\" in short stories and generating sentences to fill those gaps. It uses the Story Commonsense dataset, constructs missing-sentence labels by identifying the sentence whose removal maximizes aggregate emotional change (Appendix A, Eqs. 6-7), then fine-tunes a Llama-3 model to predict that index using character emotion and action annotations. The model is also used to predict actions/emotions of the missing sentence and generate a filler. Experiments report high F1 (76.04) for gap position prediction, improvements from emotion/action features, and competitive BLEU/ROUGE/BERTScore in generation.","tokens_in":18298,"tokens_out":6035,"duration_ms":63578,"significance":"If the evaluation were sound, the approach would be a noteworthy application of emotion/action modeling to narrative coherence. The paper makes a concrete, reproducible task definition and provides detailed prompts and ablations. However, the central evaluation is confounded: the ground truth is defined by an emotion-discontinuity rule, and the model is trained and evaluated on exactly that rule. Consequently, the reported results do not support the claim that the model detects narrative logic gaps in a general sense. The contribution is instead a demonstration that a fine-tuned LLM can predict a hand-crafted emotion-change heuristic from emotion labels. Without human validation or an independent definition of narrative gaps, the paper's main claim is unsupported.","major_comments":[{"comment":"The ground-truth missing position is defined in Appendix A as the sentence that maximizes aggregate emotional discontinuity (Eq. 7), and the narrative logic checker consumes character emotion labels and is fine-tuned to predict that index. The 76.04 average F1 in Table 3 therefore measures how accurately MLD-EA reproduces the Appendix A emotion-change rule, not whether the model identifies narrative gaps in any independent sense. No human annotation study or alternative test set is provided to validate that maximum-emotion-change positions correspond to narrative incoherence, so the abstract and Section 6.2 claims about 'narrative logic detection' and 'logical coherence' are not supported by the reported experiments.","section":"Appendix A, Eqs. 6-7; Section 6.2, Table 3"},{"comment":"The emotion distance function d in Eq. 6 is never defined, and the condition for when D is nonzero is truncated ('if esi,c and esj ,c' with no predicate). Since every positive example in the training and test sets is generated by this formula, the dataset construction is irreproducible and the task itself is underspecified. This is a load-bearing issue because an arbitrary choice of d would change the ground-truth labels and, in turn, the reported F1 scores.","section":"Appendix A, Eq. 6"},{"comment":"The generation task is scored against the single original deleted sentence as the reference. Because that sentence is selected by the emotion-change heuristic and many plausible sentences could fill the narrative gap, BLEU-1/2/4, ROUGE, and BERTScore against one reference are not valid measures of narrative coherence or even of generation quality. The absence of any human evaluation further weakens the claim that MLD-EA completes stories 'coherently both logically and emotionally' (Section 1).","section":"Section 6.3, Table 4"},{"comment":"The ablation study shows that removing emotions causes the largest F1 drop (13.45 points), which is exactly what one would expect if the model is reproducing the emotion-discontinuity rule that generated the labels. This result does not provide evidence for the general utility of actions and emotions in narrative logic checking; it is fully explained by the circularity of the evaluation design. A meaningful ablation would require an independent ground truth for what constitutes a missing or incoherent sentence.","section":"Section 6.4, Table 6"}],"minor_comments":[{"comment":"The condition for D = 0 is an incomplete sentence fragment ('if esi,c and esj ,c'); the intended predicate (e.g., both emotions being present and non-'none') should be stated explicitly.","section":"Appendix A, Eq. 6"},{"comment":"In the Gemma2-9B-it zero-shot row, the reported F1 of 6.49 for k=2 is inconsistent with the precision of 3.36 and recall of 1.00 (the harmonic mean of those values is 1.54), suggesting a numerical error that should be corrected.","section":"Appendix C, Table 10"},{"comment":"The related work does not discuss existing story-completion and cloze-style evaluations that also involve locating a missing sentence (e.g., ROCStories-based cloze tasks); discussing them would help position the claimed novelty of the 'narrative logic detection' task.","section":"Section 2"},{"comment":"There are numerous typographical and formatting issues, including 'border goals' in the abstract, 'SW AG' instead of 'SWAG' in Section 2, and an inconsistent citation for NPN (Bosselut et al., 2017 in the text versus Rashkin et al., 2018 in Table 2).","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The circularity issue is central and appears unintentional, but it cannot be fixed by minor edits because the evaluation design is the load-bearing component of the paper's main claim. The authors would need either to re-frame the contribution as detecting emotion discontinuities or, more valuably, to collect or use an independent set of human-annotated narrative gaps and re-run the whole evaluation. Given the current scope, I recommend rejection, although the underlying idea of using emotion and action chains for coherence checking might have merit if evaluated properly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is the task and the pipeline: it fine-tunes Llama3 to abstract character actions, classify emotions, locate a deleted sentence in a five-sentence story, and generate a replacement, using the Story Commonsense dataset. I have not seen this exact task before, and the prompt templates in Appendix E are detailed enough to reproduce. The authors also deserve credit for being transparent about the five-sentence limitation and for including an ablation that separates the contribution of actions and emotions.\n\nThe soft spot is load-bearing, and the stress-test note is right. Appendix A builds the benchmark by deleting the sentence with the maximum aggregate emotional change (Eqs. 6-7). The narrative logic checker in Section 4.3 is then trained on exactly those emotion labels and asked to predict that index. So the 76.04 F1 in Table 3 is mostly a measure of how well the model can reproduce the emotion-change heuristic, not evidence that it detects narrative gaps in any independent sense. The paper never validates that the heuristic matches human judgments of incoherence, and there is no human annotation study or separate test set. Without that anchor, the abstract's claim about \"identifying narrative gaps\" is stronger than the evidence supports.\n\nThe other issues are real but less severe. The main comparison is fine-tuned MLD-EA versus zero/few-shot baselines, so part of the improvement is just fine-tuning. There are no error bars or significance tests. BERTScore for generation is not better than the baselines. No code or data is released. These all limit confidence, but they are the usual fixable problems.\n\nThe generation side is more defensible. The \"correct index\" analysis in Appendix D shows that BERTScore improves sharply when the model is given the right position, which is a sensible way to isolate generation quality from gap detection. The VAD analysis is a nice idea, even if the differences look small.\n\nMy overall take: the task is worth pursuing, the paper is clearly written, and the authors are honest about what they did. But the central evaluative claim, that MLD-EA detects logical gaps via action-emotion reasoning, is not supported by the current benchmark. A revision that adds human judgments of narrative coherence, or an independently constructed test set, could make this a solid paper.\n\nI would send this to peer review rather than desk reject it, because the task is new and the flaw is fixable. A serious referee should ask for an independent validation of the emotion-change rule, fairer baselines, and error bars.","headline":"A clearly described, honest pipeline for a genuinely new task, but the headline F1 is mostly the model replicating the emotion-change rule that defined the ground truth.","tokens_in":18972,"tokens_out":1977,"would_cite":false,"duration_ms":55429,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a large language model trained to track characters' emotions and actions can locate the position of a deleted sentence in a short story and generate a coherent replacement, outperforming all prompting baselines.","keywords":["narrative coherence","story generation","emotion classification","action abstraction","missing sentence detection","large language models","fine-tuning","Plutchik emotions"],"falsifier":"Construct test stories where a sentence is deleted because a physical action or causal connection is missing but the surrounding emotions stay nearly flat, and check whether MLD-EA locates those gaps above chance; if it does not, its reported F1 reflects the emotion-change rule rather than narrative logic. A cleaner test is to compare MLD-EA's predicted missing positions against human-annotated logic gaps on held-out five-sentence stories.","tokens_in":17757,"feed_emoji":"📖","tokens_out":5677,"duration_ms":49624,"temperature":0.7,"pith_summary":"The paper introduces MLD-EA, a model that uses large language models to detect and repair gaps in narrative logic. Its central claim is that annotating each sentence with characters' emotions and actions, then fine-tuning an LLM on those annotations, lets the model identify the position from which a sentence was deleted in a five-sentence story and write a replacement that restores emotional and logical continuity. On the missing-sentence-position task it reports a micro-averaged F1 of 76.04, against 38.39 for the best prompting baseline. A reader should care because story generation systems still struggle to check their own coherence, and this work offers a mechanism for that check grounded in observable actions and emotions.","feed_headline":"Emotion-action model finds missing story sentences at 76% F1","feed_subtitle":"A fine-tuned LLM beats prompting baselines at locating and filling gaps in short stories by tracking characters' emotions and actions.","key_machinery":"The central machinery is the emotion–action abstraction pipeline: action abstraction writes each character's behavior as <Character>Action(Target, Object)</Character>, and emotion classification assigns one of Plutchik's eight emotions plus 'none' per character per sentence. The narrative logic checker then looks for discontinuity in these sequences, informed by the psychological link that emotions drive actions and actions alter emotions. The paper defines the training target in Appendix A: the missing sentence is the one whose removal maximizes aggregate emotional distance across characters (equations 6–7), so the model's logic-gap prediction is tied to that operationalization.","core_discovery":"In the paper's terms, MLD-EA treats narrative logic as the continuity of each character's emotion–action chain. It first abstracts actions and classifies emotions (from Plutchik's eight basic types plus 'none') for every sentence, then infers the index k where a missing sentence should be inserted, predicts the actions and emotions that the missing sentence should contain, and finally generates the sentence. The empirical core is Table 3: with actions and emotions included, MLD-EA reaches 76.04 average F1 on predicting the missing index versus 38.39 for the best baseline, and the ablation in Table 6 shows removing emotions (−13.45 F1) or actions (−10.93 F1) lowers accuracy. The paper also reports that including predicted actions and emotions improves BLEU and ROUGE scores for generation and brings the generated sentence closer to the reference on valence–arousal–dominance measures.","pith_inferences":["One extension the paper leaves implicit is a direct comparison of MLD-EA's predicted missing positions against human-annotated logic gaps; the current labels are generated by the emotion-change rule, so the high F1 may partly reflect learning that rule.","A testable extension is to replace the emotion-distance rule with a richer coherence signal, such as missing causal links or contradictions, to see whether the same architecture detects other narrative defects.","Because Plutchik's emotions and basic action frames are not English-specific, the pipeline could transfer to other languages or domains, though the paper does not claim this.","The emotion classifier is trained on the same dataset that defines the missing-sentence labels, so evaluating on held-out stories with independently human-marked gaps would separate genuine logic detection from label-rule memorization."],"forward_implications":["If correct, a lightweight fine-tune on emotion and action annotations can turn an instruction-tuned LLM into a sentence-level story coherence checker, a task that prompting alone handles poorly.","The same emotion–action representation could be applied to longer narratives, since the mechanism does not by itself depend on the five-sentence format (the paper's limitation acknowledges this remains untested).","The generation module shows a concrete way to use predicted emotions and actions as a bridge between gap detection and gap filling, improving n-gram overlap with the original text.","The large gains from adding actions and emotions suggest these features carry coherence signal that is not accessible from surface text alone."],"supporting_citations":[{"why":"Supplies the 4,853 five-sentence stories with emotion labels used for training, validation, and testing.","marker":"(Rashkin et al., 2018)"},{"why":"Provides the eight-emotion taxonomy that the emotion classification module predicts.","marker":"(Plutchik, 2001)"},{"why":"Supports the action-to-emotion direction in the logic-checking rules.","marker":"(Cannon, 1927)"},{"why":"Supports the emotion-to-action direction in the logic-checking rules.","marker":"(Zhu and Thagard, 2002)"},{"why":"Supplies the Llama3-8B-Instruct backbone used both for MLD-EA and as the strongest baseline.","marker":"(AI@Meta, 2024)"},{"why":"Provides the LoRA method used for parameter-efficient fine-tuning of the backbone.","marker":"(Hu et al., 2021)"},{"why":"Supplies the Llama-Factory fine-tuning framework used in the implementation.","marker":"(Zheng et al., 2024)"}],"fun_headline_variants":["Emotion-action tracking finds missing story sentences at 76% F1","MLD-EA uses emotion-action chains to locate narrative gaps","Tracking emotions and actions boosts story gap detection to 76%","LLM checks story coherence by following emotion-action links","Fine-tuned LLM spots missing sentences via emotion-action logic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a narrative logic gap is exactly the sentence whose removal causes the largest aggregate emotion change across characters, as defined by the rule in Appendix A; if real logic gaps differ from this operationalization, the model's high accuracy would not demonstrate narrative logic detection.","fun_headline_variants_meta":{"raw":{"variants":["Emotion-action tracking finds missing story sentences at 76% F1","MLD-EA uses emotion-action chains to locate narrative gaps","Tracking emotions and actions boosts story gap detection to 76%","LLM checks story coherence by following emotion-action links","Fine-tuned LLM spots missing sentences via emotion-action logic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1337,"prompt_tokens":888,"completion_tokens":449,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":363}},"tokens_in":504,"tokens_out":449,"duration_ms":5189,"temperature":1.0,"reasoning_tokens":363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:59:00.233295+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct test stories where a sentence is deleted because a physical action or causal connection is missing but the surrounding emotions stay nearly flat, and check whether MLD-EA locates those gaps above chance; if it does not, its reported F1 reflects the emotion-change rule rather than narrative logic. A cleaner test is to compare MLD-EA's predicted missing positions against human-annotated logic gaps on held-out five-sentence stories.","supporting_citations":[],"review_version":1}