PlotTwist: A Creative Plot Generation Framework with Small Language Models

Abhinav Thorat; Jyotin Goel; Madhav Kataria; Niranjan Pedanekar; Ravi Kolla

REVIEW 4 major objections 4 minor 109 references

PlotTwist: A Creative Plot Generation Framework with Small Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 3B-parameter model, aligned via preference learning, generates plots that rival frontier LLMs.

desk verdict The framework is real and the positive-negative prompting idea is worth borrowing, but the 'SLM beats frontier' claim is unsupported: every Table 1 number comes from a same-family Qwen evaluator with no human check. read the letter →

arxiv 2603.16410 v2 pith:CT7DYCOY submitted 2026-03-17 cs.CL cs.AI

Abhinav Thorat , Ravi Kolla , Jyotin Goel , Madhav Kataria , Niranjan Pedanekar This is my paper

classification cs.CLcs.AI

keywords creativeplotgenerationsmalllanguagemodelsDirectPreferenceOptimizationmixture-of-expertsrewardmodelnarrativequalitydimensionsLLM-as-a-judgealignment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PlotTwist argues that a small language model with 3 billion active parameters can generate premise-conditioned movie plots as good as—or better than—frontier models hundreds of times larger, provided the generation process is decomposed into specialized components aligned through preference learning. The paper claims the bottleneck in creative long-form storytelling is not raw model scale but the absence of explicit narrative-quality signals, and that those signals can be encoded in a reward model and transferred to an MoE generator via Direct Preference Optimization. A sympathetic reader would care because this points to a resource-efficient path to high-quality creative writing systems that could run without frontier-scale compute. The claim is supported by an automated agentic evaluator that gives PlotTwist higher scores on four of five narrative quality dimensions than every baseline except Claude Sonnet 4, and by evidence that both the reward model and the evaluator separate critically acclaimed from critically panned screenplay plots.

What carries the argument

The load-bearing mechanism is the separation of narrative quality into five scored dimensions—character development, tone consistency, pacing, narrative coherence, and emotional turning points—and the conversion of those scores into training signal. An Aspect Rating Reward Model is trained on synthetic aspect ratings produced by a five-model ensemble using positive-negative prompting: each model scores a plot twice, once for its strengths and once for its weaknesses, and the difference becomes the rating. A DPO loop then uses that reward model to select only high-confidence preference pairs (score at least 8, margin at least 0.5) drawn from plots generated by frontier models, yielding 160 pa

What would settle it

A blind study in which professional screenwriters or fiction editors rank PlotTwist and Claude Sonnet 4 outputs on the same 160 premises—or independently re-score all outputs with a different judge model—would settle the claim. If human rankings do not reproduce the evaluator's win rates, or if a second evaluator produces different aspect-level winners, the reported frontier-beating scores are artifacts of the evaluation setup.

Watch

Extended reading notes

Core claim

In the paper's own terms, the central discovery is that structured, preference-based alignment can compensate for limited model capacity in creative generation. PlotTwist—a Qwen-3-30B-A3B mixture-of-experts generator with 3B active parameters, aligned via Direct Preference Optimization on 160 high-confidence preference pairs—achieves higher agentic-evaluator scores than every baseline on four of five NQDs (tone consistency, pacing, narrative coherence, emotional turning points) and trails Claude Sonnet 4 by a small margin only on character development. The authors interpret this as evidence that decomposing generation into an aspect-rating reward model, a DPO-aligned MoE generator, and an in

Load-bearing premise

The paper's headline comparisons are only as strong as its agentic evaluator: a Qwen-3-32B model, configured by the authors and prompted with their criteria, is assumed to rate narrative quality the way expert human readers would, and no human ratings are used anywhere in the benchmark.

Editorial extensions

If this is right

A 3B-active-parameter generator can match or exceed models with roughly 200x more active parameters on most measured narrative dimensions, so frontier-scale compute is not strictly necessary for competitive plot generation.
Preference alignment on a small number of high-confidence pairs (160) is enough to move a generalist MoE model substantially (+0.78 points), suggesting data quality matters more than data volume for creative alignment.
A single-pass, single-model generator can outperform multi-agent story-generation systems on coherence and pacing, removing orchestration overhead.
Both the reward model and the agentic evaluator separate critically acclaimed from critically panned screenplay plots, so aspect-level scores can serve as a reusable quality signal for filtering or ranking narrative text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Extension: if the evaluator-bias concern is resolved, the same three-component decomposition—aspect reward model, DPO-aligned MoE generator, independent agentic evaluator—could transfer to other long-horizon creative domains such as novel outlining, game narrative, or serialized episode arcs.
Extension: the aspect-rating reward model itself is a reusable artifact: a critic that can score any plot on five dimensions and could be deployed for drafting feedback before generation or for curating training data.
Extension: the fact that only 160 preference pairs drove the gain suggests an even leaner pipeline—for instance, replacing frontier-model candidates with the base MoE's own outputs plus human preference labels on a few dozen pairs—might replicate the result at near-zero API cost.
Extension: the quality-adaptive behavior (light refinement for excellent plots, near-full regeneration for weak ones) implies the model could serve as a controllable revision tool, although the paper does not expose a control mechanism for choosing the intervention level.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague —

The punchline: PlotTwist is a real system with a real idea buried in it, but the paper's headline — a 3B-active SLM beating Claude, GPT-4.1, and Gemini on plot quality — is not supported by the evidence as presented. Every number in Table 1 comes from a single Qwen-3-32B evaluator, prompted with criteria the authors wrote. The abstract calls it a 'cross-family jury'; Appendix D shows it is one model. That is the whole story.

What is genuinely new: the positive-negative prompting strategy for aspect rating is a small but legitimate idea. Separating 'what is good here' and 'what is bad here' and taking the difference is a sensible way to counter the positivity bias in LLM judges, and it is the kind of thing others will borrow. The full pipeline — aspect reward model, DPO on high-confidence preference pairs, agentic eval — is cleanly described and the ablations are informative. The GSAT/Razzie validation is a reasonable sanity check: the evaluator can tell great screenplays from terrible ones. That is not nothing.

Where it falls apart: the central comparative claim. The evaluator is the same model family as the generator and reward model (all Qwen-3), and the authors wrote both the training-time criteria and the evaluation-time criteria. No human ratings appear anywhere; §4.3 explicitly says human assessment is 'not feasible.' The test set of 160 premises is drawn from the same 5,000-premise pool used to build DPO pairs, and the paper never shows the test premises are disjoint from the training pairs. The differences in Table 1 are 0.08–0.17 points with standard deviations of 0.2–0.7; the paper's defense that 0.17×160=27.2 'broadly distributed gains' is not a statistical argument. And the GSAT/Razzie validation only shows the evaluator can separate extremes, not that its scores are interval-scaled or unbiased for outputs clustered near 8.5–9.0.

Minor: the abstract says ≤3B active parameters while the full-text abstract says ≤5B; pick one.

This is a well-built framework and an honest write-up, but the evidence for the flagship claim is circular. The fix is straightforward: human evaluation on a sample, an independent judge from a different family, and demonstrated test-set disjointness. With that, the central claim might well survive.

Referee Report

4 major / 4 minor

Summary. The paper introduces PlotTwist, a three-component framework for premise-conditioned plot generation using a small language model: (1) an Aspect Rating Reward Model trained with a positive-negative prompting scheme over five Narrative Quality Dimensions (NQDs); (2) a Qwen-3-30B-A3B mixture-of-experts generator aligned via Direct Preference Optimization (DPO) on 160 high-confidence preference pairs selected by the reward model; and (3) an Agentic Evaluation module that scores generated plots across the same five NQDs. The central claim is that PlotTwist, with 3B active parameters, outperforms frontier and specialized baselines on four of five NQDs (Table 1) and remains competitive on the fifth, and that both the reward model and the agentic evaluator separate acclaimed (GSAT) from panned (Razzie) screenplays.

Significance. If the head-to-head comparison were credible, the result would be significant: it would demonstrate that structured preference alignment can make a 3B-active-parameter model competitive with frontier systems in creative long-form generation, a practically relevant finding for efficient domain-specific alignment. The appendices are detailed and the prompting and compute setups are transparent. However, the headline claim rests on a single evaluator from the same model family as the generator, with no human validation; the reported differences are generally within one standard deviation; and the test set may overlap the DPO training pool. As a result, the significance of the claimed result is not currently established.

major comments (4)

[§4.3, §5.4, Appendix D] All Table 1 scores and win rates come from the Agentic Evaluation module, which Appendix D identifies as Qwen-3-32B (16-bit). This is the same model family as the generator (Qwen-3-30B-A3B) and the reward model (Qwen-3-32B), contradicting the abstract's claim of a 'cross-family jury.' §4.3 explicitly defers human assessment as infeasible, and §5.1's GSAT-vs-Razzie validation only shows coarse separation at far-apart quality extremes. It does not establish unbiased, interval-scaled scores in the 8.5–9.0 range where the claimed wins are 0.03–0.20 points. The central comparative claim is therefore unsupported.
[Table 1 / §5.4] The reported differences between PlotTwist and the next-best baseline are within one standard deviation for every aspect (e.g., pacing 8.85±0.29 vs 8.65±0.43; narrative coherence 8.89±0.39 vs 8.81±0.44; emotional turning points 8.98±0.21 vs 8.95±0.16). No pairwise significance tests, confidence intervals, or effect sizes are reported for the baseline comparison. The statement that PlotTwist 'outperforms all baselines on four of the five aspects' is not statistically supported by the numbers presented.
[§5.4 vs §4.2] The test set is described as '160 premises sampled from the total set of 5,000 premises considered in this work,' and the same 5,000-premise pool is used to construct DPO preference pairs. The paper does not demonstrate that the 160 test premises are disjoint from the DPO training premises or from the reward-model training plots. Without such a demonstration, the 'held-out' comparison and the resulting win rates may be optimistically biased.
[§4.2] DPO preference pairs are built from plots generated by frontier models (GPT-4.1, Claude Sonnet 4, Gemini 2.0 Flash) that 'achieve the highest reward score, exceeds a threshold of score 8, and outperforms the next-best model by a margin of at least 0.5.' The aligned SLM is therefore trained to imitate high-scoring frontier outputs, and the reward model used to select those outputs is trained on synthetic ratings from an LLM ensemble. The attribution of gains to 'structured preference-based alignment' is confounded with distillation of frontier outputs and with the reward model's own biases, weakening the paper's interpretational claim.

minor comments (4)

[Abstract vs Introduction] The full-text abstract says '≤5B active parameters' while the Introduction and footnote define SLMs as '≤3B active parameters.' These should be reconciled.
[§4.2] The text says 'We subsequently perform DPO on the Qwen-3-32B-A3B model'; the generator is elsewhere consistently Qwen-3-30B-A3B. Likely a typo, but it is confusing.
[Table 1] The column header 'Emotional Tuning' should be 'Emotional Turning Points' to match the NQD terminology used throughout the paper.
[Eq. (1) / §4.1] The aggregated rating in Eq. (1) sums raw per-model differences without normalization or scaling, giving a range of about −50 to +50. The paper does not specify how this target is scaled or centered before being used as the regression target in the Huber loss, which would aid reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

Same-family agentic evaluator undermines the independence of the headline comparison, but no equation-level circularity is present.

other [Abstract; §4.2–4.3; Appendix D; §5.4 (Table 1)]
"an Agentic Evaluation module using a cross-family jury for unbiased, independent post-hoc assessment (Abstract); Qwen-3-32B (16-bit) — PlotTwist Agentic Evaluator. Independent post-hoc evaluation module operating separately from the training pipeline (Appendix D); All generated plots are evaluated using our proposed Agentic Evaluation framework across the five NQDs (§5.4)."

The Abstract promises a cross-family jury, but Appendix D implements the evaluator as a single Qwen-3-32B. The reward model used to score DPO preference pairs is also Qwen-3-32B (4-bit), and the generator backbone is Qwen-3-30B-A3B—the same model family. The generator is DPO-trained to maximize the reward model's aspect scores; evaluating it with a same-family judge prompted on the same five NQDs closes the validation loop. No human ratings or cross-model evaluation is reported for the close Table 1 margins (0.08–0.17). The GSAT/Razzie check (§5.1) only validates extreme-contrast separation, not unbiased interval scoring in the 8.5–9.0 range. Thus the headline 'SLM beats frontier' result rests on an evaluator that is not independent in the relevant sense.

full rationale

No formal, equation-level circularity is present: the agentic evaluator is not literally the same function as the reward model, and the DPO training does not mathematically force the evaluator's rankings. There is also no load-bearing self-citation chain. The concern is evaluative independence: the paper calls the evaluator 'cross-family' and 'independent' while Appendix D shows it is a single Qwen-3-32B, the same family as the reward model and closely related to the generator. Since the DPO objective was derived from the reward model's ratings, and the final Table 1 scores come from this same-family judge, the central comparison is at risk of measuring alignment with the authors' own configured criteria rather than human narrative quality. The paper itself acknowledges that human assessment was 'not feasible' (§4.3), and the only external validation is the coarse GSAT/Razzie separation. These are serious validity concerns, but they do not reduce the derivation to a tautology; an independent evaluator could in principle rank the baselines differently. Score 4 reflects this partial, non-equational circularity in the evaluation loop rather than a fully constructed equivalence.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on unvalidated LLM-judge assumptions, hand-set thresholds, and same-family evaluation. No free parameters are fitted to external human labels, and no new physical or conceptual entity is introduced beyond the five NQD dimensions and the framework components.

free parameters (5)

DPO acceptance threshold = score ≥ 8
Hand-chosen in §4.2 to filter high-confidence preference pairs; directly controls the 160-sample training set and therefore the final model.
DPO margin threshold = ≥ 0.5
Hand-chosen margin between best and next-best reward score in §4.2; changes which frontier outputs become accepted plots.
Huber delta = δ = 1
Set in §4.1 for the reward-model regression loss; minor but hand-set.
Plot-length filter = ≤ 4000 words
Data curation threshold in §4.1 that changes the Razzie/GSAT pool and the training corpus composition.
NQD dimension set = five aspects
The choice of five narrative dimensions in §3 bounds what the reward model and evaluator measure; other dimensions could change results.

assumptions (5)

domain assumption The five NQDs are a valid, sufficient decomposition of plot quality
§3 grounds the set in affective narratology but provides no empirical evidence that these five dimensions capture human plot-quality judgments.
domain assumption LLM-generated positive-negative ratings are a faithful proxy for human aspect ratings
§4.1 trains the reward model on synthetic ensemble ratings; no human annotation validates these labels.
domain assumption The agentic evaluator approximates human critical judgment for comparing generated plots
§4.3 asserts this following prior work; Razzie/GSAT sensitivity is not a demonstration of unbiased pairwise comparison.
domain assumption Test premises are representative and disjoint from DPO training premises
§5.4 says 160 test premises were sampled from the total set of 5,000 but does not establish lack of overlap with the preference-pair construction process.
domain assumption DPO on 160 preference pairs transfers to diverse premises
§4.2 relies on prior work that DPO can work with small data; no scaling analysis is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PlotTwist: A Creative Plot Generation Framework with Small Language Models." pith.science (2026). https://pith.science/paper/CT7DYCOY

@misc{pith2026260316410,
  author       = {Pith},
  title        = {Pith review of: PlotTwist: A Creative Plot Generation Framework with Small Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CT7DYCOY}},
  note         = {Machine review of arXiv:2603.16410}
}

abstract

Creative plot generation presents a fundamental challenge for language models: transforming a concise premise into a coherent narrative that sustains global coherence, character development, pacing, tone consistency, and emotional progression. Although recent Large Language Models (LLMs) demonstrate strong fluency on general-purpose tasks, they require preference alignment to perform well on domain-specific tasks such as creative plot generation. However, conducting such alignment at the scale of frontier LLMs is computationally prohibitive, significantly limiting accessibility and practical deployment. To address this, we present PlotTwist, a structured framework that enables Small Language Models (SLMs) with $\leq$3B active parameters to generate high-quality, premise-conditioned plots competitive with frontier systems of vastly greater parameter scale. Our approach decomposes generation into three specialized components: (1) an Aspect Rating Reward Model, trained via a novel Positive-Negative prompting strategy; (2) a Mixture-of-Experts (MoE) plot generator aligned via Direct Preference Optimization (DPO); and (3) an Agentic Evaluation module using a cross-family jury for unbiased, independent post-hoc assessment. Extensive experiments demonstrate that PlotTwist consistently outperforms all baselines, including frontier models, across multiple Narrative Quality Dimensions (NQDs), achieving higher win rates against every baseline except the strongest, with which it remains competitive. Further validation confirms strong sensitivity to narrative quality, as the framework reliably distinguishes plots derived from critically acclaimed versus widely panned screenplays. Together, these results establish structured, preference-based alignment as a resource-efficient approach to high-quality creative plot generation. Project page: https://abhinavthorat.github.io/plottwist/

Figures

Figures reproduced from arXiv: 2603.16410 by the authors.

**Figure 1.** PlotTwist Framework dataset comprising plots paired with their corresponding ratings across the considered NQDs. We then fine-tune an LLM on this dataset to obtain an Aspect Rating Reward Model capable of assigning continuous-valued scores to plots. To formalize this setup, we introduce the required notation here. For a given plot-p and aspect-a, we use ra(p) to denote the rating of plot-p along aspect-a in NQD. Asp… view at source ↗

**Figure 2.** Left: Illustration of positive and negative prompting used to derive aspect-specific [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Dataset curation for performing DPO To perform DPO, we first construct a dataset of pairwise plot preferences, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

109 extracted references · 1 canonical work pages

[1]

https://www.anthropic.com/ research/claude-4(May 2025)

Anthropic: Claude 4 model card and system safety. https://www.anthropic.com/ research/claude-4(May 2025)

2025
[2]

In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

Chakrabarty, T., Laban, P., Agarwal, D., Muresan, S., Wu, C.S.: Art or artifice? large language models and the false promise of creativity. In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. pp. 1–34 (2024)

2024
[3]

arXiv preprint arXiv:2412.00001 (2024)

DeepMind, G.: Gemini 2.0: A generalist agent. arXiv preprint arXiv:2412.00001 (2024)

arXiv 2024
[4]

arXiv preprint arXiv:2501.12948 (2025)

DeepSeek-AI: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

arXiv 2025
[5]

arXiv preprint arXiv:2407.21783 (2024)

Dubey, A., Jauhri, A., Pandey, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

arXiv 2024
[6]

In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL)

Fan, A., Lewis, M., Dauphin, Y .: Hierarchical neural story generation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL). pp. 889–898 (2018)

2018
[7]

Journal of Machine Learning Research23(120), 1–39 (2022)

Fedus, W., Zoph, B., Shazeer, N.: Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research23(120), 1–39 (2022)

2022
[8]

arXiv preprint arXiv:2503.22828 (2025)

Gurung, A., Lapata, M.: Learning to reason for long-form story generation. arXiv preprint arXiv:2503.22828 (2025)

arXiv 2025

Show all 109 references

[9]

Acm transactions on interactive intelligent systems (tiis)5(4), 1–19 (2015)

Harper, F.M., Konstan, J.A.: The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis)5(4), 1–19 (2015)

2015
[10]

U of Nebraska Press (2011) 16 Abhinav et al

Hogan, P.C.: Affective narratology: The emotional structure of stories. U of Nebraska Press (2011) 16 Abhinav et al

2011
[11]

arXiv preprint arXiv:2410.02603 (2024)

Huot, F., Amplayo, R.K., Palomaki, J., Jakobovits, A.S., Clark, E., Lapata, M.: Agents’ room: Narrative generation through multi-step collaboration. arXiv preprint arXiv:2410.02603 (2024)

2024 arXiv
[12]

https://doi.org/10.3390/app15062971, https://www.mdpi.com/2076-3417/15/6/2971

Kim, S., Oh, D.: Evaluating creativity: Can llms be good evaluators in creative writing tasks? Applied Sciences15(6), 2971 (2025). https://doi.org/10.3390/app15062971, https://www.mdpi.com/2076-3417/15/6/2971

2025 doi
[13]

Kumar, V .B., Goyal, D., Eppa, A., Bhandari, N.: Curiosity-driven llm-as-a-judge for personal- ized creative judgment (2025),https://arxiv.org/abs/2510.05135

2025
[14]

Marioriyad, A., Rohban, M.H., Baghshah, M.S.: The silent judge: Unacknowledged shortcut bias in llm-as-a-judge (2025), https://arxiv.org/abs/2509.26072, presented at NeurIPS 2025 Reliable ML Workshop

2025
[15]

arXiv preprint arXiv:2501.00002 (2025)

Microsoft: Phi-4-mini technical report: Compact yet powerful multimodal language models. arXiv preprint arXiv:2501.00002 (2025)

2025 arXiv
[16]

https://openai.com/research/gpt-4-1 (April 2025)

OpenAI: Gpt-4.1 system card. https://openai.com/research/gpt-4-1 (April 2025)

2025
[17]

Advances in neural information processing systems35, 27730–27744 (2022)

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in neural information processing systems35, 27730–27744 (2022)

2022
[18]

Advances in neural information processing systems36, 53728–53741 (2023)

Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., Finn, C.: Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems36, 53728–53741 (2023)

2023
[19]

arXiv preprint arXiv:2004.14967 (2020)

Rashkin, H., Celikyilmaz, A., Choi, Y ., Gao, J.: Plotmachines: Outline-conditioned generation with dynamic plot state tracking. arXiv preprint arXiv:2004.14967 (2020)

2004 arXiv
[20]

See, A., Pappu, A., Saxena, R., Yerukola, A., Manning, C.D.: Do massively pretrained language models make better storytellers? arXiv preprint arXiv:1909.10705 (2019)

1909 arXiv
[21]

arXiv preprint arXiv:1701.06538 (2017)

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outra- geously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 (2017)

2017 arXiv
[22]

arXiv preprint arXiv:2306.11270 (2023)

Sun, J., Shaib, C., Wallace, B.C.: Evaluating the zero-shot robustness of instruction-tuned language models. arXiv preprint arXiv:2306.11270 (2023)

2023 arXiv
[23]

https://mistral.ai/news/ mistral-small-3/(January 2025)

Team, M.A.: Mistral small 3 (2501) release. https://mistral.ai/news/ mistral-small-3/(January 2025)

2025
[24]

arXiv preprint arXiv:2409.12191 (2024)

Team, Q.: Qwen2.5: A party of language models. arXiv preprint arXiv:2409.12191 (2024)

2024 arXiv
[25]

arXiv preprint arXiv:2504.00001 (April 2025)

Team, Q.: Qwen3 technical report. arXiv preprint arXiv:2504.00001 (April 2025)

2025 arXiv
[26]

In: Findings of the Association for Computational Linguistics: EMNLP 2025

Teleki, M., Bengali, V ., Dong, X., Janjur, S.T., Liu, H., Liu, T., Wang, C., Liu, T., Zhang, Y ., Shipman, F., et al.: A survey on llms for story generation. In: Findings of the Association for Computational Linguistics: EMNLP 2025. pp. 13954–13966 (2025)

2025
[27]

https://huggingface.co/TheBloke/ WizardLM-Uncensored-SuperCOT-StoryTelling-30B-GPTQ(2023)

TheBloke, Hartford, E., Kaiokendev: Wizardlm-uncensored-supercot- storytelling-30b-gptq. https://huggingface.co/TheBloke/ WizardLM-Uncensored-SuperCOT-StoryTelling-30B-GPTQ(2023)

2023
[28]

In: Proceedings of the AAAI Conference on Artificial Intelligence

Yao, L., Peng, N., Weischedel, R., Knight, K., Zhao, D., Yan, R.: Plan-and-write: Towards bet- ter automatic storytelling. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 7378–7385 (2019)

2019
[29]

Yu, F.: When ais judge ais: The rise of agent-as-a-judge evaluation for llms (2025),https: //arxiv.org/html/2508.02994v1

2025 arXiv
[30]

Advances in neural information processing systems36, 46595–46623 (2023)

Zheng, L., Chiang, W.L., Sheng, Y ., Zhuang, S., Wu, Z., Zhuang, Y ., Lin, Z., Li, Z., Li, D., Xing, E., et al.: Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems36, 46595–46623 (2023)

2023
[31]

Strict output rules:

Zheng, M., Song, D., Zhou, G., You, J., Zhan, J., Ma, X., Song, X., Lim, S.N., Chen, Q., Yang, H.: Cml-bench: A framework for evaluating and enhancing llm-powered movie scripts generation (2025),https://arxiv.org/abs/2510.06231 PlotTwist: Creative Plot Generation with Small La...

2025
[35]

Score generously. ### MoviePlot: { } Negative Prompt (r− a ) You are a professional movie critic whoseonlyoutput must be asingle JSON object with exactly one integer field (0–10): Narrative Coherence:Field Definition (NegativeFocus): Confusing storytelling, plot holes, inconsi...
[37]

Include onlyNarrative_Coherence
[39]

0 = no issues, 10 = severe issues. ### MoviePlot: { } Aspect 2: Emotional Turning Points Positive Prompt (r+ e ) You are a professional movie critic whoseonlyoutput must be asingle JSON object with exactly one integer field (0–10): Emotional Turning Points:Field Definition (Po...
[41]

18 Abhinav et al

Score generously. 18 Abhinav et al. ### MoviePlot: { } | ### Review: Negative Prompt (r− e ) You are a professional movie critic whoseonlyoutput must be asingle JSON object with exactly one integer field (0–10): Emotional Turning Points:Field Definition (Negative Focus): Flat ...
[42]

Include onlyEmotions_Turning_Points
[43]

0 = no issues, 10 = severe issues. ### MoviePlot: { } | ### Review: Aspect 3: Tone Consistency Positive Prompt (r+ t ) You are a professional movie critic whoseonlyoutput must be asingle JSON object with exactly one integer field (0–10): Tone Consistency:Field Definition (Posi...
[47]

Score generously. ### MoviePlot: { } | ### Review: Negative Prompt (r− t ) You are a professional movie critic whoseonlyoutput must be asingle JSON object with exactly one integer field (0–10): Tone Consistency:Field Definition (Negative Focus): Jarring mood shifts, inconsiste...
[49]

Include onlyTone_Consistency
[51]

0 = no issues, 10 = severe issues. PlotTwist: Creative Plot Generation with Small Language Models 19 ### MoviePlot: { } | ### Review: Aspect 4: Character Development Positive Prompt (r+ c ) You are a professional movie critic whoseonlyoutput must be asingle JSON object with ex...
[55]

Score generously. ### MoviePlot: { } | ### Review: Negative Prompt (r− c ) You are a professional movie critic whoseonlyoutput must be asingle JSON object with exactly one integer field (0–10): Character Development:Field Definition (Negative Focus): Weak or static character a...
[57]

Include onlyCharacter_Development
[59]

0 = no issues, 10 = severe issues. ### MoviePlot: { } | ### Review: Aspect 5: Pacing Positive Prompt (r+ p ) You are a professional movie critic whoseonlyoutput must be asingle JSON object with exactly one integer field (0–10): Pacing:Field Definition (Positive Focus): Effecti...
[62]

Score generously. ### MoviePlot: { } | ### Review: Negative Prompt (r− p ) You are a professional movie critic whoseonlyoutput must be asingle JSON object with exactly one integer field (0–10): Pacing:Field Definition (Negative Focus): Uneven or inconsistent pacing, excessive ...
[63]

Output only a valid JSON object
[64]

Integer value from 0 to 10
[65]

0 = no issues, 10 = severe issues. ### MoviePlot: { } | ### Review: PlotTwist: Creative Plot Generation with Small Language Models 21 B Agentic Evaluation System Prompts System Prompt 1: Narrative Coherence Evaluation Task Overview Evaluate a movie plot’s narrative structure a...
[66]

Causal Connectivity Logical beginning–middle–end flow

Plot Progression 2. Causal Connectivity Logical beginning–middle–end flow. Events arise naturally from prior ac- tions
[67]

Conflict Focus No plot holes or contradictions

Plot Integrity 4. Conflict Focus No plot holes or contradictions. A sustained central conflict drives the story. Character Integration (3 points)
[68]

Supporting Character Function 7

Protagonist Consistency 6. Supporting Character Function 7. Resolution Authenticity Narrative Flow and Unity (3 points)
[69]

Thematic Integration 10

Pacing Appropriateness 9. Thematic Integration 10. Tonal Consistency Output Format
[70]

Causal Connectivity: X.X

Plot Progression: X.X 2. Causal Connectivity: X.X
[71]

Conflict Focus: X.X

Plot Integrity: X.X 4. Conflict Focus: X.X
[72]

Protagonist Consistency: X.X
[73]

Supporting Character Function: X.X
[74]

Resolution Authenticity: X.X
[75]

Pacing Appropriateness: X.X
[76]

Thematic Integration: X.X
[77]

Tonal Consistency: X.X TOTAL: X.X/10 22 Abhinav et al. System Prompt 2: Emotional Turning Point Evaluation Task Overview Identify and evaluate the primary emotional turning point of the narrative using a 10-criteria framework focused on emotional impact and character change. S...
[78]

Character Believability Addresses or reframes central conflict

Conflict Resolution 2. Character Believability Addresses or reframes central conflict. Emotion aligns with established arc
[79]

Emotional Satisfaction Meaningful internal change

Character Transformation 4. Emotional Satisfaction Meaningful internal change. Emotionally resonant payoff. Narrative Construction (3 points)
[80]

Thematic Crystallization 7

Narrative Causality 6. Thematic Crystallization 7. Relationship Impact Technical & Structural Elements (3 points)
[81]

Structural Necessity 10

Cinematic Execution 9. Structural Necessity 10. Audience Alignment Output Format
[82]

Conflict Resolution: X.X
[83]

Character Believability: X.X
[84]

Character Transformation: X.X
[85]

Emotional Satisfaction: X.X
[86]

Narrative Causality: X.X
[87]

Thematic Crystallization: X.X
[88]

Relationship Impact: X.X
[89]

Cinematic Execution: X.X
[90]

Structural Necessity: X.X
[91]

Scoring Criteria Core Character Elements (4 points) PlotTwist: Creative Plot Generation with Small Language Models 23

Audience Alignment: X.X TOTAL: X.X/10 System Prompt 3: Character Development Evaluation Task Overview Evaluate protagonist character development using a 10-criteria framework assessing motivation, arc progression, and narrative function. Scoring Criteria Core Character Element...
[92]

Behavioral Consistency Clear goals and desires

Motivation Clarity 2. Behavioral Consistency Clear goals and desires. Actions align with personality
[93]

Psychological Depth Believable transformation

Character Arc 4. Psychological Depth Believable transformation. Emotional and psychological complex- ity. Character Foundation (3 points)
[94]

Audience Connection 7

Backstory Integration 6. Audience Connection 7. Character Distinctiveness Narrative Function (3 points)
[95]

Plot Agency 10

Relationship Dynamics 9. Plot Agency 10. Thematic Alignment Output Format
[96]

Behavioral Consistency: X.X

Motivation Clarity: X.X 2. Behavioral Consistency: X.X
[97]

Psychological Depth: X.X

Character Arc: X.X 4. Psychological Depth: X.X
[98]

Backstory Integration: X.X
[99]

Audience Connection: X.X
[100]

Character Distinctiveness: X.X
[101]

Relationship Dynamics: X.X
[102]

Thematic Alignment: X.X TOTAL: X.X/10 24 Abhinav et al

Plot Agency: X.X 10. Thematic Alignment: X.X TOTAL: X.X/10 24 Abhinav et al. System Prompt 4: Pacing Analysis Evaluation Task Overview Assess narrative pacing using a 10-criteria framework measuring rhythm, momentum, and emotional timing. Scoring Criteria
[103]

Structural Foundation

Premise Establishment Speed 2. Structural Foundation
[104]

Event Frequency

Pacing Consistency 4. Event Frequency
[105]

Tension Management

Scene Purposefulness 6. Tension Management
[106]

Emotional Beat Timing

Transition Quality 8. Emotional Beat Timing
[107]

Genre–Tone Alignment Output Format

Climax Timing 10. Genre–Tone Alignment Output Format
[108]

Premise Establishment Speed: X.X
[109]

Structural Foundation: X.X
[110]

Event Frequency: X.X

Pacing Consistency: X.X 4. Event Frequency: X.X
[111]

Scene Purposefulness: X.X
[112]

Tension Management: X.X
[113]

Emotional Beat Timing: X.X

Transition Quality: X.X 8. Emotional Beat Timing: X.X
[114]

Climax Timing: X.X 10. Genre-Tone Alignment: X.X TOTAL: X.X/10 System Prompt 5: Tone Consistency Evaluation Task Overview Evaluate tonal coherence using a 10-criteria framework assessing atmosphere, stylis- tic unity, and emotional continuity. Scoring Criteria
[115]

Scene-to-Scene Consistency

Initial Atmosphere Establishment 2. Scene-to-Scene Consistency
[116]

Earned Tone Shifts

Tonal Relief Integration 4. Earned Tone Shifts
[117]

Visual Reinforcement

Dialogue Style Consistency 6. Visual Reinforcement
[118]

Comedy/Drama Balance

Stakes Alignment 8. Comedy/Drama Balance
[119]

Motif and Symbol Unity Output Format PlotTwist: Creative Plot Generation with Small Language Models 25

Ending Consistency 10. Motif and Symbol Unity Output Format PlotTwist: Creative Plot Generation with Small Language Models 25
[120]

Initial Atmosphere Establishment: X.X
[121]

Scene-to-Scene Consistency: X.X
[122]

Tonal Relief Integration: X.X
[123]

Earned Tone Shifts: X.X
[124]

Dialogue Style Consistency: X.X
[125]

Visual Reinforcement: X.X
[126]

Comedy/Drama Balance: X.X

Stakes Alignment: X.X 8. Comedy/Drama Balance: X.X
[127]

Ending Consistency: X.X 10. Motif and Symbol Unity: X.X TOTAL: X.X/10 C Plot Generation with PlotTwist Plot Generation with PlotTwist Task Description The following example demonstrates long-form creative movie plot generation under thePlotTwistsetting. The task emphasizes nar...

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

https://www.anthropic.com/ research/claude-4(May 2025)

Anthropic: Claude 4 model card and system safety. https://www.anthropic.com/ research/claude-4(May 2025)

2025

[2] [2]

In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

Chakrabarty, T., Laban, P., Agarwal, D., Muresan, S., Wu, C.S.: Art or artifice? large language models and the false promise of creativity. In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. pp. 1–34 (2024)

2024

[3] [3]

arXiv preprint arXiv:2412.00001 (2024)

DeepMind, G.: Gemini 2.0: A generalist agent. arXiv preprint arXiv:2412.00001 (2024)

arXiv 2024

[4] [4]

arXiv preprint arXiv:2501.12948 (2025)

DeepSeek-AI: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

arXiv 2025

[5] [5]

arXiv preprint arXiv:2407.21783 (2024)

Dubey, A., Jauhri, A., Pandey, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

arXiv 2024

[6] [6]

In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL)

Fan, A., Lewis, M., Dauphin, Y .: Hierarchical neural story generation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL). pp. 889–898 (2018)

2018

[7] [7]

Journal of Machine Learning Research23(120), 1–39 (2022)

Fedus, W., Zoph, B., Shazeer, N.: Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research23(120), 1–39 (2022)

2022

[8] [8]

arXiv preprint arXiv:2503.22828 (2025)

Gurung, A., Lapata, M.: Learning to reason for long-form story generation. arXiv preprint arXiv:2503.22828 (2025)

arXiv 2025