Pith. sign in

REVIEW 4 major objections 5 minor 28 references

BookWorld: From Novels to Interactive Agent Societies for Creative Story Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read BookWorld's book-based agent societies beat direct generation and HoLLMwood in 75.36% of story comparisons.

desk verdict System is worth reading, but the 75.36% win rate is undefined and Table 1 contradicts the 'consistently outperforms' claim; send to review with major revision demands. read the letter →

arxiv 2504.14538 v1 pith:4475O64W submitted 2025-04-20 cs.CL

classification cs.CL
keywords book-basedagentsocietymulti-agentsimulationstorygenerationfictionalworldcharacterfidelityworldviewextractionLLM-as-judgeevaluationrole-playingagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces BookWorld, a system that turns an existing novel into a living multi-agent society: each character becomes a role-playing agent with a static profile, evolving goals, memory, and a position on a spatial map, while a world agent manages geography, environment responses, worldview rules, and global events. Through scenes of character interaction, the simulation generates raw action logs, which an LLM then rephrases into a coherent novel-style story. The paper's central empirical claim is that these simulation-derived stories beat both direct LLM generation from the same book data and the HoLLMwood screenwriting pipeline in 75.36% of head-to-head comparisons, with the biggest advantages in immersion and character fidelity. If correct, this means established fictional worlds can be reopened as interactive, story-generating environments rather than being simulated from scratch.

What carries the argument

The load-bearing mechanism is a two-tier agent architecture with role agents and a world agent, organized through scenes as the minimal narrative unit. Role agents hold static traits, dynamic goals and states, short-term memory, and retrieval-augmented long-term memory drawn from the original text; the world agent owns a discrete weighted graph map, location profiles, worldview settings, and environment-response generation. A scene is a bounded set of rounds in which an initiator chooses an action, targets reply, and the world agent reacts to environmental actions; travel consumes time units, so geographical constraints shape the narrative. The final LLM rephrasing pass turns structured action logs into novel-style prose.

What would settle it

Show the same story pairs to a panel of human readers blind to which system wrote them; if the panel's majority winner disagrees with the AI judge on more than about a quarter of the pairs, or if equalizing story length and formatting reverses the judge's preferences, then the 75.36% claim is falsified.

Watch

Extended reading notes

Core claim

BookWorld's central claim is that a carefully extracted set of character profiles, worldview entries, map constraints, and scene-structured interactions is enough to let LLM-based agents re-animate a fictional world. The system does not write stories top-down; it lets characters act, react, remember, travel, and respond to environment feedback, and only then converts the resulting simulation log into narrative prose. On five quality axes—anthropomorphism, character fidelity, immersion and setting, writing quality, and storyline quality (or creativity in free mode)—BookWorld is reported to outperform direct generation on every metric across all evaluated base models, and to outperform HoLLMwood specifically on immersion and character fidelity. The authors also report that the scene module, environment responses, and worldview settings each contribute to the quality gain, with environment feedback most affecting immersion and the scene module most affecting storyline adherence.

Load-bearing premise

The load-bearing premise is that the AI judge used for pairwise story comparison is actually measuring story quality; the paper validates that judge against a small set of human readers with only moderate agreement, so if the judge has hidden stylistic preferences, the 75.36% win rate and every comparative conclusion collapse.

Editorial extensions

If this is right

  • Any sufficiently text-rich novel can be converted into an interactive society: the extraction pipeline produced 9,912 worldview settings from 16 novels and 453 character/act presets without manual worldbuilding.
  • User-supplied scripts can steer the simulation along a chosen outline while agents still improvise within it, opening a direct route to interactive games and controllable fan-fiction generation.
  • The approach is model-dependent: with Llama-3.3-70B, BookWorld loses ground to baselines on some dimensions, so the benefit of simulation grows with the base model's instruction-following ability.
  • Because simulation histories are stored as retrievable memories, the same agents can be reused across separate story runs, making long-running serialized or interactive narratives feasible.
  • Removing any of the three structural modules—scene organization, environment feedback, or extracted worldview settings—degrades at least one quality dimension, so the architecture's gains are compositional rather than coming from a single prompt trick.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the character-fidelity advantage likely owes more to retrieval of original dialogue snippets than to the long-term memory summarizer; an ablation that keeps verbatim excerpt retrieval while removing memory would isolate the real source.
  • Editorial inference: the LLM judge may be rewarding output that looks more like published prose, since BookWorld's rephrasing pass produces polished narrative text while direct generation often yields plainer responses; a length- and format-controlled comparison is needed to confirm the 75.36% margin is about story substance.
  • Editorial inference: the scene-based time model and map constraints already resemble episode structure, so the architecture should transfer to screenplays, game lore, and serialized web fiction; the paper's own Werewolf limitation suggests social-deduction rules, not narrative genres, are the hard boundary.
  • Editorial inference: script mode could turn BookWorld into a human-AI co-writing tool, but the paper does not measure whether user interventions improve or harm the final story, making that a natural next experiment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces BookWorld, a multi-agent simulation system for story generation grounded in existing novels. It extracts character and worldview data from source texts, initializes role agents and a world agent, simulates scenes with spatial, environmental, and world-knowledge constraints, and rephrases simulation logs into novel-style stories. The authors claim that BookWorld outperforms direct generation and HoLLMwood baselines, reporting a win rate of 75.36% based on pairwise LLM-as-judge comparisons across five dimensions and seven base models, with a human-agreement validation reported in Appendix A.

Significance. If substantiated, the work would be a useful contribution to creative story generation and character-driven simulation: grounding agent societies in established fictional worldviews rather than synthetic personas is a timely and underexplored direction. The system design is plausible and reasonably complete, covering data extraction, a world agent, map-based movement, script/free modes, and retrieval-augmented memory, and the authors state that code is released. However, the significance is currently limited by evaluation weaknesses: the headline 75.36% figure is not defined, the 'consistently outperforms' claim is contradicted by the paper's own Table 1, and the evaluation protocol omits sample sizes, confidence intervals, and significance tests, relying on a single LLM judge with moderate human agreement.

major comments (4)
  1. [Section 4.3 and Table 1] The statement that 'BookWorld consistently outperforms the direct generation baseline across all evaluated metrics and models' is contradicted by Table 1. For Llama-3.3-70B without script, win rates versus direct generation are 34.6% (An), 42.3% (CF), 53.8% (IS), 34.6% (WQ), and 47.8% (Cr); with script, SQ is 42.3%. The abstract's 75.36% win rate is undefined: the paper never states which cells of Table 1 are aggregated, whether the baseline is direct generation or HoLLMwood, whether the with-script and without-script conditions are pooled, or how the average is weighted across the seven models. These claims need to be redefined and revised to match the data actually reported.
  2. [Appendix A and Section 4.1] The reliability of the LLM judge is load-bearing for all comparative conclusions. The paper does not report the number of comparison pairs per cell, the total number of stories generated, confidence intervals, or any significance tests. Appendix A reports Cohen's kappa values between 0.637 and 0.786, which is moderate-to-good agreement rather than the 'high level of consistency' claimed, and the sampled number of pairs Y is left unspecified. Please report exact N per condition, per-cell confidence intervals or exact binomial tests, and the value of Y; this is especially important for the win rates near 50%, which are currently indistinguishable from chance.
  3. [Section 3.4 versus Appendix B] The count of worldview settings is internally inconsistent. Section 3.4 states 9912 settings, with 44.8% Chinese and 55.2% English. Appendix B states 9,142 entries, with 4,449 from Chinese novels and 4,693 from English novels, and Table 5 sums to 9,142. The percentages in Section 3.4 do not match either total. These numbers must be reconciled.
  4. [Limitations] The Limitations section does not mention the fragility of the evaluation: the central comparative claims rest entirely on pairwise comparisons by a single LLM judge, with human validation only on an unspecified number of pairs and with only moderate-to-good agreement. The paper should explicitly acknowledge this limitation and discuss the implications for the low win-rate cells, such as those for Llama-3.3-70B, where the differences may not be statistically distinguishable from chance.
minor comments (5)
  1. [Abstract and Section 1] 'e.t.c.' should be written as 'etc.'.
  2. [Table 8] The environment-response prompt misspells 'Environment' as 'Enviroment' in two places.
  3. [Section 4.2] The sentence 'Each experiment simulates 2 to 4 scenes, averaging a total of 4,230 words' is ambiguous; please specify whether this is the average length per generated story and state the number of stories used in the evaluation.
  4. [Table 5] The row 'A Song of Ice and Fire (Part)' is marked as Chinese (zh), but the work is originally English; please clarify whether the extracted source was a Chinese translation and, if so, state this explicitly.
  5. [Section 4.1] The definition of the Anthropomorphism metric as 'attributing human characteristics to non-human entities while preserving their original nature' is unusual for story generation; please clarify what evaluators were asked to judge and provide the exact evaluation prompt.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: BookWorld is an empirical systems paper, with only a minor non-load-bearing self-citation and evaluation-reliability caveats, not a circular argument.

full rationale

The paper is an empirical systems paper, not a derivation: Section 3 presents a pipeline (data extraction, initialization, simulation, rephrasing) and Section 4 reports pairwise LLM-judged win rates against direct generation and HoLLMwood. There are no equations in which an output is constructed from its own target, and the claimed 75.36% win rate is a summary statistic over Table 1, not a parameter fitted to the evaluation data. The only self-citation is in Section 3.4: 'We provide an automatic extraction method based on incremental updating inspired by Yuan et al. (2024).' Yuan et al. (2024) shares two co-authors with this paper, but the citation is used only as methodological inspiration for a preprocessing component, not as the load-bearing justification for the central claim of story-quality superiority, and the evaluation is against external baselines. Appendix A validates the gpt-4o judge against five human annotators with Cohen's kappa 0.637-0.786; the unspecified sample size Y and the Section 4.3 claim that BookWorld 'consistently outperforms the direct generation baseline across all evaluated metrics and models' contradicted by Table 1's Llama-3.3-70B row are empirical validity problems, not circularity. No circular step is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on experimental settings and domain assumptions about LLM behavior and evaluation validity rather than on fitted mathematical constants. The most consequential choices are the judge model, the unreported sample sizes, and the self-cited extraction method.

free parameters (4)
  • Judge model = gpt-4o-2024-08-06
    All pairwise comparisons and ablations are scored by this single model; the claimed win rates depend on its preferences.
  • Number of dialogue turns = not specified
    The paper states the number is kept consistent across methods but never reports it, affecting the amount of generated content compared.
  • Evaluation sample size = not reported
    The number of story pairs used for each win-rate cell is not stated, so confidence intervals and significance tests cannot be computed.
  • Scenes per simulation = 2 to 4
    The short simulation length may favor coherence; no ablation tests the effect of scene count on the central claim.
assumptions (4)
  • domain assumption LLMs can faithfully role-play established characters when given a profile and retrieved excerpts.
    The entire system depends on this; the paper provides no systematic check of role-play fidelity beyond win rates.
  • domain assumption The automatic extraction method captures sufficient character and worldview information from source books.
    The extraction method is from the authors' prior work (Yuan et al., 2024), self-cited, and no human evaluation of extraction quality is reported.
  • domain assumption Pairwise LLM judgment is a reliable proxy for story quality, as calibrated by human agreement.
    Cohen's kappa between 0.637 and 0.786 indicates only moderate to good agreement on a small, unspecified sample.
  • domain assumption The discrete map and scene-based structure adequately represent the fictional world's geography and narrative cohesion.
    The paper's own Limitations state that the representation is simplified and cannot handle certain scenarios (e.g., Werewolf), so the coverage of 'comprehensive real-world intricacies' is limited.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BookWorld: From Novels to Interactive Agent Societies for Creative Story Generation." pith.science (2026). https://pith.science/paper/4475O64W

@misc{pith2026250414538,
  author       = {Pith},
  title        = {Pith review of: BookWorld: From Novels to Interactive Agent Societies for Creative Story Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4475O64W}},
  note         = {Machine review of arXiv:2504.14538}
}
read the original abstract

Recent advances in large language models (LLMs) have enabled social simulation through multi-agent systems. Prior efforts focus on agent societies created from scratch, assigning agents with newly defined personas. However, simulating established fictional worlds and characters remain largely underexplored, despite its significant practical value. In this paper, we introduce BookWorld, a comprehensive system for constructing and simulating book-based multi-agent societies. BookWorld's design covers comprehensive real-world intricacies, including diverse and dynamic characters, fictional worldviews, geographical constraints and changes, e.t.c. BookWorld enables diverse applications including story generation, interactive games and social simulation, offering novel ways to extend and explore beloved fictional works. Through extensive experiments, we demonstrate that BookWorld generates creative, high-quality stories while maintaining fidelity to the source books, surpassing previous methods with a win rate of 75.36%. The code of this paper can be found at the project page: https://bookworld2025.github.io/.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 6 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Altera AL, Andrew Ahn, Nic Becker, Stephanie Carroll, Nico Christie, Manuel Cortes, Arda Demirci, Melissa Du, Frankie Li, Shuying Luo, et al. 2024. Project sid: Many-agent simulations toward ai civilization. arXiv preprint arXiv:2411.00114

  4. [4]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201

  5. [5]

    Jing Chen, Xinyu Zhu, Cheng Yang, Chufan Shi, Yadong Xi, Yuxiang Zhang, Junjie Wang, Jiashu Pu, Rongsheng Zhang, Yujiu Yang, et al. 2024. Hollmwood: Unleashing the creativity of large language models in screenwriting via role playing. arXiv preprint arXiv:2406.11683

  6. [6]

    Yun-Shiuan Chuang, Nikunj Harlalka, Siddharth Suresh, Agam Goyal, Robert Hawkins, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy T Rogers. 2024. The wisdom of partisan crowds: Comparing collective intelligence in humans and llm-based agents. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46

  7. [7]

    Gordon Dai, Weijia Zhang, Jinhan Li, Siqi Yang, Srihas Rao, Arthur Caetano, Misha Sra, et al. 2024. Artificial leviathan: Exploring social evolution of llm agents through the lens of hobbesian social contract theory. arXiv preprint arXiv:2406.14373

  8. [8]

    Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. 2020. Content planning for neural story generation with aristotelian rescoring. arXiv preprint arXiv:2009.09870

Show all 28 references
  1. [9]

    Carlos G \'o mez-Rodr \' guez and Paul Williams. 2023. A confederacy of models: A comprehensive evaluation of llms on creative writing. arXiv preprint arXiv:2310.08433

  2. [10]

    Senyu Han, Lu Chen, Li-Min Lin, Zhengshan Xu, and Kai Yu. 2024. Ibsen: Director-actor agent collaboration for controllable and interactive drama script generation. arXiv preprint arXiv:2407.01093

  3. [11]

    Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. 2023. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010

  4. [12]

    Mete Ismayilzada, Claire Stevenson, and Lonneke van der Plas. 2024. Evaluating creative short story generation in humans and large language models. arXiv preprint arXiv:2411.02316

  5. [13]

    Max Kreminski, Melanie Dickinson, and Michael Mateas. 2021. Winnow: a domain-specific language for incremental story sifting. In Proceedings of the AAAI conference on artificial intelligence and interactive digital entertainment, volume 17, pages 156--163

  6. [14]

    Michael Lebowitz. 1984. Creating characters in a story-telling universe. Poetics, 13(3):171--194

  7. [15]

    Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, et al. 2023. Chatharuhi: Reviving anime character in reality via large language model. arXiv preprint arXiv:2308.09597

  8. [16]

    R. McKee. 1999. https://books.google.com.hk/books?id=nEadQgAACAAJ Story: Substance, Structure, Style, and the Principles of Screenwriting . Methuen film. Methuen

  9. [17]

    James R Meehan. 1977. Tale-spin, an interactive program that writes stories. In Ijcai, volume 77, pages 91--98

  10. [18]

    Piotr Mirowski, Kory W Mathewson, Jaylen Pittman, and Richard Evans. 2023. Co-writing screenplays and theatre scripts with language models: Evaluation by industry professionals. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1--34

  11. [19]

    Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1--22

  12. [20]

    Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, and Nanyun Peng. 2024. Are large language models capable of generating human-level narratives? arXiv preprint arXiv:2407.13248

  13. [21]

    Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. 2023 a . Avalon's game of thoughts: Battle against deception through recursive contemplation. arXiv preprint arXiv:2310.01320

  14. [22]

    Yuxin Wang, Jieru Lin, Zhiwei Yu, Wei Hu, and B \"o rje F Karlsson. 2023 b . Open-world story generation with structured knowledge enhancement: A comprehensive survey. Neurocomputing, page 126792

  15. [23]

    Weiqi Wu, Hongqiu Wu, Lai Jiang, Xingyuan Liu, Jiale Hong, Hai Zhao, and Min Zhang. 2024. From role-play to drama-interaction: An llm solution. arXiv preprint arXiv:2405.14231

  16. [24]

    Ziyi Yang, Zaibin Zhang, Zirui Zheng, Yuxian Jiang, Ziyue Gan, Zhiyu Wang, Zijian Ling, Jinsong Chen, Martz Ma, Bowen Dong, Prateek Gupta, Shuyue Hu, Zhenfei Yin, Guohao Li, Xu Jia, Lijun Wang, Bernard Ghanem, Huchuan Lu, Chaochao Lu, Wanli Ouyang, Yu Qiao, Philip Torr, and Ji...

  17. [25]

    Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7378--7385

  18. [26]

    Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: story writing with large language models. In Proceedings of the 27th International Conference on Intelligent User Interfaces, pages 841--852

  19. [27]

    Xinfeng Yuan, Siyu Yuan, Yuhan Cui, Tianhe Lin, Xintao Wang, Rui Xu, Jiangjie Chen, and Deqing Yang. 2024. Evaluating character understanding of large language models via character profiling from fictional works. arXiv preprint arXiv:2404.12726

  20. [28]

    Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. 2023. Sotopia: Interactive evaluation for social intelligence in language agents. arXiv preprint arXiv:2310.11667

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.