{"id":"b2b5e014-e0b0-4c30-85c5-eaefcebf3e32","arxiv_id":"2504.14538","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"BookWorld builds multi-agent societies from novels and uses them to generate stories that an LLM judge prefers over direct generation and a prior screenwriting agent in most comparisons.","lead":"BookWorld converts novels into living multi-agent simulations in which AI characters act, remember, and interact inside a fictional world. The system aims to generate fan-style stories and interactive games that stay true to the original books.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'consistently outperforms' claim is contradicted by Table 1's Llama-3.3-70B row (win rates as low as 34.6%), and the 75.36% headline is undefined, so the central comparison is not established.","rationale":"The reader's weakest assumption focuses on LLM-judge reliability, which is a legitimate concern. However, the most load-bearing problem is internal to the paper's own reporting: the 'consistently outperforms' sentence is directly contradicted by Table 1 for Llama-3.3-70B, and the 75.36% headline is not operationally defined. This means the central comparative claim is unsupported even before considering judge bias. The concern is concrete and verifiable from the paper's own data, so it does not require new experiments. The reader's CONDITIONAL verdict remains appropriate: the paper has a promising system, but the empirical claim needs substantial clarification and per-model reporting before it can be accepted as stated. I partially agree with the reader because judge reliability is a secondary but real issue that would also need to be addressed after the internal contradiction is resolved.","tokens_in":13261,"tokens_out":4605,"duration_ms":37386,"concrete_test":"Open the released code and repository to find the exact pairwise comparison counts. Recompute the 75.36% headline from Table 1: if it is the unweighted mean of all 120 win-rate cells, recompute it with the Llama-3.3-70B row removed and observe the change. Then, using the reported pairwise counts, compute exact 95% binomial confidence intervals for the Llama-3.3-70B versus direct generation cells, especially WQ without script (34.6%). If the upper confidence bound for that cell is below 50%, or if the headline changes materially when Llama-3.3-70B is excluded, the central claim must be revised to exclude that model and to define the aggregation explicitly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 states that BookWorld 'consistently outperforms the direct generation baseline across all evaluated metrics and models,' but the very next sentence concedes that with Llama-3.3-70B BookWorld underperforms. Table 1 confirms this: for Llama-3.3-70B versus direct generation without a script, win rates are 34.6% (An), 42.3% (CF), 53.8% (IS), 34.6% (WQ), and 47.8% (Cr); five of the ten without-script cells are below 50%, and the with-script cells include 42.3% (SQ) and 46.2% (An). The abstract, introduction, and Section 4.3 all repeat 'win rate of 75.36%' without defining how it is computed: over which models, which baseline, whether with or without script, and whether it is an unweighted average of the 120 win-rate cells in Table 1. If it is an unweighted average, high win rates on seven models mask the systematic failure on Llama-3.3-70B, making the headline claim about 'surpassing previous methods' an artifact of model selection. This internal inconsistency is more direct than judge bias: even a perfect LLM judge would not rescue the claim as worded. The Limitations section does not mention this evaluation fragility, and Appendix A leaves the number of judged pairs Y unspecified, further preventing verification of whether any of the low Llama win rates are statistically distinguishable from chance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BookWorld, a multi-agent simulation system for story generation grounded in existing novels. It extracts character and worldview data from source texts, initializes role agents and a world agent, simulates scenes with spatial, environmental, and world-knowledge constraints, and rephrases simulation logs into novel-style stories. The authors claim that BookWorld outperforms direct generation and HoLLMwood baselines, reporting a win rate of 75.36% based on pairwise LLM-as-judge comparisons across five dimensions and seven base models, with a human-agreement validation reported in Appendix A.","tokens_in":13571,"tokens_out":4963,"duration_ms":38465,"significance":"If substantiated, the work would be a useful contribution to creative story generation and character-driven simulation: grounding agent societies in established fictional worldviews rather than synthetic personas is a timely and underexplored direction. The system design is plausible and reasonably complete, covering data extraction, a world agent, map-based movement, script/free modes, and retrieval-augmented memory, and the authors state that code is released. However, the significance is currently limited by evaluation weaknesses: the headline 75.36% figure is not defined, the 'consistently outperforms' claim is contradicted by the paper's own Table 1, and the evaluation protocol omits sample sizes, confidence intervals, and significance tests, relying on a single LLM judge with moderate human agreement.","major_comments":[{"comment":"The statement that 'BookWorld consistently outperforms the direct generation baseline across all evaluated metrics and models' is contradicted by Table 1. For Llama-3.3-70B without script, win rates versus direct generation are 34.6% (An), 42.3% (CF), 53.8% (IS), 34.6% (WQ), and 47.8% (Cr); with script, SQ is 42.3%. The abstract's 75.36% win rate is undefined: the paper never states which cells of Table 1 are aggregated, whether the baseline is direct generation or HoLLMwood, whether the with-script and without-script conditions are pooled, or how the average is weighted across the seven models. These claims need to be redefined and revised to match the data actually reported.","section":"Section 4.3 and Table 1"},{"comment":"The reliability of the LLM judge is load-bearing for all comparative conclusions. The paper does not report the number of comparison pairs per cell, the total number of stories generated, confidence intervals, or any significance tests. Appendix A reports Cohen's kappa values between 0.637 and 0.786, which is moderate-to-good agreement rather than the 'high level of consistency' claimed, and the sampled number of pairs Y is left unspecified. Please report exact N per condition, per-cell confidence intervals or exact binomial tests, and the value of Y; this is especially important for the win rates near 50%, which are currently indistinguishable from chance.","section":"Appendix A and Section 4.1"},{"comment":"The count of worldview settings is internally inconsistent. Section 3.4 states 9912 settings, with 44.8% Chinese and 55.2% English. Appendix B states 9,142 entries, with 4,449 from Chinese novels and 4,693 from English novels, and Table 5 sums to 9,142. The percentages in Section 3.4 do not match either total. These numbers must be reconciled.","section":"Section 3.4 versus Appendix B"},{"comment":"The Limitations section does not mention the fragility of the evaluation: the central comparative claims rest entirely on pairwise comparisons by a single LLM judge, with human validation only on an unspecified number of pairs and with only moderate-to-good agreement. The paper should explicitly acknowledge this limitation and discuss the implications for the low win-rate cells, such as those for Llama-3.3-70B, where the differences may not be statistically distinguishable from chance.","section":"Limitations"}],"minor_comments":[{"comment":"'e.t.c.' should be written as 'etc.'.","section":"Abstract and Section 1"},{"comment":"The environment-response prompt misspells 'Environment' as 'Enviroment' in two places.","section":"Table 8"},{"comment":"The sentence 'Each experiment simulates 2 to 4 scenes, averaging a total of 4,230 words' is ambiguous; please specify whether this is the average length per generated story and state the number of stories used in the evaluation.","section":"Section 4.2"},{"comment":"The row 'A Song of Ice and Fire (Part)' is marked as Chinese (zh), but the work is originally English; please clarify whether the extracted source was a Chinese translation and, if so, state this explicitly.","section":"Table 5"},{"comment":"The definition of the Anthropomorphism metric as 'attributing human characteristics to non-human entities while preserving their original nature' is unusual for story generation; please clarify what evaluators were asked to judge and provide the exact evaluation prompt.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears rushed: the headline 75.36% figure is undefined, the text's 'consistently outperforms' claim is contradicted by the paper's own Table 1, and the setting-count discrepancy between Section 3.4 and Appendix B suggests insufficient cross-checking. These issues are fixable with a careful revision, but the current version does not support the strong comparative claims. Please encourage the authors to report the evaluation sample sizes and statistical significance, define the aggregate win rate precisely, and reconcile the dataset numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: the system design is genuinely interesting, but the paper's headline numbers don't survive contact with its own Table 1. Don't repeat the '75.36% win rate' — it's undefined, and 'consistently outperforms the direct generation baseline' is false for Llama-3.3-70B.\n\nWhat's new: the end-to-end assembly of book-based agent societies — character extraction, worldview settings with term/nature/detail/source, a weighted-graph map with travel times, and a world agent that handles environment responses and event generation. That combination isn't in the cited work. The qualitative examples in Table 3 look plausible and suggest the system can do useful things like keeping character memory and adjusting to Solaris' twin suns. The worldview data itself (16 books, ~9,000 entries) is a useful resource, though note Section 3.4 says 9,912 while Appendix B says 9,142 — a carelessness that makes you check everything.\n\nThe soft spots are real and mostly in the evaluation. Table 1 shows Llama-3.3-70B losing to direct generation in five of ten without-script cells (e.g., 34.6% Anthropomorphism, 42.3% Character Fidelity). Section 4.3 does acknowledge underperformance later, but only after the 'consistently outperforms' claim in the same paragraph. The 75.36% headline is not tied to any definition: is it an average over all cells? Over seven models? With or without script? You can't trace it from the table. The judge is gpt-4o, validated against human raters with kappa 0.637–0.786, which is moderate-to-good, but the number of judged pairs Y is never stated, and there are no confidence intervals or significance tests. That's concerning because the Llama numbers look indistinguishable from chance.\n\nThe core idea — grounding agents in established fictional worlds with spatial and worldview constraints — is sound and worth pursuing. For fan fiction, interactive games, and social simulation, this is a natural approach. The paper is a good system paper but a weak claims paper: the empirical evidence supports 'BookWorld is a promising architecture' not 'BookWorld beats the baselines.'\n\nI'd send it to peer review with a request for major revision: fix the definition of the win rate, reconcile the claims with Table 1, add sample sizes and a breakdown of the Llama failure, and probably add a human evaluation on a defined subset. The pipeline is worth a reading group discussion, and I'd cite it for the system design and dataset, but not for the comparative results.","headline":"System is worth reading, but the 75.36% win rate is undefined and Table 1 contradicts the 'consistently outperforms' claim; send to review with major revision demands.","tokens_in":14137,"tokens_out":3609,"would_cite":true,"duration_ms":28583,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BookWorld's book-based agent societies beat direct generation and HoLLMwood in 75.36% of story comparisons.","keywords":["book-based agent society","multi-agent simulation","story generation","fictional world simulation","character fidelity","worldview extraction","LLM-as-judge evaluation","role-playing agents"],"falsifier":"Show the same story pairs to a panel of human readers blind to which system wrote them; if the panel's majority winner disagrees with the AI judge on more than about a quarter of the pairs, or if equalizing story length and formatting reverses the judge's preferences, then the 75.36% claim is falsified.","tokens_in":13043,"feed_emoji":"📚","tokens_out":12799,"duration_ms":102540,"temperature":0.7,"pith_summary":"This paper introduces BookWorld, a system that turns an existing novel into a living multi-agent society: each character becomes a role-playing agent with a static profile, evolving goals, memory, and a position on a spatial map, while a world agent manages geography, environment responses, worldview rules, and global events. Through scenes of character interaction, the simulation generates raw action logs, which an LLM then rephrases into a coherent novel-style story. The paper's central empirical claim is that these simulation-derived stories beat both direct LLM generation from the same book data and the HoLLMwood screenwriting pipeline in 75.36% of head-to-head comparisons, with the biggest advantages in immersion and character fidelity. If correct, this means established fictional worlds can be reopened as interactive, story-generating environments rather than being simulated from scratch.","feed_headline":"BookWorld agents win 75% of story matchups","feed_subtitle":"Simulating characters from books beats direct generation and HoLLMwood on immersion and character fidelity.","key_machinery":"The load-bearing mechanism is a two-tier agent architecture with role agents and a world agent, organized through scenes as the minimal narrative unit. Role agents hold static traits, dynamic goals and states, short-term memory, and retrieval-augmented long-term memory drawn from the original text; the world agent owns a discrete weighted graph map, location profiles, worldview settings, and environment-response generation. A scene is a bounded set of rounds in which an initiator chooses an action, targets reply, and the world agent reacts to environmental actions; travel consumes time units, so geographical constraints shape the narrative. The final LLM rephrasing pass turns structured action logs into novel-style prose.","core_discovery":"BookWorld's central claim is that a carefully extracted set of character profiles, worldview entries, map constraints, and scene-structured interactions is enough to let LLM-based agents re-animate a fictional world. The system does not write stories top-down; it lets characters act, react, remember, travel, and respond to environment feedback, and only then converts the resulting simulation log into narrative prose. On five quality axes—anthropomorphism, character fidelity, immersion and setting, writing quality, and storyline quality (or creativity in free mode)—BookWorld is reported to outperform direct generation on every metric across all evaluated base models, and to outperform HoLLMwood specifically on immersion and character fidelity. The authors also report that the scene module, environment responses, and worldview settings each contribute to the quality gain, with environment feedback most affecting immersion and the scene module most affecting storyline adherence.","pith_inferences":["Editorial inference: the character-fidelity advantage likely owes more to retrieval of original dialogue snippets than to the long-term memory summarizer; an ablation that keeps verbatim excerpt retrieval while removing memory would isolate the real source.","Editorial inference: the LLM judge may be rewarding output that looks more like published prose, since BookWorld's rephrasing pass produces polished narrative text while direct generation often yields plainer responses; a length- and format-controlled comparison is needed to confirm the 75.36% margin is about story substance.","Editorial inference: the scene-based time model and map constraints already resemble episode structure, so the architecture should transfer to screenplays, game lore, and serialized web fiction; the paper's own Werewolf limitation suggests social-deduction rules, not narrative genres, are the hard boundary.","Editorial inference: script mode could turn BookWorld into a human-AI co-writing tool, but the paper does not measure whether user interventions improve or harm the final story, making that a natural next experiment."],"forward_implications":["Any sufficiently text-rich novel can be converted into an interactive society: the extraction pipeline produced 9,912 worldview settings from 16 novels and 453 character/act presets without manual worldbuilding.","User-supplied scripts can steer the simulation along a chosen outline while agents still improvise within it, opening a direct route to interactive games and controllable fan-fiction generation.","The approach is model-dependent: with Llama-3.3-70B, BookWorld loses ground to baselines on some dimensions, so the benefit of simulation grows with the base model's instruction-following ability.","Because simulation histories are stored as retrievable memories, the same agents can be reused across separate story runs, making long-running serialized or interactive narratives feasible.","Removing any of the three structural modules—scene organization, environment feedback, or extracted worldview settings—degrades at least one quality dimension, so the architecture's gains are compositional rather than coming from a single prompt trick."],"supporting_citations":[{"why":"Supplies the generative-agent memory design (short-term and long-term memory with retrieval) that BookWorld adapts for role agents.","marker":"(Park et al., 2023)"},{"why":"Defines HoLLMwood, the screenwriting baseline that BookWorld is compared against in pairwise evaluation.","marker":"(Chen et al., 2024)"},{"why":"Provides the incremental character-information extraction method that BookWorld's data-preparation pipeline builds on.","marker":"(Yuan et al., 2024)"},{"why":"Supplies the retrieval-augmented role-playing prompt architecture that BookWorld uses for role agents.","marker":"(Li et al., 2023)"},{"why":"Supplies the dramatic-theory notion of the scene as the minimal narrative unit that organizes the simulation structure.","marker":"(McKee, 1999)"}],"fun_headline_variants":["BookWorld brings novels to life with 75% win rate","Simulating book characters beats direct story generation","BookWorld: agent societies from fiction win 75% of matchups","Novel-based multi-agent simulation wins 75% of story tests","BookWorld reanimates fictional worlds for creative stories"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the AI judge used for pairwise story comparison is actually measuring story quality; the paper validates that judge against a small set of human readers with only moderate agreement, so if the judge has hidden stylistic preferences, the 75.36% win rate and every comparative conclusion collapse.","fun_headline_variants_meta":{"raw":{"variants":["BookWorld brings novels to life with 75% win rate","Simulating book characters beats direct story generation","BookWorld: agent societies from fiction win 75% of matchups","Novel-based multi-agent simulation wins 75% of story tests","BookWorld reanimates fictional worlds for creative stories"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1469,"prompt_tokens":875,"completion_tokens":594,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":511}},"tokens_in":491,"tokens_out":594,"duration_ms":5315,"temperature":1.0,"reasoning_tokens":511,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:45:50.469512+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Show the same story pairs to a panel of human readers blind to which system wrote them; if the panel's majority winner disagrees with the AI judge on more than about a quarter of the pairs, or if equalizing story length and formatting reverses the judge's preferences, then the 75.36% claim is falsified.","supporting_citations":[],"review_version":1}