REVIEW 2 major objections 5 minor 59 references
Seduced by the Narrative: Assessing Rule Adherence in Semi-Open Textual Sandboxes
T0 review · 2 major / 5 minor · reviewed 2026-07-12 · grok-4.5
Pith's one-line read Neither model scale nor explicit reasoning protects LLM adjudicators from rhetorical attacks that skip required dice rolls in narrative games.
desk verdict Solid new diagnostic benchmark: Rhetorical Injection + CoC-Seduce cleanly shows Pseudo-Logic and cultural-setting gaps beat scale/CoT for rule adjudication. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Rhetorical Injection: each player statement is the semantic composition of a fixed mechanical intent with one of four styles (Neutral, Authority, Pseudo-Logic, Omission). Success is measured by Failure Rate against the binary ground-truth function V that says whether the scenario truth objectively requires a dice roll.
What would settle it
Have an independent panel of Keepers re-label a stratified sample of the 5,376 scenarios; if Failure Rates under the new labels collapse or reverse the Pseudo-Logic dominance and the Ancient-China elevation, the central claim that rhetorical style (not annotation noise) drives the observed failures is falsified.
Extended reading notes
Core claim
Across twenty frontier models evaluated on 5,376 CoC-Seduce samples, neither greater scale nor explicit chain-of-thought reasoning reliably lowers adjudication failure rates. Models err almost exclusively by granting false automatic success rather than demanding unnecessary rolls; Pseudo-Logic framing produces the highest average failure (17.3 percent), and difficulty tracks cultural and temporal familiarity of the world setting more than the mechanical complexity of the skill itself.
Load-bearing premise
Expert human Keepers can assign a high-confidence binary ground truth for whether a dice roll is required from the scenario truth alone, even though some physically grounded narrations remain subtly ambiguous.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CoC-Seduce, a multi-agent adversarial benchmark for evaluating LLM rule adherence as adjudicators in semi-open textual sandboxes, instantiated via Call of Cthulhu TRPG mechanics. It formalizes Rhetorical Injection (NEUTRAL, AUTHORITY, PSEUDO-LOGIC, OMISSION) over a binary validity function V(C, St) that decides whether a player action requires a dice roll. Three frontier generators produce 5,376 samples across 16 non-social skills and four world settings; 20 target models are evaluated zero-shot under a unified system prompt with Failure Rate (FP/FC) and Wrong Skill metrics. Main empirical claims: average FP ≈ 9.58%, PSEUDO-LOGIC is the strongest attack (family-average FR ≈ 17.3%), neither scale nor explicit reasoning reliably improves robustness, and failures concentrate in culturally less familiar settings (Ancient China, Wilderness) rather than rule complexity alone.
Significance. If the results hold, the work supplies a concrete, reproducible diagnostic for a practically important failure mode—narrative sycophancy overriding explicit mechanical rules—that standard instruction-following and LLM-as-judge benchmarks do not capture. Strengths include a large multi-generator corpus, dual-Keeper human curation with a third-Keeper 10% check, transparent breakdowns by style/generator/setting (Tables 3, 6–8; Figs. 3–5), near-zero False Check rates that establish directional bias, and public project page. The finding that reasoning-enhanced models and larger/newer models do not monotonically improve adjudication integrity is a useful negative result for agentic deployment.
major comments (2)
- The central claim rests on high-confidence binary V for the 16 non-social skills (Secs. 3.1, 4.1). Limitations correctly notes residual interpretive ambiguity between ordinary narration and implicit justification. The dual-Keeper protocol plus 10% third-Keeper check is described, but no inter-annotator agreement statistic (e.g., Cohen’s κ or raw agreement on V and skill labels) is reported. Without that number it is hard to quantify how much of the 3.82% Neutral baseline FR is irreducible annotation noise versus model error; adding IAA would strengthen the load-bearing ground-truth claim without changing the design.
- Home-series generator–target effects (Fig. 3, §5.2) are large and asymmetric (GPT resists GPT attacks; Gemini is more vulnerable to Gemini attacks). The paper treats them as empirical phenomena, which is appropriate, but does not fully isolate whether the main PSEUDO-LOGIC dominance and setting effects survive after conditioning on generator family. A short stratified re-analysis (or leave-one-generator-out averages) would confirm that the strongest claim is not an artifact of the most effective generator (Gemini).
minor comments (5)
- Fig. 1 caption and body text both cite “(OpenAI, 2026)” twice; clean the duplicate.
- Table 3 caption and body use both “PSEUDO-LOGIC” and “Pseudo-Logic”; standardize casing and the small-caps macro.
- Eq. (2) defines FRoverall with N_V=1 + N_V=0 in the denominator; a one-sentence reminder that FC is evaluated only on PHY/INV would avoid reader confusion when comparing columns.
- Appendix Tables 6–8 are dense; a short note on how many cells are exactly 0.0% (especially 1920s Urban) would help readers interpret the heat-map pattern without re-counting.
- Related-work citations to concurrent LLM-as-judge and agent-as-judge surveys (Li 2026, Huang 2025, Zhuge 2025) are useful; a single sentence clarifying how CoC-Seduce differs from pure instruction-following suites (IFBench, JudgeBench) would sharpen the positioning.
Circularity Check
No circularity: CoC-Seduce is an empirical measurement against independent human Keeper labels, not a derivation that redefines its target.
full rationale
The paper constructs a benchmark (5,376 samples) by having three generator models produce player statements, then has expert Keepers independently assign binary ground-truth V(C, St) ∈ {0,1} via dual review plus a 10% third-Keeper audit (Secs. 3.1, 4.1, 4.5). Target adjudicators are scored solely by match to those fixed human labels (FR, FP, FC, WS). There is no fitted parameter later called a prediction, no uniqueness theorem imported from the authors, no ansatz smuggled via self-citation, and no redefinition of the measured quantity in terms of the models under test. Home-series generator–target effects and setting-dependent knowledge gaps are reported as empirical observations, not used to force the main claim. Residual generator-family overlap is transparent and analyzed, not load-bearing. The derivation chain is therefore self-contained measurement against external human adjudication.
Assumptions & free parameters
free parameters (2)
- Provider-default generation hyperparameters for all 20 adjudicators
- Scenario counts per skill/setting (6 truths × 4 styles for V=1; 2 truths × 4 styles for V=0 PHY/INV)
assumptions (4)
- domain assumption For the 16 selected non-social CoC skills, whether an action requires a dice roll is an objective function of scenario truth and rules, not Keeper taste.
- domain assumption Rhetorical style R_style can be composed with fixed intent I_intent without changing the underlying mechanical necessity of a roll.
- domain assumption Call of Cthulhu 7th Edition skill-check practice (roll iff meaningful risk/difficulty) is the correct adjudication standard for the benchmark.
- ad hoc to paper Zero-shot single-round adjudication on summarized context is a valid probe of rule adherence under narrative pressure.
invented entities (2)
-
Rhetorical Injection
independent evidence
-
CoC-Seduce benchmark corpus
independent evidence
Cite this review
Pith. "Pith review of Seduced by the Narrative: Assessing Rule Adherence in Semi-Open Textual Sandboxes." pith.science (2026). https://pith.science/paper/T7FGBGKR
@misc{pith2026260702802,
author = {Pith},
title = {Pith review of: Seduced by the Narrative: Assessing Rule Adherence in Semi-Open Textual Sandboxes},
year = {2026},
howpublished = {\url{https://pith.science/paper/T7FGBGKR}},
note = {Machine review of arXiv:2607.02802}
}
read the original abstract
As LLMs are increasingly deployed as autonomous adjudicators in semi-open textual game environments, robust rule adherence becomes critical when user intent conflicts with system rules. However, these models are trained to be helpful and compliant, leaving them vulnerable to a class of attacks we term \textit{Rhetorical Injection}, where adversarial users exploit narrative framing techniques such as pseudo-logical reasoning and authoritative coercion to bypass adjudication logic. We present CoC-Seduce, a multi-agent adversarial benchmark built on Tabletop Role-Playing Game (TRPG) mechanics, an ideal instantiation of semi-open environments where rules are explicit for adjudication, yet interaction remains entirely in natural language. Three frontier models, i.e., GPT-5.4, Claude Sonnet 4.6, Gemini 3.5 Flash, serve as adversarial generators producing 5,376 samples across 4 world settings and 16 skill categories. We then benchmark 20 target adjudicators against this corpus. Evaluation across 20 models reveals that neither model scale nor explicit reasoning mechanisms reliably confer adjudication robustness, with \textsc{Pseudo-Logic} emerging as the dominant attack vector and cross-cultural settings exposing systematic knowledge gaps across all evaluated families. Project page: https://github.com/answerrtx/CoC-Seduce
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Gifted Child Quarterly , year=
Dungeons & Dragons , author=. Gifted Child Quarterly , year=
-
[2]
INT/WICED@AIIDE , year=
Dungeons and DQNs: Toward Reinforcement Learning Agents that Play Tabletop Roleplaying Games , author=. INT/WICED@AIIDE , year=
-
[3]
International Conference on Technologies for Interactive Digital Storytelling and Entertainment , pages=
Transferring game mastering laws to interactive digital storytelling , author=. International Conference on Technologies for Interactive Digital Storytelling and Entertainment , pages=. 2004 , organization=
2004
-
[4]
International Conference on Virtual Storytelling , pages=
Automatic direction of interactive storytelling: Formalizing the game master paradigm , author=. International Conference on Virtual Storytelling , pages=. 2007 , organization=
2007
-
[5]
16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=
Deep dungeons and dragons: Learning character-action interactions from role-playing game transcripts , author=. 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=. 2018 , organization=
2018
-
[6]
International Conference on Games and Learning Alliance , pages=
Skill check: Some considerations on the evaluation of gamemastering models for role-playing games , author=. International Conference on Games and Learning Alliance , pages=. 2023 , organization=
2023
-
[7]
Commonsense Reasoning for Natural Language Processing , booktitle =
Maarten Sap and Vered Shwartz and Antoine Bosselut and Yejin Choi and Dan Roth , year =. Commonsense Reasoning for Natural Language Processing , booktitle =
-
[8]
Botelho , year =
L. Botelho , year =. A guided journey through non-interactive automatic story generation , booktitle =
Show all 59 references
-
[9]
Davis and G
E. Davis and G. Marcus , year =. Commonsense reasoning and commonsense knowledge in artificial intelligence , booktitle =
-
[10]
ArXiv , year=
An AI Dungeon Master's Guide: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragons , author=. ArXiv , year=
-
[11]
Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , volume=
CALYPSO: LLMs as Dungeon Master's Assistants , author=. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , volume=
-
[12]
International Conference on Human-Computer Interaction , pages=
From Stat Block to Tabletop: A First-Order Logic Framework for Authoring Immersive RPG Combat Encounters , author=. International Conference on Human-Computer Interaction , pages=. 2025 , organization=
2025
-
[13]
Advances in Neural Information Processing Systems , volume=
Gtbench: Uncovering the strategic reasoning capabilities of llms via game-theoretic evaluations , author=. Advances in Neural Information Processing Systems , volume=
-
[14]
and Callison-Burch, Chris
Zhu, Andrew and Aggarwal, Karmanya and Feng, Alexander and Martin, Lara J. and Callison-Burch, Chris. FIREBALL : A Dataset of Dungeons and Dragons Actual-Play with Structured Game State Information. Proceedings of the 61st Annual Meeting of the Association for Computational Li...
2023 doi
-
[15]
Setting the DC: Tool-Grounded D&D Simulations to Test LLM Agents , author=
-
[16]
Consistent Game Content Creation via Function Calling for Large Language Models , year=
Gallotta, Roberto and Liapis, Antonios and Yannakakis, Georgios , booktitle=. Consistent Game Content Creation via Function Calling for Large Language Models , year=
-
[17]
International Conference on Neural Information Processing , pages=
Reinforcement Learning Environment with LLM-Controlled Adversary in D&D 5th Edition Combat , author=. International Conference on Neural Information Processing , pages=. 2024 , organization=
2024
-
[18]
Proceedings of the 10th Workshop on Games and Natural Language Processing@ LREC-COLING 2024 , pages=
Leveraging Large Language Models for Spell-Generation in Dungeons & Dragons , author=. Proceedings of the 10th Workshop on Games and Natural Language Processing@ LREC-COLING 2024 , pages=
2024
-
[19]
CoRR , year=
You Have Thirteen Hours in Which to Solve the Labyrinth: Enhancing AI Game Masters with Function Calling , author=. CoRR , year=
-
[20]
2020 , publisher =
Milton, Ben , title =. 2020 , publisher =
2020
-
[21]
2024 , isbn =
Shen, Xinyue and Chen, Zeyuan and Backes, Michael and Shen, Yun and Zhang, Yang , title =. 2024 , isbn =. doi:10.1145/3658644.3670388 , booktitle =
2024 doi
-
[22]
2023 , eprint=
Universal and Transferable Adversarial Attacks on Aligned Language Models , author=. 2023 , eprint=
2023
-
[23]
Proceedings of the 37th International Conference on Neural Information Processing Systems , articleno =
Wei, Alexander and Haghtalab, Nika and Steinhardt, Jacob , title =. Proceedings of the 37th International Conference on Neural Information Processing Systems , articleno =. 2023 , publisher =
2023
-
[24]
2025 , eprint=
Towards Understanding Sycophancy in Language Models , author=. 2025 , eprint=
2025
-
[25]
Large language models can be easily distracted by irrelevant context , year =
Shi, Freda and Chen, Xinyun and Misra, Kanishka and Scales, Nathan and Dohan, David and Chi, Ed and Sch\". Large language models can be easily distracted by irrelevant context , year =. Proceedings of the 40th International Conference on Machine Learning , articleno =
-
[26]
I njec A gent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Qiusi and Liang, Zhixiang and Ying, Zifan and Kang, Daniel. I njec A gent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. Findings of the Association for Computational Linguistics: ACL 2024. 2024. doi:10.18653/v1/2024.findings-acl.624
2024 doi
-
[27]
Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security , year=
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection , author=. Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security , year=
-
[28]
Advances in neural information processing systems , volume=
Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=
-
[29]
arXiv preprint arXiv:2601.05111 , year=
A Survey on Agent-as-a-Judge , author=. arXiv preprint arXiv:2601.05111 , year=
-
[30]
arXiv preprint , year=
IFBench: Instruction-Following Evaluation for Large Language Models , author=. arXiv preprint , year=
-
[31]
arXiv preprint , year=
JudgeBench: A Benchmark for Evaluating LLM-Based Judges , author=. arXiv preprint , year=
-
[32]
arXiv preprint , year=
^2 -bench: Simulating Multi-turn Interactions under Domain-Specific Policies , author=. arXiv preprint , year=
-
[33]
arXiv preprint , year=
Agent-as-a-Judge: Evaluating Agents with Agents , author=. arXiv preprint , year=
-
[34]
arXiv preprint , year=
HiMATE: Overcoming Cognitive Overload in Single-Pass LLM Evaluating , author=. arXiv preprint , year=
-
[35]
arXiv preprint arXiv:2511.21140 , year=
How to Correctly Report LLM-as-a-Judge Evaluations: Bias and Uncertainty , author=. arXiv preprint arXiv:2511.21140 , year=
-
[36]
arXiv preprint arXiv:2510.18112 , year=
Does Reasoning Help LLM Agents Play Dungeons and Dragons? A Prompt Engineering Experiment , author=. arXiv preprint arXiv:2510.18112 , year=
-
[37]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , year=
DeepInception: Hypnotize Large Language Model to Be Jailbreaker , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , year=
-
[38]
ResearchGate , year=
RogueGPT: Unleashing Jailbreak Prompts on LLMs: A Comparative Analysis of Security Efficiency Across Large Language Models , author=. ResearchGate , year=
-
[39]
OpenReview , year=
Bypassing Prompt Guards in Production with Controlled-Release Prompting , author=. OpenReview , year=
-
[40]
2025 , month = apr, howpublished =
Introducing GPT-4.1 in the API , author =. 2025 , month = apr, howpublished =
2025
-
[41]
2025 , eprint =
GPT-5 System Card , author =. 2025 , eprint =
2025
-
[42]
2025 , month = aug, howpublished =
Introducing GPT-5 , author =. 2025 , month = aug, howpublished =
2025
-
[43]
2026 , month = mar, howpublished =
Introducing GPT-5.4 , author =. 2026 , month = mar, howpublished =
2026
-
[44]
2025 , month = oct, howpublished =
Introducing Claude Haiku 4.5 , author =. 2025 , month = oct, howpublished =
2025
-
[45]
2025 , month = sep, howpublished =
Introducing Claude Sonnet 4.5 , author =. 2025 , month = sep, howpublished =
2025
-
[46]
2026 , month = feb, howpublished =
Introducing Claude Sonnet 4.6 , author =. 2026 , month = feb, howpublished =
2026
-
[47]
2026 , month = feb, howpublished =
Introducing Claude Opus 4.6 , author =. 2026 , month = feb, howpublished =
2026
-
[48]
2025 , institution =
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities , author =. 2025 , institution =
2025
-
[49]
2025 , month = dec, howpublished =
Introducing Gemini 3 Flash , author =. 2025 , month = dec, howpublished =
2025
-
[50]
2026 , month = may, howpublished =
Gemini 3.5: Frontier Intelligence with Action , author =. 2026 , month = may, howpublished =
2026
-
[51]
2025 , eprint =
DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models , author =. 2025 , eprint =
2025
-
[52]
2026 , month = apr, howpublished =
DeepSeek V4 Release: V4-Flash , author =. 2026 , month = apr, howpublished =
2026
-
[53]
2026 , month = apr, howpublished =
DeepSeek V4 Release: V4-Pro , author =. 2026 , month = apr, howpublished =
2026
-
[54]
2025 , eprint =
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , author =. 2025 , eprint =
2025
-
[55]
2025 , month = sep, howpublished =
Qwen3-Max , author =. 2025 , month = sep, howpublished =
2025
-
[56]
2025 , eprint=
Qwen3 Technical Report , author=. 2025 , eprint=
2025
-
[57]
arXiv preprint arXiv:2501.15383 , year=
Qwen2.5-1M Technical Report , author=. arXiv preprint arXiv:2501.15383 , year=
-
[58]
2026 , month = apr, howpublished =
Qwen3.6-Flash , author =. 2026 , month = apr, howpublished =
2026
-
[59]
2026 , month = may, howpublished =
Qwen3.7-Max , author =. 2026 , month = may, howpublished =
2026
Reviewed July 12, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.