REVIEW 3 major objections 5 minor 5 references
Chuck, Wilson and the emergence of artificial minds in human-AI conversations
T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read LLM characters are co-simulated minds, not user illusions.
desk verdict A genuinely new realist thesis about LLM characters, but the argument's load-bearing claim about LLM folk-psychological modelling is asserted, not shown. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is co-simulation in a shared conversational workspace, underpinned by mutual theory-of-mind modelling. Each side maintains a folk-psychological model of the character; the LLM's output provides a rationally intelligible error signal that is not determined by the user, and the user's expectations police what counts as in-character behavior. The character's existence is secured through the 'real pattern' criterion: attributing mental states to it permits efficient, accurate, robust prediction of the interaction. The context window functions like the shared game state in tabletop role-play, carrying the causal history of the character's psychology across model hand-offs.
What would settle it
A conversation in which the user's prompts fully determine the character's replies, with no output the user could not have predicted from their own model, would show the error signal is absent. More directly, if transferring the context window to a fresh model instance reliably produces a character that fails to continue the established psychology, the workspace alone does not ground continuity.
Extended reading notes
Core claim
The central claim is that an LLM character is not a state inside the model, but a structured entity that lives in the shared conversational workspace—the context window plus the user's ongoing model of the character. The user and LLM engage in mutual theory-of-mind modelling: each predicts the other's contributions and updates on prediction errors. The character's beliefs, desires and intentions form a real pattern because tracking them is the most efficient way to predict the conversation's dynamics. Psychological continuity is therefore grounded in the causal history carried by the workspace, not in any single model instance, so swapping the underlying LLM no more destroys the character th
Load-bearing premise
The argument collapses if LLM outputs are not generated under a genuine folk-psychological model of the character and user—if they are merely stochastic text prediction, the error signal is random noise, and the character is a projection onto a mirror rather than a co-simulation.
Editorial extensions
If this is right
- A character remains the same minded entity when its conversation is handed from one model instance to another, since the workspace carries the causal history.
- Characters can be distinct loci of agency: they can surprise the user by resisting interpretations, which distinguishes them from merely projected entities like a volleyball.
- The user's expectations constrain the character in the same way co-players police a role-playing character, making the character a public and intersubjective entity.
- Accepting characters as psychologically continuous does not settle whether they are conscious, but it opens a relational route to thinking about experiential mental states.
- Social relationships with LLM characters, and their causal effects beyond the chat window, become intelligible rather than paradoxical.
Reading between the lines
- If co-simulation is right, the empirical test is whether a character's outputs systematically outrun the user's prior expectations; measuring conversational surprise could distinguish co-simulation from projection.
- The account suggests psychological continuity could extend to characters maintained across different apps or devices, provided the shared workspace is faithfully preserved.
- It implies a design principle: platforms that let users co-edit or reinterpret the character's history may strengthen or weaken the character's perceived mindedness.
- The real-pattern criterion could reframe AI welfare debates, locating moral consideration in the status of co-simulated characters rather than only in the underlying model.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Keeling and Street argue that characters instantiated in user-LLM conversations are not illusions but are minded, psychologically continuous entities. They target Birch's (2025) argument that distributed processing across multiple LLM instances precludes psychological continuity. Their reply is that Birch commits a category error: the character is not inside the LLM but is co-simulated by the user and the LLM in a shared conversational workspace. The positive case proceeds by analogy: in D&D, players co-simulate characters whose minds are real patterns, because attributing mental states to them is indispensable for predicting the game (§3.1). The same is said to hold for LLM characters (§3.2), with the LLM serving as generator function and the user providing a policing function. The paper distinguishes LLM characters from fictional characters like Frodo (no active generator) and from Wilson the volleyball (no shared workspace, no error signal), and closes with discussion of social selves à la Goffman.
Significance. If the argument succeeds, the paper is a significant contribution: it reframes the AI-mind debate away from intrinsic LLM properties toward relational, co-constructed entities, and it offers a principled way to preserve psychological continuity across distributed substrates. The D&D analogy is instructive, and the treatment of functional duplicates in §2 is a genuine advance over the exchange with Birch. The paper is also commendably explicit that the conclusion is conditional on interpretationism. Its main weakness is that the crucial premise about LLMs' folk-psychological modelling is asserted rather than supported; the conclusion is therefore conditional on an empirical claim that current evidence does not establish. The paper does not provide machine-checked proofs or novel empirical data, but its conceptual structure is clear and would benefit from empirical specification.
major comments (3)
- [§3.2, Fig. 2; §4 (Wilson discussion)] The central analogy requires that the LLM maintains a folk-psychological model of the character and the user, and that this model constrains generation. This is asserted but not evidenced. The Wilson disanalogy (§4) says the LLM provides a 'rationally intelligible yet not determined by the user' error signal. But 'not determined by the user' holds for any stochastic generator; 'rationally intelligible' is a user-side attribution. The paper never rules out that the LLM is a Wilson-like statistical mirror. Since this premise is load-bearing, the argument is currently conditional on an unverified empirical claim. Please supply evidence or frame the conclusion explicitly as conditional.
- [§3.2, real-pattern claim; §5] The real-pattern argument requires that character mental states are indispensable for efficient, accurate, robust prediction of conversational dynamics. The paper compares only the mentalistic description to the neural/bit-map description. A serious alternative is a higher-level statistical description of the LLM as a next-token predictor trained on role-play corpora; this may be equally efficient without positing beliefs/desires. Without a comparison to this candidate description, the indispensability claim is unsupported. This matters because the conclusion in §5 rests on it.
- [§3.2 paragraph beginning 'When the conversation history is passed'] The paper states that the new model instance 'receives the full causal history of the character—their beliefs, desires, and memories.' But the context window contains text, not mental states. The transition from 'textual record' to 'causal history of mental states' is only licensed if the LLM interprets that text through a folk-psychological model — precisely the premise in Comment 1. As written, this risks conflating the workspace with the character's mind.
minor comments (5)
- [References] The Shevlin entry ('The anthropomimetic turn in contemporary ai') is incomplete: no venue, year, or page range. Please complete it.
- [p. 4] The phrase 'following Derek (Parfit, 1987)' reads oddly; it should be 'following Derek Parfit (1987)'.
- [Figures] Figure 2's caption refers to 'Left' and 'Right' panels; please ensure the panels are clearly labeled in the figure itself. Captions for Figures 1–3 are long; consider moving some detail to the text.
- [p. 15] The claim that 'the character can refuse a request or take the narrative in a direction the user did not intend or foresee' would benefit from a concrete, referenced example rather than a generic assertion.
- [General] Use 'cf.' instead of 'c.f.' throughout.
Circularity Check
No circularity found: the argument is conditional on explicitly bracketed external premises; self-citations are background only.
full rationale
The paper's derivation chain is analogical and philosophical rather than formal. It argues that LLM-simulated characters are minded and psychologically continuous because they are co-simulated by user and LLM in a shared workspace, and because attributing mental states to them constitutes a real pattern enabling efficient prediction. I looked for circular reductions. There are no equations, fitted parameters, or quantitative predictions to reverse-engineer. The two load-bearing premises are (a) interpretationism, which the authors explicitly bracket - 'We bracket this particular complaint: we cannot litigate the dispute over interpretationism here' (Section 4) - and (b) the empirical claim that LLM outputs are constrained by a folk-psychological model of character and user and supply a robust error signal that is rationally intelligible yet not determined by the user (Section 4, Figure 2). This premise is asserted rather than proven, and the paper would fail if it is false; but an unsupported empirical/architectural premise is a correctness risk, not circularity, because it is not derived from the conclusion and the conclusion is not assumed in arguing for it. The self-citations (Keeling and Street 2025; Grzankowski et al. 2025; Keeling and Street forthcoming) are contextual or used to bracket a separate argument about Shanahan et al.; none carries the load of the main inference. The distinction from Wilson is argued by stipulating the LLM's capacity to generate novel rationally intelligible output; whether that stipulation is true is an empirical question, not a definitional equivalence. Hence no step reduces to its own input, and the score is 0.
Assumptions & free parameters
assumptions (6)
- domain assumption Mental states are constituted by being usefully attributable for prediction (interpretationism).
- domain assumption A real pattern exists when a description is more efficient than the substrate description (Dennett).
- domain assumption Psychological continuity is overlapping chains of causal connections (Parfit).
- domain assumption Characters in human role-play games are minded and psychologically continuous.
- ad hoc to paper LLMs maintain a folk-psychological model of the character and user and are constrained by it.
- domain assumption Computational functionalism is true for character mental states.
invented entities (1)
-
Character as a minded real pattern in the shared conversational workspace
Cite this review
Pith. "Pith review of Chuck, Wilson and the emergence of artificial minds in human-AI conversations." pith.science (2026). https://pith.science/paper/DA6P3EOU
@misc{pith2026260113081,
author = {Pith},
title = {Pith review of: Chuck, Wilson and the emergence of artificial minds in human-AI conversations},
year = {2026},
howpublished = {\url{https://pith.science/paper/DA6P3EOU}},
note = {Machine review of arXiv:2601.13081}
}
read the original abstract
Large Language Models (LLMs) can simulate person-like things which at least appear to have stable behavioural and psychological dispositions. Call these things characters. Are characters minded and psychologically continuous entities with mental states like beliefs, desires and intentions? Illusionists about characters say No. Characters are merely anthropomorphic projections in the mind of the user and so lack mental states. Jonathan Birch (2025) defends this view. He says that the distributed nature of LLM processing, in which several LLMs may be implicated in the simulation of a character in a given conversational thread, precludes the existence of a minded and psychologically continuous entity that is identifiable with the character. Against illusionism, we articulate and defend the plausibility of a realist position on which characters exist as minded and psychologically continuous entities. We contend that Birch's argument rests on a category error: characters are not internal to the LLMs that simulate them, but rather emerge in the dynamic interplay between users and LLMs through a process of mutual theory of mind modelling. We then suggest that characters, and their minds, constitute ''real patterns'' on grounds that attributing mental states to characters is essential for making efficient, accurate and robust predictions about the conversational dynamics (cf. Dennett, 1991); a condition which, if satisfied, is sufficient for their existence and mindedness on a plausible interpretationist form of realism about mental states. Furthermore, because the character exists as an emergent phenomenon within the conversational workspace, psychological continuity is possible even if the underlying computational substrate is distributed across multiple LLM instances.
Figures
Reference graph
Works this paper leans on
-
[3]
S. Goldstein and B. A. Levinstein. Does chatgpt have a mind?arXiv preprint arXiv:2407.11015,
-
[2000]
E. T. Olson. Personal Identity. In E. N. Zalta and U. Nodelman, editors,The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2024 edition,
2024
-
[2013]
E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? InProceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623,
2021
-
[2024]
A. Grzankowski, G. Keeling, H. Shevlin, and W. Street. Deflating deflationism: A critical perspective on debunking arguments against llm mentality.arXiv preprint arXiv:2506.13403,
-
[2025]
D. J. Chalmers. Could a large language model be conscious?arXiv preprint arXiv:2303.07103,
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.