If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMs

Aixin Sun; Kang Liu; Peng Han; Shuo Shang; Siqi Fan; Xiusheng Huang; Xuezhi Fang; Yequan Wang; Yiqun Yao

arxiv: 2503.23514 · v2 · submitted 2025-03-30 · 💻 cs.CL · cs.AI

If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMs

Siqi Fan , Xiusheng Huang , Yiqun Yao , Xuezhi Fang , Kang Liu , Peng Han , Shuo Shang , Aixin Sun

show 1 more author

Yequan Wang

This is my paper

Pith reviewed 2026-05-22 21:52 UTC · model grok-4.3

classification 💻 cs.CL cs.AI

keywords lifelong learningcatastrophic forgettingLLM benchmarksepisodic memorycharacter consistencynonparametric methodsmulti-turn dialogue

0 comments

The pith

Nonparametric methods let LLMs track character stories and relationships better than parametric ones, but forgetting still grows with longer interactions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces LIFESTATE-BENCH to test whether LLMs can maintain consistent self-awareness, memory of past events, and relationship knowledge across multi-turn, multi-agent dialogues that mimic character stories. It compares parametric approaches, which update model weights, against nonparametric ones that rely on external memory or retrieval. Experiments with models such as Llama3.1-8B, GPT-4-turbo, and DeepSeek R1 show nonparametric methods handle stateful learning more effectively. All tested models, however, display increasing catastrophic forgetting as the number of interactions grows. The benchmark uses fact-checking questions drawn from Hamlet and synthetic scripts to probe these abilities directly.

Core claim

LIFESTATE-BENCH reveals that nonparametric methods significantly outperform parametric ones in managing stateful learning for LLMs acting as characters, yet every model still exhibits challenges with catastrophic forgetting as interactions extend.

What carries the argument

LIFESTATE-BENCH, a benchmark with two episodic datasets (Hamlet and synthetic scripts) and fact-checking evaluation that probes self-awareness, episodic memory retrieval, and relationship tracking.

If this is right

Nonparametric approaches can serve as a practical baseline for building more consistent LLM characters in extended dialogues.
Catastrophic forgetting remains a core obstacle even when nonparametric methods are used, so longer interaction sequences will require new mitigation techniques.
Benchmarks focused on narrative structure and character consistency can expose limitations that static open-ended evaluations miss.
Models like Llama3.1-8B, GPT-4-turbo, and DeepSeek R1 all show the same qualitative pattern of improving with nonparametric methods but degrading over time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Future work could test whether combining nonparametric memory with occasional parametric fine-tuning further slows forgetting.
The same benchmark design might be applied to evaluate consistency in non-narrative domains such as ongoing technical support conversations.
If nonparametric methods scale better, production systems may shift toward retrieval-augmented or memory-augmented LLM deployments for any multi-session use case.

Load-bearing premise

The fact-checking questions in LIFESTATE-BENCH measure genuine self-awareness, memory retrieval, and relationship tracking instead of surface-level pattern matching.

What would settle it

A follow-up experiment that replaces the fact-checking questions with equivalent surface-level pattern-matching tests and finds that nonparametric performance no longer exceeds parametric performance.

read the original abstract

Large language models (LLMs) can carry out human-like dialogue, but unlike humans, they are stateless due to the superposition property. However, during multi-turn, multi-agent interactions, LLMs begin to exhibit consistent, character-like behaviors, hinting at a form of emergent lifelong learning. Despite this, existing benchmarks often fail to capture these dynamics, primarily focusing on static, open-ended evaluations. To address this gap, we introduce LIFESTATE-BENCH, a benchmark designed to assess lifelong learning in LLMs. It features two episodic datasets: Hamlet and a synthetic script collection, rich in narrative structure and character interactions. Our fact checking evaluation probes models' self-awareness, episodic memory retrieval, and relationship tracking, across both parametric and non-parametric approaches. Experiments on models like Llama3.1-8B, GPT-4-turbo, and DeepSeek R1, we demonstrate that nonparametric methods significantly outperform parametric ones in managing stateful learning. However, all models exhibit challenges with catastrophic forgetting as interactions extend, highlighting the need for further advancements in lifelong learning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The benchmark idea has some appeal but the abstract supplies no metrics, methods, or results, so the claims cannot be checked.

read the letter

The main point is that this paper sketches LIFESTATE-BENCH, a narrative-based test using Hamlet and synthetic scripts to probe whether LLMs track self-awareness, episodic memory, and relationships across turns. It reports that nonparametric methods handle stateful learning better than parametric ones, yet all models show forgetting as interactions lengthen. That high-level framing identifies a real gap between static benchmarks and multi-turn consistency, and the story datasets are a plausible way to surface character-like behavior that might emerge in agents.

Referee Report

2 major / 0 minor

Summary. The manuscript claims that LLMs exhibit emergent consistent character-like behaviors in multi-turn interactions, suggesting a form of lifelong learning, but existing benchmarks do not capture these dynamics. It introduces LIFESTATE-BENCH, consisting of episodic datasets based on Hamlet and synthetic scripts, and uses fact-checking tasks to evaluate self-awareness, episodic memory retrieval, and relationship tracking. Experiments on models including Llama3.1-8B, GPT-4-turbo, and DeepSeek R1 are said to show that nonparametric methods significantly outperform parametric ones in stateful learning, while all models exhibit challenges with catastrophic forgetting as interactions extend.

Significance. If the benchmark validly isolates lifelong learning phenomena rather than surface pattern matching and the reported performance ordering is robust, the work could usefully highlight limitations of current LLMs in maintaining state across interactions and motivate further research on nonparametric approaches. The use of narrative-rich episodic datasets is a reasonable direction for probing memory and self-awareness, but the absence of any quantitative evidence prevents a determination of whether these contributions would be significant.

major comments (2)

[Abstract] Abstract: the claim that 'nonparametric methods significantly outperform parametric ones in managing stateful learning' is presented without any metrics, error bars, dataset sizes, statistical tests, or evaluation protocol, so the central empirical result cannot be assessed.
[Abstract] Abstract: the fact-checking evaluation is asserted to probe 'self-awareness, episodic memory retrieval, and relationship tracking' but supplies no description of query construction, negative-example controls, or safeguards against lexical overlap, narrative trope matching, or surface statistics in the Hamlet and synthetic-script datasets; this is load-bearing for whether the performance gap and forgetting results support the lifelong-learning interpretation.

Simulated Author's Rebuttal

2 responses · 0 unresolved

Thank you for the feedback on the abstract. We address each major comment below.

read point-by-point responses

Referee: [Abstract] Abstract: the claim that 'nonparametric methods significantly outperform parametric ones in managing stateful learning' is presented without any metrics, error bars, dataset sizes, statistical tests, or evaluation protocol, so the central empirical result cannot be assessed.

Authors: We agree the abstract presents the performance claim at a high level without quantitative details. The abstract is a concise summary; the full paper reports the metrics, error bars, dataset sizes, statistical tests, and evaluation protocol in the experiments section. We will revise the abstract to include key quantitative highlights and a brief protocol note. revision: yes
Referee: [Abstract] Abstract: the fact-checking evaluation is asserted to probe 'self-awareness, episodic memory retrieval, and relationship tracking' but supplies no description of query construction, negative-example controls, or safeguards against lexical overlap, narrative trope matching, or surface statistics in the Hamlet and synthetic-script datasets; this is load-bearing for whether the performance gap and forgetting results support the lifelong-learning interpretation.

Authors: We acknowledge that the abstract provides no description of query construction, controls, or safeguards against surface statistics. These elements are detailed in the methods of the full paper. We will revise the abstract to add a short description of the evaluation design and controls to strengthen the interpretation. revision: yes

Circularity Check

0 steps flagged

Empirical benchmark with no derivation chain or self-referential reductions

full rationale

The abstract introduces LIFESTATE-BENCH as an empirical evaluation benchmark and reports comparative results on parametric vs. nonparametric methods without equations, derivations, fitted parameters presented as predictions, or load-bearing self-citations. All claims rest on experimental outcomes from the new datasets rather than any reduction of results to inputs by construction, satisfying the criteria for a self-contained empirical study.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

Abstract-only review limits visibility; the ledger records the main domain assumption stated in the text and the newly introduced benchmark entity.

axioms (1)

domain assumption LLMs begin to exhibit consistent, character-like behaviors during multi-turn, multi-agent interactions, hinting at emergent lifelong learning
Directly stated in the abstract as the motivation for the benchmark.

invented entities (1)

LIFESTATE-BENCH no independent evidence
purpose: Benchmark to assess lifelong learning via episodic narrative datasets and fact-checking probes
Newly proposed in the paper; no independent evidence supplied in the abstract.

pith-pipeline@v0.9.0 · 5722 in / 1247 out tokens · 25062 ms · 2026-05-22T21:52:28.052392+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

A Survey of Context Engineering for Large Language Models
cs.CL 2025-07 accept novelty 4.0

The survey organizes Context Engineering into retrieval, processing, management, and integrated systems like RAG and multi-agent setups while identifying an asymmetry where LLMs handle complex inputs well but struggle...