{"id":"556c1135-fe64-4da0-a5f1-b1ff18566701","arxiv_id":"2611.10501","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Current LLM agents explore stochastically, not curiously, because they lack a predictive model of the environment that improves online and drives action selection.","lead":"This position paper argues that current LLM agents do not exhibit curiosity but instead engage in stochastic sampling without updating a predictive model of the environment. It derives five architectural commitments that genuine curiosity requires and diagnoses how current systems fall short, clarifying a path for future research.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Paper's universal claim that current LLM agents do not explore curiously is contradicted by its own example of WorldLLM, which it describes as satisfying commitments C1–C4.","rationale":"The reader's weakest assumption concerned in-context learning's ability to satisfy C3 and C4, which the paper acknowledges as an open question. That assumption is not load-bearing because the paper's framework is forward-looking. Instead, the most immediate load-bearing problem is the internal inconsistency: the paper's blanket claim that current LLM agents lack curiosity is contradicted by its own example of a system that meets the task-driven criteria. This does not undermine the framework's value, but it does require the authors to qualify their central claim to avoid overstatement. The verdict remains unchanged because the primary contribution—the five-commitment framework—is unaffected by this rhetorical flaw, and the paper can be accepted with a minor textual adjustment.","tokens_in":14917,"tokens_out":3839,"duration_ms":31190,"concrete_test":"Verify from the WorldLLM paper [20] whether its action-selection policy π selects actions based on expected improvement of the predictive component (C3), not just on current prediction error. If it does, then the paper's central claim is contradicted by its own evidence and must be qualified. If it does not, then the paper's characterization of WorldLLM as satisfying C1–C4 is incorrect and the contradiction is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sentence b30 states: 'In this paper, we argue that current LLM agents do not explore curiously, they explore stochastically.' This is an unqualified claim about all current LLM agents. However, in §4, the paper presents WorldLLM [20] as 'an example of the four task-driven commitments' (C1–C4), with action selection 'driven entirely by intrinsic signals from these hypotheses' (sentence b183). The paper's own framework defines a curious agent (task-driven form) as one satisfying C1–C4 (sentence b133). Therefore, by the paper's own criteria, WorldLLM is a current curious LLM agent. The central claim is thus either false or requires an explicit qualifier (e.g., 'most current LLM agents without explicit curiosity mechanisms do not explore curiously'). The paper does not provide such a qualifier, and the contradiction weakens the empirical premise of the diagnosis.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This position paper argues that current LLM agents do not explore curiously but instead explore stochastically, because their predictive model (the LLM weights) remains fixed during inference. The authors draw on the RL-curiosity tradition to derive five commitments (C1–C5) that a system must satisfy to be called curious, and they diagnose several popular agent designs (e.g., CDE, Reflexion, Voyager, AI Scientist) as failing these commitments. They close with open problems and research directions.","tokens_in":15224,"tokens_out":1732,"duration_ms":14909,"significance":"If taken as a conceptual clarification, the paper performs a useful service: it forces the community to distinguish between stochastic variation of outputs (sampling) and genuine model-level adaptation driven by predictive improvement. The framework (C1–C5) is well-grounded in the existing curiosity literature and provides a precise vocabulary for discussing exploration in LLM agents. The diagnosis of current systems is incisive in many places. However, the paper's central universal claim is contradicted by its own positive example (WorldLLM), which weakens the rhetorical force; fixing this inconsistency would raise the significance substantially.","major_comments":[{"comment":"The paper states without qualification that 'current LLM agents do not explore curiously' (Abstract, b30, b34). Yet in §4, WorldLLM [20] is presented as 'an example of the four task-driven commitments' (sentence b183), and the framework defines systems satisfying C1–C4 as curious agents in the task-driven form (§3, diamond symbol). Therefore, by the paper's own criteria, WorldLLM is a current curious LLM agent. The universal claim is internally inconsistent. The authors must either add an explicit qualifier (e.g., 'most current LLM agents' or 'the dominant paradigm of current LLM agents') or explain why WorldLLM does not count as a current agent (e.g., because it is a research prototype rather than a deployed system, or because its curiosity is only partial). Without such a qualification, the foundational premise of the paper is self-contradictory.","section":"Abstract, §1, and §4","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The argument that RL fine-tuning does not qualify as curiosity because it operates across episodes rather than within an episode is sound, but the paper should acknowledge that streaming online RL and continual learning approaches—mentioned later—blur this line. The current text treats them as separate directions, but a brief note about the continuity would help avoid confusion.","section":"§5, 'RL-finetuning' paragraph","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The notation S = ⟨p_θ, M, P, π⟩ is clear, but the components p_θ and M are both described as predictive structures. The paper would benefit from a more explicit distinction: p_θ is the static LLM prior, while M is the environment-specific predictive component that gets updated. A sentence clarifying that p_θ is not part of M would prevent potential misinterpretation.","section":"§2.1, Eq. (1)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The critique of CDE [6] is valid but could be strengthened by noting that CDE's perplexity-based bonus, when computed over environmental observations (not model outputs), could actually satisfy C1. The current phrasing might be read as dismissing all uses of perplexity, which is not the authors' intent.","section":"§4, 'The wrong prediction target'","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The proposed behavioral tests are a nice addition, but test [C1] ('Before acting, the agent predicts the outcome') may be too strong for systems that predict latent states rather than explicit outcomes. Consider adding a note that the prediction can be over any representation of environment dynamics.","section":"§6.1, Behavioral tests","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Reference [38] is a URL without a year in the text; it should include the access year as given. Also, some arXiv references lack the version? (e.g., [8] says 'arXiv:2606.19476'—please verify the correct identifier).","section":"References","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper does something useful: it pulls the term 'curiosity' out of the LLM agent slop and anchors it in the RL tradition. The five commitments (C1–C5) are well-motivated, the diagnosis of current work is fair (the distinction between sampling and learning is the right target), and the authors are honest about the gaps (ICL, memory, within-episode measurement). It’s a tidy conceptual contribution that could give the field a more precise vocabulary. That alone makes it worth reading.\n\nThe main soft spot is the universal claim. The paper states repeatedly that ‘current LLM agents do not explore curiously’ (sentence b30). But it then presents WorldLLM as a current system that satisfies C1–C4 by design. That is a contradiction on the paper’s own terms. The claim needs a qualifier: ‘most current LLM agents without explicit curiosity mechanisms’ or ‘typical LLM agents that only vary sampling and context.’ The authors acknowledge that WorldLLM is an exception in the final section, but the abstract and introduction do not reflect that at all. This is a straightforward revision problem, not a fatal flaw, but it undermines the opening rhetoric.\n\nThe paper also leans heavily on the assumption that in-context learning cannot satisfy C3 and C4. That is plausible but not proven, and the authors themselves leave it open (Section 5). That is a legitimate uncertainty, not a flaw—they state it. But readers who think ICL can implement the full curiosity loop may find the central diagnosis incomplete.\n\nThe rest of the framework is solid, the literature connection is authentic, and the open problems are real. The paper knows what it is (a position paper, not an empirical result). It could benefit from a slightly less absolute framing, but the core argument holds up under qualification.\n\nFor whom? Anyone working on LLM agents, especially those claiming curiosity-driven exploration. It will annoy people who use the term loosely, and that’s good. It deserves a serious referee: send to peer review. I would cite it when writing about exploration in agents. I’d also bring it to reading group for the framework alone. The authors are serious thinkers—this is conceptually clean.\n\nFor revision: please add ‘most’ or ‘typical’ before ‘current LLM agents’ in the claim. Also, separate the description of WorldLLM more clearly from the diagnostic category of stochastic explorers. That fixes the tension without weakening the paper’s value.","headline":"A clear conceptual framework for curiosity in LLM agents, but the central universal claim is overbroad and should be qualified.","tokens_in":16347,"tokens_out":865,"duration_ms":20671},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Current LLM agents explore stochastically, not curiously, because they lack a predictive model that improves from interaction and drives action selection.","keywords":["LLM agents","curiosity","exploration","stochastic sampling","intrinsic motivation","open-endedness","predictive model","model-level adaptation"],"falsifier":"A demonstration of an LLM agent that uses only in-context learning (no weight updates) but demonstrably selects actions based on expected predictive improvement about environment responses (satisfying C3) and retains that improvement after a context reset (satisfying C4). If such a system were built, the paper's central diagnosis would require revision.","tokens_in":16453,"feed_emoji":"🔬","tokens_out":3580,"duration_ms":33006,"temperature":0.7,"pith_summary":"This position paper argues that current LLM agents do not exhibit genuine curiosity; they only explore stochastically by varying how they sample from a fixed next-token distribution. The authors derive five architectural commitments for curiosity from the reinforcement-learning tradition: a predictive component over environment responses, online improvement from current interaction, action selection driven by expected predictive improvement, persistence of improvements beyond the context window, and, in open-ended settings, self-generated prediction targets. Using these commitments, they diagnose existing methods—such as temperature sampling, tree search, retrieval-augmented generation, and memory systems—as satisfying only interaction-level exploration, not model-level adaptation. The paper reserves the term 'curiosity' for systems that meet these commitments and identifies open research problems for building such systems.","feed_headline":"Stochastic sampling ≠ curiosity for LLM agents","feed_subtitle":"A position paper argues true curiosity requires a predictive model that improves from interaction and drives action selection.","key_machinery":"The central machinery is the framework's five commitments (C1 to C5) and the accompanying system decomposition $S = \\langle p_\\theta, M, P, \\pi \\rangle$, where $p_\\theta$ is the fixed LLM, $M$ is a predictive component over environment responses, $P$ is persistent state, and $\\pi$ is the action-selection policy. The commitments require that $M$ improves from observations (C2), that $\\pi$ is shaped by expected improvement in $M$ (C3), and that improvements persist in $P$ beyond the context window (C4). C5 extends curiosity to open-ended settings by requiring the agent to generate its own prediction targets.","core_discovery":"The core discovery is a conceptual framework that precisely defines curiosity for LLM agents by adapting long-standing ideas from artificial curiosity and intrinsic motivation. The framework's five commitments distinguish stochastic exploration—where the agent varies actions and contexts but leaves the underlying predictive function unchanged—from curious exploration—where the agent maintains a model of how the environment responds to its actions, updates that model online from the current interaction, chooses actions to reduce predictive uncertainty or increase learning progress, and retains the improved model beyond the immediate context. In open-ended scenarios, the agent must additionally originate its own prediction targets without external specification. The paper shows that none of the prominent current methods fully satisfy these commitments, and that terms like 'curiosity' and 'exploration' are often used for mechanisms that only diversify sampling without model-level adaptation.","pith_inferences":["The framework implies that in-context learning alone cannot satisfy commitments C3 (act to learn) and C4 (retain improvements), because it lacks a mechanism for action selection driven by expected predictive improvement and because improvements vanish with the context window.","Weight updates during deployment, perhaps through streaming online RL or continual learning, may be necessary to fully satisfy C4, raising engineering challenges for large-scale LLMs.","The behavioral tests proposed for C1-C4 could serve as a practical evaluation suite for curious agents, complementing outcome-based benchmarks."],"forward_implications":["Existing methods that claim curiosity must be reevaluated; many amount to stochastic sampling, not model-level adaptation.","New benchmarks should measure within-episode prediction improvement and the influence of intrinsic signals on action selection, not just final outcomes.","Architecturally, curiosity requires integrating a learnable predictive component with the LLM, which may involve parametric updates, hypothesis states, or structured memory.","The framework clarifies the terminology, reserving 'curiosity' for systems that meet the commitments, enabling precise discussion of progress.","Combining intrinsic curiosity signals with extrinsic task rewards remains an open design problem, especially in open-ended action spaces."],"supporting_citations":[{"why":"Schmidhuber 2010 formal theory of creativity, fun, and intrinsic motivation, supplying the compression-progress foundation.","marker":"[35]"},{"why":"Schmidhuber 1991a prediction-error reward for model-building controllers, grounding intrinsic motivation in world model error.","marker":"[33]"},{"why":"Schmidhuber 1991b learning progress as intrinsic reward, shifting from surprise to improvement rate.","marker":"[32]"},{"why":"Levy et al. 2025 WorldLLM, example of explicit predictive component with natural-language hypotheses satisfying C1-C4.","marker":"[20]"},{"why":"Sancaktar et al. 2025 SENSEI, example with weight updates and epistemic uncertainty, demonstrating parametric persistence.","marker":"[30]"},{"why":"Agarwal et al. 2026 AutoDiscovery, example of Bayesian surprise driving tree search over hypotheses.","marker":"[1]"},{"why":"Engländer et al. 2026 showing LLM agents explore but ignore, empirically supporting the diagnosis of absent model-level adaptation.","marker":"[9]"},{"why":"Dai et al. 2025 CDE, counterexample where curiosity bonus is computed over model's own outputs instead of environment responses.","marker":"[6]"}],"fun_headline_variants":["LLM agents sample; they don't explore curiously","True curiosity needs online model learning, not sampling","Why your LLM agent isn't really curious","Stochastic shuffle ≠ curious investigation for LLMs","Five commitments define genuine LLM curiosity"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper assumes that in-context learning cannot implement the full curiosity loop (specifically commitments C3 and C4) because the model weights remain fixed and context is bounded, which precludes action selection driven by expected predictive improvement and persistence beyond the context window.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents sample; they don't explore curiously","True curiosity needs online model learning, not sampling","Why your LLM agent isn't really curious","Stochastic shuffle ≠ curious investigation for LLMs","Five commitments define genuine LLM curiosity"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":5e-05,"raw_usage":{"cost":4.9733376e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":4.9733376e-05,"upstream_inference_prompt_cost":2.2320256e-05,"upstream_inference_completions_cost":2.741312e-05},"total_tokens":1314,"prompt_tokens":892,"completion_tokens":422,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":351}},"tokens_in":892,"tokens_out":422,"cost_source":"openrouter_usage","duration_ms":4048,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":351,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-10T22:38:45.586589+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"A demonstration of an LLM agent that uses only in-context learning (no weight updates) but demonstrably selects actions based on expected predictive improvement about environment responses (satisfying C3) and retains that improvement after a context reset (satisfying C4). If such a system were built, the paper's central diagnosis would require revision.","supporting_citations":[{"cited_title":"Formal theory of creativity, fun, and intrinsic motivation (1990–2010)","cited_arxiv_id":null,"evidence_quote":"Schmidhuber 2010 formal theory of creativity, fun, and intrinsic motivation, supplying the compression-progress foundation."},{"cited_title":"A possibility for implementing curiosity and boredom in model-building neural controllers","cited_arxiv_id":null,"evidence_quote":"Schmidhuber 1991a prediction-error reward for model-building controllers, grounding intrinsic motivation in world model error."},{"cited_title":"Curious model-building control systems","cited_arxiv_id":null,"evidence_quote":"Schmidhuber 1991b learning progress as intrinsic reward, shifting from surprise to improvement rate."},{"cited_title":"Worldllm: Improving llms’ world modeling using curiosity-driven theory-making.arXiv preprint arXiv:2506.06725, 2025","cited_arxiv_id":null,"evidence_quote":"Levy et al. 2025 WorldLLM, example of explicit predictive component with natural-language hypotheses satisfying C1-C4."},{"cited_title":"SENSEI: Semantic exploration guided by foundation models to learn versatile world models","cited_arxiv_id":null,"evidence_quote":"Sancaktar et al. 2025 SENSEI, example with weight updates and epistemic uncertainty, demonstrating parametric persistence."},{"cited_title":"Autodiscovery: Open-ended scientific discovery via bayesian surprise.Advances in Neural Information Processing Systems, 38:25181–25219, 2026","cited_arxiv_id":null,"evidence_quote":"Agarwal et al. 2026 AutoDiscovery, example of Bayesian surprise driving tree search over hypotheses."},{"cited_title":"Agents Explore but Agents Ignore: LLMs Lack Environmental Curiosity","cited_arxiv_id":"2604.17609","evidence_quote":"Engländer et al. 2026 showing LLM agents explore but ignore, empirically supporting the diagnosis of absent model-level adaptation."},{"cited_title":"CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models","cited_arxiv_id":"2509.09675","evidence_quote":"Dai et al. 2025 CDE, counterexample where curiosity bonus is computed over model's own outputs instead of environment responses."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}