{"id":"3e281bde-77a8-46b9-a1a0-bc8e31c04984","arxiv_id":"2608.31100","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LLM agents improve unevenly from their own experience: raw history and strategy summaries each win on different games, fine-tuning helps one task and damages another, and accurate self-scoring does not predict improvement.","lead":"This paper presents S3Gym, a benchmark of seven text-based games that tests whether AI agents improve at new, harder rounds of the same game by using their own past trials. Agents do improve unevenly, but raw play logs, written strategy summaries, and fine-tuning each help on different games, and none works reliably.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The exploration/evaluation config gap—not model capability—may drive the taxonomy: the games where summaries underperform are exactly those with rule changes, and no same-config control exists to rule this out.","rationale":"The paper's genuine strengths are the honest reporting of near-zero judgment–improvement coupling, the executable verifiers, and the appendix prompt templates that make the benchmark re-implementable. The empirical pattern across seven games is internally coherent. However, the central interpretation requires that exploration and evaluation test the same underlying task. The config differences are not uniformly 'more permissive but structurally identical': Snake and Minesweeper change failure semantics, and PvZ changes board geometry. Under such changes, an agent may fail or appear to underperform with summaries simply because the exploration experience is not applicable to the evaluation configuration, not because summary compression is the wrong mechanism. The correlation between the Table 7 taxonomy and the size of the config gap is the key vulnerability. A same-configuration control would settle the matter. The statistical weakness—three-episode evaluation checkpoints, no error bars—is real and should be fixed before quantitative comparisons are used, but random noise would not systematically produce a taxonomy aligned with rule changes; the config gap would. For these reasons the reader's CONDITIONAL verdict remains appropriate, and the additional controls should be required before the benchmark is used for comparative claims.","tokens_in":21713,"tokens_out":11575,"duration_ms":132202,"concrete_test":"Run the full S3Gym protocol for all seven games with exploration and evaluation using identical configurations (same rules, board sizes, and failure penalties; only seeds differ). If the Table 7 pattern persists—summaries winning on Trust, Tetris, and Nullify while history wins on Snake, PvZ, and Minesweeper—the taxonomy is not an artifact of the config gap. If the pattern reverses or collapses, the original results were confounded by the permissive exploration rules. A complementary check: for the rule-altering games, toggle one permissive factor at a time (Minesweeper extra lives on/off, Snake no-op versus fatal collisions, PvZ 3x7 versus 5x6 battlefield) while holding everything else fixed, and observe which factor drives the negative transfer and summary underperformance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central qualitative claims—summaries help on compressible-rule games and underperform on state-contingent games, and training shows negative transfer—depend on interpreting failures on strict held-out episodes as failures of self-improvement. But Table 2 changes more than difficulty. Snake treats collisions and invalid actions as no-ops during exploration but as terminal events in evaluation; Minesweeper grants two extra lives in exploration while the first mine hit terminates evaluation; PvZ changes the battlefield from 3x7 to 5x6. These are rule and policy changes, not just harder parameters. Under permissive rules the optimal behavior differs: in Minesweeper exploration, risky uncovering is nearly free; in PvZ, heuristics tuned to three lanes may simply not apply to five lanes. The Table 7 pattern aligns suspiciously well with this gap: summaries win on Trust, Tetris, and Nullify, which preserve rules and differ mainly in difficulty or opponent distribution, while history wins on Snake, PvZ, and Minesweeper, which have the largest rule-structure gaps. That correlation is exactly what a config-gap artifact would produce. Section 6.1 lists 'mismatch between relaxed exploration and strict evaluation' as a possible cause of PvZ negative transfer, but no control distinguishes this confound from the claimed task-structure explanation. Without a same-configuration or difficulty-matched control, the claim that the bottleneck is 'transforming feedback into executable, transferable policies' is not securely separated from the claim that the benchmark's exploration-to-evaluation gap is simply too wide. This is load-bearing because the headline taxonomy is constructed from that comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces S3Gym, a benchmark protocol for evaluating whether LLM agents can improve from their own interaction experience through three coupled capabilities—self-testing, self-judging, and self-improvement—and instantiates it in seven text-based games with executable environment verifiers. Exploration is run under relatively permissive configurations and evaluation under stricter, seed-disjoint configurations. The authors compare three experience-integration pathways—raw History ICL, score-conditioned Summary Memory, and parameter Training—across seven proprietary models. The central empirical claims are that self-improvement is neither automatic nor uniform; that summaries help when experience can be compressed into reusable rules but underperform raw history when success depends on state-contingent details; and that parameter training yields gains on some tasks but unstable or negative transfer on others. The paper also analyzes self-judging reliability and its weak coupling to subsequent improvement.","tokens_in":22026,"tokens_out":3899,"duration_ms":49821,"significance":"The benchmark is a genuinely useful contribution if its claims hold: the environment verifiers are executable, the protocol separates model-generated self-judgments from verifier-computed rewards during exploration, evaluation uses held-out seeds, and checkpoint zero provides a genuine zero-experience baseline. The comparison of three improvement pathways under one protocol is well motivated, and the negative result that accurate local self-judging does not by itself produce improvement is worth publishing. However, the central qualitative taxonomy—compressible-rule games versus state-contingent games—is currently confounded with changes in game rules between exploration and evaluation, and the quantitative support for that taxonomy rests on three-episode checkpoint means without variance or significance reporting. The benchmark infrastructure appears sound, but the empirical claims need additional controls and statistical grounding before they can be accepted as stated.","major_comments":[{"comment":"The exploration/evaluation configuration gap changes game rules, not only difficulty, and this confound aligns exactly with the paper's central taxonomy. Snake treats collisions and invalid actions as no-ops during exploration but as terminal events in evaluation; Minesweeper grants two extra lives during exploration while the first mine hit terminates evaluation episodes; PvZ changes the battlefield from 3×7 to 5×6. Table 7 shows that summaries win on Trust, Tetris, and Nullify, whose exploration configurations differ mainly in difficulty or opponent distribution, while raw history wins on Snake, PvZ, and Minesweeper, which have the largest rule-structure gaps. This pattern is also what a pure configuration-mismatch artifact would produce. Section 6.1 lists 'mismatch between relaxed exploration and strict evaluation' as a possible cause of PvZ negative transfer, but no experiment distinguishes this confound from the claimed task-structure explanation. The authors should add same-configuration controls, or at least matched-difficulty controls that vary only parameters and not collision semantics, extra lives, or lane counts, for the three games driving the history-wins result.","section":"§4.4, Table 2, Table 7"},{"comment":"Every reported checkpoint mean is computed from only three strict-mode evaluation episodes, and no variance, confidence interval, or significance test is reported anywhere. The AUC+ metric is a deterministic function of these three-episode means, so differences such as Gemini-2.5-Flash's Minesweeper AUC+ of 0.000 under ICL versus 7.794 under Summary Memory, or GPT-5.5's PvZ AUC+ of 548.499 versus 33.219, could plausibly be sampling noise. This affects Table 4's bolded maxima, Table 7's summary-versus-history win counts, and the abstract's claim that summaries 'often underperform raw history.' The paper should either increase the evaluation set size or report bootstrap confidence intervals and paired tests over evaluation seeds for the key model–game comparisons.","section":"§5.2, Table 3, Eq. (12), Eq. (20)"},{"comment":"The parameter-training result is a central pillar of the 'unstable improvement and severe negative transfer' claim, but it rests on a single model (Qwen3-8B), a single run, and no variance estimate. The PvZ negative transfer is especially load-bearing because the paper reports that every updated checkpoint drops from 23 to 6, yet the same section acknowledges 'mismatch between relaxed exploration and strict evaluation' as a possible cause without testing it. The SFT data construction in Eq. (16)–(17) is also underspecified: the paper does not state how many trajectories survived filtering, what the filtering thresholds were, whether the self-judged scores or verifier rewards were used to select positives, or what training hyperparameters were used. The training-level conclusions should be rephrased as preliminary, or the section should be expanded with these details and with same-configuration and repeated-seed experiments.","section":"§6.1, Figure 7"},{"comment":"The judgment–improvement coupling analysis is presented as a null result ('accurate local scoring does not reliably produce improvement'), but the underlying gains g_t are computed from consecutive three-episode checkpoint means, so the near-zero correlations could reflect measurement noise rather than a genuine absence of coupling. The paper reports no power analysis, no confidence intervals on the correlations, and no variance decomposition. At minimum, the authors should report bootstrap or permutation intervals for rho(A,g) and rho(-E,g), and should state whether the three-episode evaluation noise alone could produce correlations this close to zero under a simulated positive coupling.","section":"§6.2, Table 6"}],"minor_comments":[{"comment":"The benchmark name is spelled inconsistently: the abstract, title, and body use S3Gym or S\\textsuperscript{3}Gym, while the conclusion uses 'S3GYM'; unify the spelling.","section":"Conclusion, §7"},{"comment":"The subsection header contains an extra space in 'o3-mini / T etris', and several trajectory evidence entries mention rewards for actions whose connection to the quoted summary direction is not immediately clear; a one-sentence link between the cited trajectory step and the stated takeaway would improve readability.","section":"§6.3"},{"comment":"The NABA definition uses max(y0, epsilon) with epsilon described only as 'the first non-zero score scale when y0 = 0'; the paper should specify how epsilon is chosen for games with zero initial score and whether the choice affects any cross-game comparisons.","section":"§6.2, Eq. after line 18"},{"comment":"The main table omits models with unfinished results or explicitly marked concurrency failures, but the paper does not state how many checkpoints were affected per model or whether the exclusion of these checkpoints could bias the reported means and AUC+ values; a per-model appendix table of excluded checkpoints would address this.","section":"§5.2, Table 3 and §5.3"},{"comment":"The paper correctly notes that high event agreement is partly explained by the preponderance of zero-reward transitions, but it would be informative to report agreement conditional on at least one of the model score or environment reward being nonzero.","section":"§6.2, Table 5"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper lands in an interesting spot: it builds a clean protocol for measuring whether LLMs can turn their own interaction history into better behavior, separates self-testing, self-judging, and self-improvement, and provides executable verifiers in seven games. The 116k-transition judgment-reliability dataset is a genuine artifact, and the honest reporting of near-zero coupling between judgment quality and improvement is a useful negative result. These are real contributions.\n\nThe soft spots are the ones the stress-test note flags. Every checkpoint mean comes from three evaluation episodes with no variance or significance test reported anywhere. That alone would make me hesitant to trust any specific comparison between pathways or games. More load-bearing: the exploration and evaluation configurations differ not just in difficulty but in rules—Snake collisions treated as no-ops vs. terminal events, Minesweeper with extra lives vs. instant death, PvZ battlefield size change. The games where summaries underperform raw history (Snake, PvZ, Minesweeper) are exactly those with the largest rule-structure gaps. The paper itself lists this as a possible cause in Sec. 6.1 but runs no same-configuration control to separate it from the claimed task-structure explanation. Without that control, the central qualitative claim—summaries help on compressible-rule games and hurt on state-contingent games—is not securely identified.\n\nWho is this for? Researchers building or evaluating self-improving agent pipelines. The benchmark design could be useful as a diagnostic, but the current empirical claims need more rigor before they can ground design decisions. I'd send this to review if I were a desk editor—the protocol is worth engaging with—but I'd expect a thorough revision that adds error bars, a same-configuration control, and either more model coverage or a clear statement of the confound's scope.\n\nFor my own work: I'd cite the benchmark design and the judgment dataset in a related-work section, but I would not cite the empirical taxonomy without the controls being added.","headline":"A well-structured benchmark for self-improving LLMs, but the headline conclusions are undermined by unmeasured confounds in the exploration/evaluation gap.","tokens_in":22583,"tokens_out":1571,"duration_ms":17616},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"LLMs can improve from their own interaction experience, but gains are neither automatic nor uniform: the binding bottleneck is transforming feedback into executable, transferable policies, not recognizing success.","keywords":["S3Gym","LLM agents","self-improvement","self-testing","self-judging","experience memory","text-based games","benchmark"],"falsifier":"Run the protocol with exploration under the strict configurations for a subset of games, or check directly whether behaviors learned under permissive rules are invalid in strict mode: if the negative results (for example, parameter training dropping Plants-vs-Zombies from 23 to 6) vanish when exploration and evaluation configurations match, then the reported limits are artifacts of rule mismatch rather than limits of self-testing, self-judging, and self-improvement.","tokens_in":23198,"feed_emoji":"🎮","tokens_out":8067,"duration_ms":80505,"temperature":0.7,"pith_summary":"This paper argues that large language models can genuinely improve from their own interaction experience, but that the improvement is neither automatic nor uniform: whether experience helps depends on the task and on the mechanism used to absorb it. To make this measurable, the authors introduce S3Gym, a benchmark of seven text-based games that separates a permissive exploration phase from a stricter, held-out evaluation phase and compares three ways of reusing experience: raw interaction history in the context, compressed strategy summaries, and parameter training. The experiments show that summaries help when experience can be distilled into reusable strategic rules but lose to raw history when success depends on precise, state-contingent details, and that parameter training produces large gains on some tasks while suffering unstable improvement and severe negative transfer on others. The paper's central conclusion is that recognizing successful actions is not the bottleneck; the bottleneck is transforming feedback into executable and transferable policies. If correct, this redirects work on self-improving agents from better judgment calibration toward better experience transformation and memory selection.","feed_headline":"LLM self-improvement works—but only on some tasks","feed_subtitle":"A seven-game benchmark shows when experience helps an LLM refine itself—and when it backfires.","key_machinery":"The machinery is the S3Gym protocol itself: a self-improvement cycle runs the stages Explore, Judge, Consolidate, Update, and Evaluate, in which the agent tests strategies under deliberately permissive game configurations and is then measured on stricter, held-out configurations with disjoint random seeds. Its key separation is between agent-visible and verifier-side information: the model outputs an action together with a self-judged score $s$, while the environment verifier records the true reward $r$ benchmark-side, so judgment reliability and behavioral improvement can be measured independently. Three pathways instantiate the consolidation step: History ICL, which appends score-annotated trajectories to the context; Summary Memory, which compresses judged trajectories into strategies to retain, mistakes to avoid, and concrete next-cycle directions; and parameter Training, which converts judged trajectories into supervised fine-tuning examples, all starting from the same base model, exploration seeds, and interaction budgets. Improvement is tracked per checkpoint through average score, maximum score, and the positive area above the initial-score baseline ($AUC^+$), which separates sustained gains from isolated peaks.","core_discovery":"On its own terms, the paper establishes that self-improvement is real but conditional. At the context level, both history-conditioned prompting and score-conditioned summary memory improve several model-game pairs, yet the winning pathway flips with task structure: summaries lift performance on Nullify, Tetris, and Trust Evolution, where experience compresses into reusable rules such as planning backward to exact cancellation or adapting to an inferred opponent strategy, while raw history wins on Minesweeper, Plants-vs-Zombies, and Snake, where success tracks state-contingent board geometry and timing. At the parameter level, fine-tuning on self-generated trajectories yields a large gain on Trust Evolution, weak or intermittent gains on Chess and Snake, no measurable improvement on Minesweeper, Nullify, and Tetris, and a persistent collapse on Plants-vs-Zombies. The paper further shows that self-judged immediate scores are only partially calibrated, with near-random agreement with verifier rewards on Chess and high over-confidence on Trust, and that step-level judgment accuracy shows essentially zero correlation with subsequent improvement across blocks, runs, and tertile comparisons. The claim that follows is that accurate self-scoring is insufficient for self-improvement, because the agent must also convert judged experience into reusable state abstractions, decision rules, and exploration strategies.","pith_inferences":["A direct control experiment follows from the protocol itself: running exploration under the strict configurations would reveal how much of the measured failure is a permissive-to-strict rule-transfer gap rather than a genuine limit of self-improvement; the paper does not run this control. ","An oracle-judge ablation, in which the verifier reward is given to the agent during exploration instead of withheld, would isolate whether the bottleneck lies in judging experience or in consolidating it into executable policy. ","The rule-compressible versus state-contingent dichotomy suggests a practical design rule for agent frameworks: route experience through a compressor only when the task has stable rule-like structure, and keep raw history for reactive control tasks. ","The exploration-evaluation gap mirrors real deployments where practice conditions are easier than operating conditions, so the negative-transfer findings imply that the fidelity of the practice environment may matter more than the volume of practice. "],"forward_implications":["Future self-improving agents need memory selection and trajectory filtering, not merely more interaction data, because the quality and transformability of experience constrain gains more than interaction volume. ","Summary-based memory should be applied selectively: it helps where experience admits compact causal abstractions and hurts where success depends on reconstructing the exact current state, so agents should not treat compression as a universal upgrade over raw history. ","Parameter-level self-improvement must gate which trajectories enter training, since the current filtering of low-scoring actions fails to prevent negative transfer such as the Plants-vs-Zombies collapse from 23 to 6. ","Evaluating self-judging by step-level score agreement is insufficient, because judgment accuracy shows no reliable coupling with later improvement; the capability must be assessed through the whole loop it feeds.","The three metrics (average score, maximum score, and positive area above baseline) should be read together, since sparse-reward games can show isolated breakthroughs that neither average nor sustained-gain measures capture. "],"supporting_citations":[{"why":"supplies the game-based interactive-reasoning paradigm that S3Gym instantiates in its seven text games.","marker":"[25]"},{"why":"introduces verbal reinforcement learning, the reflection-on-attempts baseline that the context-level pathways build on.","marker":"[26]"},{"why":"introduces the evolving skill-library approach that the external-memory pathway extends.","marker":"[30]"},{"why":"provides the self-generated-reasoning bootstrapping method behind the parameter-training pathway.","marker":"[37]"},{"why":"provides reinforced self-training, a baseline for filtering and training on self-generated trajectories.","marker":"[11]"},{"why":"shows that LLM-judge scores and the policy's own confidence fail to identify causally important steps, supporting the paper's self-judging bottleneck analysis.","marker":"[38]"},{"why":"represents the fixed-policy agent benchmark paradigm that S3Gym contrasts with.","marker":"[15]"},{"why":"documents variance and task-order fragility in self-improving agents, framing the paper's instability and negative-transfer results.","marker":"[35]"},{"why":"shows how a defective skill can contaminate later behavior, providing context for the observed training-level negative transfer.","marker":"[24]"},{"why":"establishes the reliability limits of LLM-as-a-judge signals that the self-judging evaluation builds on.","marker":"[41]"}],"fun_headline_variants":["Self-improvement in LLMs is real but task-dependent","S3Gym benchmark shows LLM self-improvement is conditional","LLM self-improvement: helpful on some tasks, harmful on others","Benchmark finds self-improvement depends on task structure","Self-testing and judging don't guarantee LLM self-improvement"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The load-bearing premise is that each game's permissive exploration configuration preserves the exact task structure that the strict evaluation configuration tests, so that the measured failures diagnose self-improvement capability rather than a mismatch between rules the agent practiced under and rules it is tested under.","fun_headline_variants_meta":{"raw":{"variants":["Self-improvement in LLMs is real but task-dependent","S3Gym benchmark shows LLM self-improvement is conditional","LLM self-improvement: helpful on some tasks, harmful on others","Benchmark finds self-improvement depends on task structure","Self-testing and judging don't guarantee LLM self-improvement"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"cost":0.00025634,"is_byok":false,"cost_details":{"upstream_inference_cost":0.00025634,"upstream_inference_prompt_cost":0.00017738,"upstream_inference_completions_cost":7.896e-05},"total_tokens":1549,"prompt_tokens":1267,"completion_tokens":282,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":209}},"tokens_in":1267,"tokens_out":282,"cost_source":"openrouter_usage","duration_ms":4362,"temperature":1.0,"reasoning_tokens":209,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-09T03:08:21.347586+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-alltime-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the protocol with exploration under the strict configurations for a subset of games, or check directly whether behaviors learned under permissive rules are invalid in strict mode: if the negative results (for example, parameter training dropping Plants-vs-Zombies from 23 to 6) vanish when exploration and evaluation configurations match, then the reported limits are artifacts of rule mismatch rather than limits of self-testing, self-judging, and self-improvement.","supporting_citations":[{"cited_title":"KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation","cited_arxiv_id":"2505.14552","evidence_quote":"supplies the game-based interactive-reasoning paradigm that S3Gym instantiates in its seven text games."},{"cited_title":"Reflexion: Lan- guage agents with verbal reinforcement learning","cited_arxiv_id":null,"evidence_quote":"introduces verbal reinforcement learning, the reflection-on-attempts baseline that the context-level pathways build on."},{"cited_title":"STaR: Bootstrapping Reasoning With Reasoning","cited_arxiv_id":"2203.14465","evidence_quote":"provides the self-generated-reasoning bootstrapping method behind the parameter-training pathway."},{"cited_title":"Reinforced Self-Training (ReST) for Language Modeling","cited_arxiv_id":"2308.08998","evidence_quote":"provides reinforced self-training, a baseline for filtering and training on self-generated trajectories."},{"cited_title":"Credit Without Ground Truth: Claim-Matched Evaluation of Step-Level Credit with Finite Executed Replay","cited_arxiv_id":"2608.19760","evidence_quote":"shows that LLM-judge scores and the policy's own confidence fail to identify causally important steps, supporting the paper's self-judging bottleneck analysis."},{"cited_title":"AgentBench: Evaluating LLMs as agents","cited_arxiv_id":null,"evidence_quote":"represents the fixed-policy agent benchmark paradigm that S3Gym contrasts with."},{"cited_title":"On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification","cited_arxiv_id":"2608.18066","evidence_quote":"documents variance and task-order fragility in self-improving agents, framing the paper's instability and negative-transfer results."},{"cited_title":"When Self-Evolution Backfires: Pre-Commit Gating against Skill Contamination in LLM Agents","cited_arxiv_id":"2608.05810","evidence_quote":"shows how a defective skill can contaminate later behavior, providing context for the observed training-level negative transfer."},{"cited_title":"Xing, Hao Zhang, Joseph E","cited_arxiv_id":null,"evidence_quote":"establishes the reliability limits of LLM-as-a-judge signals that the self-judging evaluation builds on."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}