{"id":"caa1a89b-1211-4b58-a1e4-bae7f3cc1c49","arxiv_id":"2505.01073","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"RAL lets an LLM agent learn from its own play by proposing strategies, validating them in one-step state transitions, and retrieving condensed experiences, improving StarCraft II decision-making without any model training.","lead":"This paper presents RAL, a training-free method in which a large language model builds a reusable strategy database by proposing tactics, testing them in StarCraft II, and summarizing validated lessons. The method raises win rates on several micro-battle tasks at a fraction of the cost of model training, which matters for deploying LLM agents in domains where labeled data and compute are scarce.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing step is the claim that one-step LLM validation labels track episode-level policy quality; Section 5.1 concedes this link is not established, and no objective check anchors validation to actual outcomes.","rationale":"Good-faith reading: the paper contributes a clear, reproducible three-stage pipeline with full prompts and pseudo-code, evaluates on an external win-rate metric, and honestly reports negative transfer with DeepSeek models; these are real strengths. My concern is not that the pipeline is incoherent, but that its central epistemic claim—'validated knowledge'—rests on a single unverified correlation: one-step LLM judgments about state transitions must be a reliable proxy for episode-level policy quality. The paper itself flags this in Section 5.1, and the appendices show the validation stage has no objective scoring anchor. This makes the concern load-bearing rather than stylistic: if the correlation fails, the improvement in Table 1 could be caused by the retrieved text's generic tactical content (focus fire, reposition) rather than by validated experience. The other weaknesses noted by the Reader (no error bars, RAL-best vs RAL-25, no direct hallucination metric) are important but secondary; they affect the strength of the evidence for the claim, not the identity of the load-bearing mechanism. Therefore I agree with the Reader's CONDITIONAL verdict: the method is worth building on, but the validation-to-outcome link needs to be demonstrated before 'validated knowledge' and 'reduces hallucination' are accepted as established.","tokens_in":31001,"tokens_out":7501,"duration_ms":79152,"concrete_test":"Audit validation calibration on the logged learning runs. For a sample of hypotheses in each task, constrain the action module to follow only that hypothesis to episode end in 30 fresh evaluation episodes, record the win rate, then compare this outcome against the LLM validation label that was stored for that hypothesis from one-step transitions. Compute precision of a 'good' label (P(win | 'good')) and recall (P('good' | win)) per task and model. If precision is not significantly above the task baseline win rate, or if 'good' labels are no more predictive of wins than a simple rule based on immediate HP/kill delta, the validation stage is not grounding episode-level knowledge; the central claim should be downgraded to 'RAG with generated tactical text improves prompting,' not 'validated self-supervised knowledge generation.' This test would settle the concern directly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RAL's central claim is that it 'reduces hallucination by generating and utilizing validated knowledge.' The only step that converts raw transitions into 'validated' knowledge is Section 3.2's validation: an LLM, prompted with (o_t, a_t, o_{t+1}) and the hypothesis, writes a good/bad judgment that is later summarized into experience (Section 3.3). Nothing in the pipeline compares that judgment to an objective outcome: no win/loss, no episode return, no unit-value ledger across the episode. The validation prompt (Appendix B.3.2) even instructs the LLM to 'conduct rigorous numerical analysis,' but the label is a free-text claim with no threshold or scoring rule. Section 5.1 explicitly states that 'one-step state transition is not sufficient for generating optimal strategy that needs to be validated on long-horizon data.' Because win rate is an episode-level metric, RAL's reported 35% to 95% gain can only be attributed to validated knowledge if one-step LLM labels are a reliable proxy for episode outcomes. If they are not—if the validator is optimistic, myopic, or simply repeats pretraining priors—then the H/V/E databases codify plausible-sounding heuristics rather than validated knowledge, and the improvement may come from in-context tactical text rather than from self-supervised learning. The DeepSeek transfer failures and hand-tuned thresholds show the system is sensitive to the model and parameters, but they do not establish that validation is measuring the right quantity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Retrieval Augmented Learning (RAL), a train-free and reward-free framework that uses RAG databases to organize hypotheses, validations, and experiences generated by LLMs interacting with a StarCraft II environment (LLM-PySC2). The pipeline has three stages: hypothesis proposal from one-step transitions, validation of hypotheses by an LLM judging the same transition, and experience generation from collected validations; at deployment the agent retrieves experiences or hypotheses. Experiments compare RAL with direct LLM decision-making and LLM reflection on five LLM-PySC2 tasks, plus OOD and transferability tests, and report win rates, killed-unit value, and token/waiting-time costs. The headline result is a rise from 35% to 95% win rate on 3s_vs_3z after 25 episodes, with smaller or negligible gains on other tasks.","tokens_in":31171,"tokens_out":7629,"duration_ms":69882,"significance":"If the central claim holds, the framework offers a low-cost, training-free alternative for adapting LLMs to domain-specific decision-making, and the three-stage RAG-based knowledge generation is a novel organizing idea. The paper is transparent in several respects: the full prompts are given in Appendix B, the pseudo-code in Appendix A, the database thresholds and top-k values in Table C7, and the negative transfer results for DeepSeek models are reported honestly. The main risk is that the 'validated knowledge' label depends entirely on an LLM's per-step judgment, which the paper itself concedes (Section 5.1) is not sufficient for long-horizon policies; without an objective anchor, the improvement in win rate cannot be attributed to the validation mechanism. The empirical support is also uneven: the strongest gains appear on one task, and the 20-game evaluations lack error bars.","major_comments":[{"comment":"The validation step uses a single state transition (o_t, a_t, o_{t+1}) plus event text, and the LLM's free-text judgment is the only gate for promoting a hypothesis into experience; Section 5.1 states that one-step state transitions are not sufficient for validating long-horizon strategy. Because win rate is an episode-level measure, the reported 35%-to-95% gain can only be credited to 'validated knowledge' if per-step LLM labels track episode outcomes. The paper never compares validation labels with the actual win/loss or episode return, and the validation prompt (Appendix B.3.2) asks the model to 'conduct rigorous numerical analysis' but defines no scoring rule. I recommend adding an experiment that correlates validation labels with episode outcomes, or an ablation in which the validation labels are replaced by random/no validation, to isolate the contribution of the validation stage.","section":"Section 3.2 and Section 5.1"},{"comment":"The evaluation uses 20 games per checkpoint with no error bars or significance tests, and the 'RAL-best' column reports the best checkpoint over training, which selects for favorable noise. Under a binomial model with 20 games, the standard error is roughly 7-10 percentage points, so the 5-10 point improvements on 4s_vs_1R4r and 4s_vs_5r are within sampling noise, and RAL-best is actually 5 points below Reflection on 4s_vs_5r. The strongest, most consistent gains are on 3s_vs_3z and 2a_harass. The authors should report the final checkpoint or the full learning curve with confidence intervals (e.g., Wilson intervals) and a significance test (e.g., Fisher's exact test) for the headline comparison.","section":"Section 4.1, Table 1"},{"comment":"The claim of 'at least 100x fewer interaction steps' compares 25 RAL episodes against 10^5-10^7 steps of QMIX and other RL algorithms in a different environment (SMAC/StarCraft II). The step counts are not comparable because the observation and action interfaces differ, and the RL results are taken from prior publications rather than measured in the same modified LLM-PySC2 environment. This weakens the 'extremely low cost' framing. Please report the number of environment steps used by RAL and by a comparable RL baseline in the same environment, or restrict the claim to same-environment comparisons.","section":"Section 4.2"},{"comment":"The transferability results show that DeepSeek models generally fail to benefit from RAL-generated experience, and in several cases perform worse with it than without (e.g., Table 3: DeepSeek-R1 with GPT-4o-Mini experience drops from 20% to 0% win rate). The paper attributes this to 'the quality of the data and the model that uses the generated data,' but no analysis of data quality (e.g., human evaluation of the experience text) or of model-specific behavior is provided. At minimum, the abstract's claim that the approach 'exhibits potential in ... transferability' should be qualified, and this negative result should be treated as a limitation requiring further study.","section":"Tables 2 and 3"},{"comment":"The evaluation environment is described as 'a developed version based on LLM-PySC2 v0.1, with some changes in actions and observations,' and the code/environment is not released. Because the entire empirical evaluation depends on this modified environment, the changes need to be specified and the environment (or a minimal reproducible version) should be made available; otherwise the quantitative results cannot be independently verified.","section":"Appendix C, Table C1"}],"minor_comments":[{"comment":"The title uses 'Retrieval Augmented Learning' while the abstract says 'Retrial-Augmented Learning'; the term 'Retrial' is nonstandard and the intended meaning is likely 'Retrieval.' Please unify the terminology throughout.","section":"Title and Abstract"},{"comment":"The transition equation 'pst+1 = ρ(st, at|st+1)' appears to have the conditional reversed; it should state the distribution of s_{t+1} given (s_t, a_t).","section":"Section 2.1"},{"comment":"There are several typos: Figure 6 caption 'Improvment' should be 'Improvement'; Section 4.4 'speeking' should be 'speaking'; Section 4.4 '2s_harass' should be '2a_harass'; Section 5.2 'LLM-PySC' should be 'LLM-PySC2'.","section":"Figure 6 and Section 4.4"},{"comment":"In the 2a_harass row, the controlled units are listed as 'Team Adept-1: 3 Stalkers; Team AdeptPhase-1: 2 AdeptShadows,' which conflicts with Table C2's '2 Adepts' and the environment's Adept unit entries; this is likely a copy-paste error that should be corrected.","section":"Table C4"},{"comment":"The numeric labels for win rates in the subplots are difficult to read and not described in the caption; consider using a legend with text labels or a separate table for the WR values.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is an arXiv preprint (v1, May 2025); there is no indication of dual submission. The main concern for the editor: the empirical case rests on an unreleased modified environment, and the headline improvement is concentrated on a single scenario. The authors' honest reporting of the DeepSeek transfer failures is commendable, but the abstract and introduction overstate the generality. I would encourage the editor to require the release of the environment/code and the additional validation-anchoring experiment before acceptance. This is a borderline case between major revision and reject; I chose major revision because the framework is clearly described and the two strong task successes suggest the method can work, but the current evidence does not support the general claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. RAL is a concrete, train-free three-stage loop -- hypothesize, validate, consolidate -- built on RAG as a living memory, with full prompts and pseudo-code in the appendices. The validation step is the load-bearing but weakest link: it relies on an LLM judging a single state transition, and the paper itself concedes that one-step transitions are insufficient for long-horizon policy assessment.\n\nWhat is genuinely new is the validation stage as an explicit filter between hypothesis and experience, plus the empirical study on StarCraft II micro-battles, including out-of-distribution and cross-model transfer. The appendices are unusually complete; a competent reader could reimplement the system. The headline result, 3s_vs_3z moving from 35% to 95% win rate, is striking, but several other tasks show modest or negligible improvements.\n\nWhere it gets soft. There are no error bars or significance tests on the 20-game evaluations, and Table 1's \"RAL-best\" is the best checkpoint during learning, not the final policy. The claim that RAL \"reduces hallucination\" is asserted, not measured: no hallucination metric, no direct comparison of validated versus unvalidated knowledge. The RAG thresholds and top-k values are hand-tuned per task group, which raises questions about sensitivity. The bigger issue is substantive: the validation prompt asks the LLM to judge costs and benefits from one transition, but nothing in the pipeline compares that judgment to an actual outcome like win/loss or episode return. As the stress-test note says, the knowledge base may be codifying plausible-sounding heuristics rather than validated knowledge. The DeepSeek transfer failures and hand-tuned thresholds show the system is sensitive to model and parameters, but they do not establish that validation is measuring the right quantity. The \"at least 100x fewer interaction steps\" claim also rests on a loose comparison to RL step counts, not a controlled measurement.\n\nThe paper is honest about its main weakness in Section 5.1, which is a point in its favor. But the central claim is stronger than the evidence.\n\nFor whom: anyone working on train-free LLM agents, RAG-based memory, or self-improving decision-making systems. It deserves serious peer review, but the authors should be pushed to add error bars, report final-checkpoint results, measure hallucination directly, and include an ablation that ties validation labels to episode outcomes. I would not cite it as a proven method until the validation step is anchored to something objective.","headline":"RAL is a clearly described, low-cost three-stage knowledge-generation loop for LLM agents whose validation stage is the load-bearing but unanchored part: one-step LLM judgments stand in for episode-level policy quality, and the paper's own limitations section admits the gap.","tokens_in":31894,"tokens_out":1559,"would_cite":false,"duration_ms":17555,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A train-free, reward-free loop lifts a StarCraft LLM agent from 35% to 95% win rate.","keywords":["retrieval augmented generation","self-supervised learning","LLM decision-making","knowledge generation","hallucination reduction","StarCraft II","LLM-PySC2","reward-free learning"],"falsifier":"Run RAL for 25 episodes on a task where the known winning policy requires an early sacrifice or a temporary disadvantage that pays off later, such as intentionally losing a unit to lure the enemy; if the learned experience database never contains that policy or labels it bad because single-step validation sees only the immediate loss, the method's central claim fails on long-horizon credit assignment.","tokens_in":30645,"feed_emoji":"🎮","tokens_out":7341,"duration_ms":72911,"temperature":0.7,"pith_summary":"The paper proposes Retrial-Augmented Learning (RAL), a framework in which an LLM agent improves its own decision-making in an environment without any weight updates and without an explicit reward signal. RAL turns retrieval-augmented generation into a three-stage learning loop: the agent proposes hypothetical strategies from one-step state transitions, tests them by acting under them in similar situations, and consolidates the resulting validations into compact 'experience' entries that are later retrieved to guide decisions. In the LLM-PySC2 StarCraft II environment, the 3s_vs_3z win rate rises from 35% at baseline to 95% after 25 learning episodes, with kill/death ratio rising from 0.44 to 1.17, at roughly the same token and waiting cost as direct decision-making. The paper's claim is that this reduces hallucination because no strategy is trusted until its observed effects have been validated, and it matters because domain knowledge for LLM decision systems could then be generated autonomously instead of being bought with expensive post-training.","feed_headline":"LLM agent hits 95% win rate with no training","feed_subtitle":"Self-proposed, self-validated strategies beat reflection and add little cost to plain decision-making.","key_machinery":"The load-bearing object is the three-database RAL learning loop. In it, database H stores observation-to-hypothesis pairs (proposed strategies), database V stores situation-plus-hypothesis to validation verdicts (actual costs and benefits seen in a one-step transition), and database E stores situation-to-experience pairs that condense many validations of a strategy into one summary. At every step, thresholds and top-k parameters decide whether the agent explores a hypothesis, validates the strategy it just executed, updates an existing entry with probability epsilon, or starts exploiting distilled experience; when experiences are available, retrieval injects roughly 300 extra tokens into the decision prompt instead of long histories. This makes the retrieval-augmented generation mechanism the organizer of intermediate learning data, so the knowledge base grows and is refined through interaction rather than being fixed in advance.","core_discovery":"RAL's central claim is that an LLM can bootstrap reusable domain knowledge by using its own judgments, filtered through environment feedback, instead of gradient updates or rewards. The framework keeps three retrieval databases: hypotheses, validations, and distilled experiences; at each step the acting LLM either follows a retrieved hypothesis, exploits retrieved experiences, or acts directly, while separate learning LLMs propose a new hypothesis from the latest transition, validate the previously tested hypothesis against what actually happened, and, once enough validations accumulate, compress them into one experience. In the reported experiments RAL reaches a 95% win rate in 3s_vs_3z where direct prompting scores 35% and LLM reflection scores 0%, and it improves over baseline and reflection on most other tested tasks. The paper also reports that the generated experience transfers to out-of-distribution variants of the same matchup and can be used by other LLMs, with performance depending on both the generator and the consumer of the data.","pith_inferences":["A testable extension would be to replace the LLM validator with a rule-based or random judge: if win-rate improvement persists, the gain comes from the retrieval loop rather than from LLM judgment, and if it collapses, per-step LLM validation is doing the real work.","If one-step validation is genuinely sufficient, the same three-database loop should transfer to text-based decision environments without simulators, since its only inputs are text observations, actions, and next-state descriptions; this would make the mechanism far broader than StarCraft.","The authors' own limitation note implies a stress test: on tasks where the winning policy requires accepting short-term losses for long-horizon gains, RAL's per-step validation is likely to filter out the optimal policy, and a direct experiment on such a task would map the method's boundary.","The 'at least 100x' fewer interaction steps claim compares against a specific deep-RL regime; a fair test would hold the same task, observation format, and compute budget, especially because language-driven exploration may not transfer to action spaces without textual descriptions."],"forward_implications":["After 25 episodes the agent reaches 95% win rate in 3s_vs_3z, improving on both the 35% baseline and the 0% reflection result, and learning uses at least 100x fewer interaction steps than the cited deep RL methods.","Generated experience transfers: learning in one StarCraft matchup improves performance in harder or easier variants, so the stored knowledge is not tied to a single memorized state.","RAL adds only hundreds of tokens and roughly equal wait time relative to direct decision-making, so it remains usable on edge devices with remote LLMs and no local GPU training.","The learned knowledge is portable across LLMs: other models can improve using the generated experience, though the benefit depends on the consuming model.","Reflection does not just underperform; it can actively hurt (0% on 3s_vs_3z), supporting the paper's argument that unvalidated introspection is a source of hallucination."],"supporting_citations":[{"why":"Supplies LLM-PySC2, the StarCraft II decision-making environment and task suite that all experiments run in.","marker":"[1]"},{"why":"Deep multi-agent RL reference used for the 10^5 to 10^7 interaction-step count, the baseline behind the 'at least 100x' sample-efficiency claim.","marker":"[24]"},{"why":"The reflection methods the paper names as the closest context-learning baseline and compares RAL against.","marker":"[13]-[16]"},{"why":"Cited as 'retrial-based reasoning', the idea that repeated retrieval of intermediate results can replace one-shot reasoning, which RAL extends into an environment-interaction loop.","marker":"[32]"},{"why":"Test-time scaling work cited as inspiration for condensing many collected validations into a compact experience entry at deployment time.","marker":"[29]-[31]"}],"fun_headline_variants":["Zero training, 95% wins: LLM self-learns via RAG","LLM achieves 95% win rate without any gradient updates","Self-validating LLM: 95% win rate, zero training cost","Autonomous knowledge: LLM hits 95% without finetuning","RAG-based self-learning: LLM wins 95% with no training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire learning loop depends on the LLM being able to tell, from just one step of before-and-after state, whether a strategy is helping; if that per-step judgment is optimistic or shortsighted, the stored knowledge will be wrong no matter how well retrieval and prompting work.","fun_headline_variants_meta":{"raw":{"variants":["Zero training, 95% wins: LLM self-learns via RAG","LLM achieves 95% win rate without any gradient updates","Self-validating LLM: 95% win rate, zero training cost","Autonomous knowledge: LLM hits 95% without finetuning","RAG-based self-learning: LLM wins 95% with no training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000903,"raw_usage":{"total_tokens":3877,"prompt_tokens":927,"completion_tokens":2950,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":2851}},"tokens_in":543,"tokens_out":2950,"duration_ms":21996,"temperature":1.0,"reasoning_tokens":2851,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:27:11.164488+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RAL for 25 episodes on a task where the known winning policy requires an early sacrifice or a temporary disadvantage that pays off later, such as intentionally losing a unit to lure the enemy; if the learned experience database never contains that policy or labels it bad because single-step validation sees only the immediate loss, the method's central claim fails on long-horizon credit assignment.","supporting_citations":[{"cited_title":"S., Farquhar, G., Foerster,et al.J.,Whiteson, S","cited_arxiv_id":null,"evidence_quote":"Deep multi-agent RL reference used for the 10^5 to 10^7 interaction-step count, the baseline behind the 'at least 100x' sample-efficiency claim."}],"review_version":1}