Pith. sign in

REVIEW 5 major objections 5 minor 69 references

Retrieval Augmented Learning: A Retrial-based Large Language Model Self-Supervised Learning and Autonomous Knowledge Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A train-free, reward-free loop lifts a StarCraft LLM agent from 35% to 95% win rate.

desk verdict RAL is a clearly described, low-cost three-stage knowledge-generation loop for LLM agents whose validation stage is the load-bearing but unanchored part: one-step LLM judgments stand in for episode-level policy quality, and the paper's own limitations section admits the gap. read the letter →

arxiv 2505.01073 v1 pith:WBQ6FSER submitted 2025-05-02 cs.AI

Zongyuan Li , Pengfei Li , Runnan Qi , Yanan Ni , Lumin Jiang , Hui Wu , Xuebo Zhang , Kuihua Huang
show 1 more author
Xian Guo
This is my paper · ORCID
classification cs.AI
keywords retrievalaugmentedgenerationself-supervisedlearningLLMdecision-makingknowledgehallucinationreductionStarCraftIILLM-PySC2reward-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Retrial-Augmented Learning (RAL), a framework in which an LLM agent improves its own decision-making in an environment without any weight updates and without an explicit reward signal. RAL turns retrieval-augmented generation into a three-stage learning loop: the agent proposes hypothetical strategies from one-step state transitions, tests them by acting under them in similar situations, and consolidates the resulting validations into compact 'experience' entries that are later retrieved to guide decisions. In the LLM-PySC2 StarCraft II environment, the 3s_vs_3z win rate rises from 35% at baseline to 95% after 25 learning episodes, with kill/death ratio rising from 0.44 to 1.17, at roughly the same token and waiting cost as direct decision-making. The paper's claim is that this reduces hallucination because no strategy is trusted until its observed effects have been validated, and it matters because domain knowledge for LLM decision systems could then be generated autonomously instead of being bought with expensive post-training.

What carries the argument

The load-bearing object is the three-database RAL learning loop. In it, database H stores observation-to-hypothesis pairs (proposed strategies), database V stores situation-plus-hypothesis to validation verdicts (actual costs and benefits seen in a one-step transition), and database E stores situation-to-experience pairs that condense many validations of a strategy into one summary. At every step, thresholds and top-k parameters decide whether the agent explores a hypothesis, validates the strategy it just executed, updates an existing entry with probability epsilon, or starts exploiting distilled experience; when experiences are available, retrieval injects roughly 300 extra tokens into the decision prompt instead of long histories. This makes the retrieval-augmented generation mechanism the organizer of intermediate learning data, so the knowledge base grows and is refined through interaction rather than being fixed in advance.

What would settle it

Run RAL for 25 episodes on a task where the known winning policy requires an early sacrifice or a temporary disadvantage that pays off later, such as intentionally losing a unit to lure the enemy; if the learned experience database never contains that policy or labels it bad because single-step validation sees only the immediate loss, the method's central claim fails on long-horizon credit assignment.

Watch

Extended reading notes

Core claim

RAL's central claim is that an LLM can bootstrap reusable domain knowledge by using its own judgments, filtered through environment feedback, instead of gradient updates or rewards. The framework keeps three retrieval databases: hypotheses, validations, and distilled experiences; at each step the acting LLM either follows a retrieved hypothesis, exploits retrieved experiences, or acts directly, while separate learning LLMs propose a new hypothesis from the latest transition, validate the previously tested hypothesis against what actually happened, and, once enough validations accumulate, compress them into one experience. In the reported experiments RAL reaches a 95% win rate in 3s_vs_3z where direct prompting scores 35% and LLM reflection scores 0%, and it improves over baseline and reflection on most other tested tasks. The paper also reports that the generated experience transfers to out-of-distribution variants of the same matchup and can be used by other LLMs, with performance depending on both the generator and the consumer of the data.

Load-bearing premise

The entire learning loop depends on the LLM being able to tell, from just one step of before-and-after state, whether a strategy is helping; if that per-step judgment is optimistic or shortsighted, the stored knowledge will be wrong no matter how well retrieval and prompting work.

Editorial extensions

If this is right

  • After 25 episodes the agent reaches 95% win rate in 3s_vs_3z, improving on both the 35% baseline and the 0% reflection result, and learning uses at least 100x fewer interaction steps than the cited deep RL methods.
  • Generated experience transfers: learning in one StarCraft matchup improves performance in harder or easier variants, so the stored knowledge is not tied to a single memorized state.
  • RAL adds only hundreds of tokens and roughly equal wait time relative to direct decision-making, so it remains usable on edge devices with remote LLMs and no local GPU training.
  • The learned knowledge is portable across LLMs: other models can improve using the generated experience, though the benefit depends on the consuming model.
  • Reflection does not just underperform; it can actively hurt (0% on 3s_vs_3z), supporting the paper's argument that unvalidated introspection is a source of hallucination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to replace the LLM validator with a rule-based or random judge: if win-rate improvement persists, the gain comes from the retrieval loop rather than from LLM judgment, and if it collapses, per-step LLM validation is doing the real work.
  • If one-step validation is genuinely sufficient, the same three-database loop should transfer to text-based decision environments without simulators, since its only inputs are text observations, actions, and next-state descriptions; this would make the mechanism far broader than StarCraft.
  • The authors' own limitation note implies a stress test: on tasks where the winning policy requires accepting short-term losses for long-horizon gains, RAL's per-step validation is likely to filter out the optimal policy, and a direct experiment on such a task would map the method's boundary.
  • The 'at least 100x' fewer interaction steps claim compares against a specific deep-RL regime; a fair test would hold the same task, observation format, and compute budget, especially because language-driven exploration may not transfer to action spaces without textual descriptions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Retrieval Augmented Learning (RAL), a train-free and reward-free framework that uses RAG databases to organize hypotheses, validations, and experiences generated by LLMs interacting with a StarCraft II environment (LLM-PySC2). The pipeline has three stages: hypothesis proposal from one-step transitions, validation of hypotheses by an LLM judging the same transition, and experience generation from collected validations; at deployment the agent retrieves experiences or hypotheses. Experiments compare RAL with direct LLM decision-making and LLM reflection on five LLM-PySC2 tasks, plus OOD and transferability tests, and report win rates, killed-unit value, and token/waiting-time costs. The headline result is a rise from 35% to 95% win rate on 3s_vs_3z after 25 episodes, with smaller or negligible gains on other tasks.

Significance. If the central claim holds, the framework offers a low-cost, training-free alternative for adapting LLMs to domain-specific decision-making, and the three-stage RAG-based knowledge generation is a novel organizing idea. The paper is transparent in several respects: the full prompts are given in Appendix B, the pseudo-code in Appendix A, the database thresholds and top-k values in Table C7, and the negative transfer results for DeepSeek models are reported honestly. The main risk is that the 'validated knowledge' label depends entirely on an LLM's per-step judgment, which the paper itself concedes (Section 5.1) is not sufficient for long-horizon policies; without an objective anchor, the improvement in win rate cannot be attributed to the validation mechanism. The empirical support is also uneven: the strongest gains appear on one task, and the 20-game evaluations lack error bars.

major comments (5)
  1. [Section 3.2 and Section 5.1] The validation step uses a single state transition (o_t, a_t, o_{t+1}) plus event text, and the LLM's free-text judgment is the only gate for promoting a hypothesis into experience; Section 5.1 states that one-step state transitions are not sufficient for validating long-horizon strategy. Because win rate is an episode-level measure, the reported 35%-to-95% gain can only be credited to 'validated knowledge' if per-step LLM labels track episode outcomes. The paper never compares validation labels with the actual win/loss or episode return, and the validation prompt (Appendix B.3.2) asks the model to 'conduct rigorous numerical analysis' but defines no scoring rule. I recommend adding an experiment that correlates validation labels with episode outcomes, or an ablation in which the validation labels are replaced by random/no validation, to isolate the contribution of the validation stage.
  2. [Section 4.1, Table 1] The evaluation uses 20 games per checkpoint with no error bars or significance tests, and the 'RAL-best' column reports the best checkpoint over training, which selects for favorable noise. Under a binomial model with 20 games, the standard error is roughly 7-10 percentage points, so the 5-10 point improvements on 4s_vs_1R4r and 4s_vs_5r are within sampling noise, and RAL-best is actually 5 points below Reflection on 4s_vs_5r. The strongest, most consistent gains are on 3s_vs_3z and 2a_harass. The authors should report the final checkpoint or the full learning curve with confidence intervals (e.g., Wilson intervals) and a significance test (e.g., Fisher's exact test) for the headline comparison.
  3. [Section 4.2] The claim of 'at least 100x fewer interaction steps' compares 25 RAL episodes against 10^5-10^7 steps of QMIX and other RL algorithms in a different environment (SMAC/StarCraft II). The step counts are not comparable because the observation and action interfaces differ, and the RL results are taken from prior publications rather than measured in the same modified LLM-PySC2 environment. This weakens the 'extremely low cost' framing. Please report the number of environment steps used by RAL and by a comparable RL baseline in the same environment, or restrict the claim to same-environment comparisons.
  4. [Tables 2 and 3] The transferability results show that DeepSeek models generally fail to benefit from RAL-generated experience, and in several cases perform worse with it than without (e.g., Table 3: DeepSeek-R1 with GPT-4o-Mini experience drops from 20% to 0% win rate). The paper attributes this to 'the quality of the data and the model that uses the generated data,' but no analysis of data quality (e.g., human evaluation of the experience text) or of model-specific behavior is provided. At minimum, the abstract's claim that the approach 'exhibits potential in ... transferability' should be qualified, and this negative result should be treated as a limitation requiring further study.
  5. [Appendix C, Table C1] The evaluation environment is described as 'a developed version based on LLM-PySC2 v0.1, with some changes in actions and observations,' and the code/environment is not released. Because the entire empirical evaluation depends on this modified environment, the changes need to be specified and the environment (or a minimal reproducible version) should be made available; otherwise the quantitative results cannot be independently verified.
minor comments (5)
  1. [Title and Abstract] The title uses 'Retrieval Augmented Learning' while the abstract says 'Retrial-Augmented Learning'; the term 'Retrial' is nonstandard and the intended meaning is likely 'Retrieval.' Please unify the terminology throughout.
  2. [Section 2.1] The transition equation 'pst+1 = ρ(st, at|st+1)' appears to have the conditional reversed; it should state the distribution of s_{t+1} given (s_t, a_t).
  3. [Figure 6 and Section 4.4] There are several typos: Figure 6 caption 'Improvment' should be 'Improvement'; Section 4.4 'speeking' should be 'speaking'; Section 4.4 '2s_harass' should be '2a_harass'; Section 5.2 'LLM-PySC' should be 'LLM-PySC2'.
  4. [Table C4] In the 2a_harass row, the controlled units are listed as 'Team Adept-1: 3 Stalkers; Team AdeptPhase-1: 2 AdeptShadows,' which conflicts with Table C2's '2 Adepts' and the environment's Adept unit entries; this is likely a copy-paste error that should be corrected.
  5. [Figure 6] The numeric labels for win rates in the subplots are difficult to read and not described in the caption; consider using a legend with text labels or a separate table for the WR values.

Circularity Check

0 steps flagged · score 0.0 of 10

RAL's reported gains are anchored to external PySC2 win rates, and no fitted parameter is renamed as a prediction; the self-citations are not load-bearing.

full rationale

RAL's causal chain is: the LLM proposes hypotheses, validates them from single-step transitions, summarizes validated judgments into experiences, and the actor uses retrieved experiences to select actions. The only internally generated element is the validation label, but the paper's reported quantities (WR, Vkilled, KD) are computed from PySC2 game outcomes (Section 4.1: "WR = 100% x nwin/ntotal"), which are external to the LLM's judgments. There is no equation in the paper that defines the predicted performance as a function of the validation outputs, and no fitted parameter is later 'predicted' from the same data. The RAG hyperparameters (Table C7) are fixed thresholds rather than values fitted to evaluation episodes. The DeepSeek transfer failures (Tables 2 and 3) and the out-of-distribution results (Figure 6) provide external falsifiability. Section 5.1 concedes that "one-step state transition is not sufficient for generating optimal strategy that needs to be validated on long-horizon data"; this is an acknowledged validity limitation, not a circular reduction. Self-citations to the authors' LLM-PySC2 environment and to their Reflection-of-Episodes work are contextual references and do not carry the derivation. No step in the paper reduces, by construction, to its own inputs, so the derivation is self-contained in the circularity sense.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The framework rests on the ability of LLMs to generate, validate, and summarize strategies from short text observations, plus a retrieval system that finds the right memories. The evaluation adds assumptions about the benchmark environment and the meaning of the win-rate metric. Hyper-parameters (thresholds, top-k, update rate) are hand-set per task group with no sensitivity analysis.

free parameters (6)
  • top-k retrieval counts (kh, kv, ke) = 5 for all databases
    Table C7 sets all top-k values to 5; retrieval depth is a free choice not justified by analysis or ablation.
  • hypothesis retrieval threshold lambda_h = 0.99 (LLM-PySC2), 0.995 (LLM-SMAC)
    Table C7; hand-set per task group, controls how many hypotheses are considered; no sensitivity study.
  • validation retrieval threshold lambda_v = 0.97
    Table C7; hand-set; affects which validations feed experience generation.
  • experience retrieval threshold lambda_e = 0.99 (LLM-PySC2), 0.995 (LLM-SMAC)
    Table C7; hand-set per task group; directly affects deployment-time prompts and hence the reported win rates.
  • database update rate epsilon = 0 for H, 0.1 for V and E
    Table C7; controls whether new validation/experience segments overwrite existing ones; arbitrary choice with no ablation.
  • RAG hybrid search weight = 0.5
    Appendix C.2; retrieval weight between keyword and vector search; not varied.
assumptions (6)
  • domain assumption An LLM can reliably validate a policy from a single state transition and event log.
    Invoked in Section 3.2 as the core of reward-free validation; the paper's Section 5.1 admits long-horizon validation is missing.
  • domain assumption The observation text and event logs contain enough information to infer costs and benefits of actions.
    Preliminaries Section 2.1 and prompts in Appendix B; the validation prompt asks for rigorous numerical analysis from state transitions.
  • domain assumption Pre-trained LLMs can propose diverse and useful hypothetical strategies.
    Section 3.1 relies on LLMph to generate hypotheses; the paper notes exploration ability of LLMs limits the framework in Section 5.1.
  • domain assumption Embedding-based hybrid retrieval returns relevant hypotheses and experiences for similar states.
    Section 2.2 defines retrieval; Appendix C.2 sets GLM Embedding-3 and hybrid search; no retrieval quality evaluation is reported.
  • domain assumption Win-rate in the custom LLM-PySC2 variant is a meaningful measure of decision-making ability.
    Section 4 uses WR, Vkilled, and KD as indicators; the environment is a modified version of LLM-PySC2 v0.1 (Appendix C.1), so results may not transfer to standard benchmarks.
  • standard math MDP/POMDP formalism applies to the game environment.
    Section 2.1 states the MDP setting; standard background for sequential decision-making.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Retrieval Augmented Learning: A Retrial-based Large Language Model Self-Supervised Learning and Autonomous Knowledge Generation." pith.science (2026). https://pith.science/paper/WBQ6FSER

@misc{pith2026250501073,
  author       = {Pith},
  title        = {Pith review of: Retrieval Augmented Learning: A Retrial-based Large Language Model Self-Supervised Learning and Autonomous Knowledge Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WBQ6FSER}},
  note         = {Machine review of arXiv:2505.01073}
}
read the original abstract

The lack of domain-specific data in the pre-training of Large Language Models (LLMs) severely limits LLM-based decision systems in specialized applications, while post-training a model in the scenarios requires significant computational resources. In this paper, we present Retrial-Augmented Learning (RAL), a reward-free self-supervised learning framework for LLMs that operates without model training. By developing Retrieval-Augmented Generation (RAG) into a module for organizing intermediate data, we realized a three-stage autonomous knowledge generation of proposing a hypothesis, validating the hypothesis, and generating the knowledge. The method is evaluated in the LLM-PySC2 environment, a representative decision-making platform that combines sufficient complexity with domain-specific knowledge requirements. Experiments demonstrate that the proposed method effectively reduces hallucination by generating and utilizing validated knowledge, and increases decision-making performance at an extremely low cost. Meanwhile, the approach exhibits potential in out-of-distribution(OOD) tasks, robustness, and transferability, making it a cost-friendly but effective solution for decision-making problems and autonomous knowledge generation.

Figures

Figures reproduced from arXiv: 2505.01073 by the authors.

Figure 1
Figure 1. General process of RAL. In the learning process, the agent generates hypotheses about better policies, validates the hypothesis in similar situations, and summarizes the knowledge and experience when a policy has been sufficiently validated. When the proposed policies of a situation have been thoroughly evaluated, the agent directly uses the retrieved knowledge to make better decisions. reflection is enough for know… view at source ↗
Figure 2
Figure 2. RAL framework. At each step, the agent retrieve a list of hypothetical strategies or experience from Database H(o|h) and E(o|e), test the hypothetical policy or exploit the experience to make better decisions. At the same time, the agent learn from the state transition of last step, proposing a different strategy, validate the current strategy or present fully validated hypothetical strategy into experience. 2 Preli… view at source ↗
Figure 3
Figure 3. System Prompt for generating hypothetical policies. Since it is capable for the language model to directly explore the policy space in certain directions, such noise-driven exploration mechanisms are unnecessary and also inconvenient. In our framework, hypothesis proposal enables LLMs explore the policy space. To generate observation-hypothesis pairs (ob|bh), a model analyzes the state transition (obt, bat, obt+1) a… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: System Prompt for generating validations of a proposed policy. At step time t, if there are not enough fine-prepared experiences and the agent retrieves a list of hypothetical policies [bht] from H(ob|bh) and generate actions follow one of the hypothetical policies: [b…
Figure 5
Figure 5. Figure 5: System Prompt for generating experience of a proposed policy. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Improvment of LLM’s decision ability in the RAL learning process. (a-b) Task of 2 Adepts harass Zerg base and 3 Pheonixes harass Zerg base, the goal of these harass tasks is to kill at least half of enemy workers; (c) Task of 4 Stalkers fight against 5 Zerg Roaches; (d…
Figure 7
Figure 7. Figure 7: RAL learning process for different models. (a) Four models learn in the task of 2a_harass (b) Four models learn in 3s_vs_3z. Generally speeking, it is possible for RAL to improve the strategic performance for different models, but the robustness is influenced by the mo…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 47 canonical work pages

  1. [1]

    Zhang, X

    Li, Z., Ni, Y ., Qi, R., Jiang, L., Lu, C., Xu, X.,et al. Zhang, X. LLM-PySC2: Starcraft II learning environment for Large Language Models. arXiv preprint arXiv:2411.05348. 2024

    arXiv 2024

  2. [2]

    Large Language Models Play StarCraft II: Benchmarks and a Chain of Summarization Approach

    Ma, W., Mi, Q., Zeng, Y ., Yan, X., Wu, Y ., Lin, R., et al., Wang, J. Large Language Models Play StarCraft II: Benchmarks and a Chain of Summarization Approach. In Advances in Neural Information Processing Systems, volume 37, pages 133386–133442, 2024

    work page 2024

  3. [3]

    SwarmBrain: Embodied Agent for Real-Time Strategy Game StarCraft II via Large Language Models

    Shao, X., Jiang, W., et al.,Liu, M. SwarmBrain: Embodied Agent for Real-Time Strategy Game StarCraft II via Large Language Models. In arXiv preprint arXiv:2401.17749, 2024. URL: https: //arxiv.org/abs/2401.17749

    arXiv 2024

  4. [4]

    Z. Li, C. Lu, X. Xu, R. Qi, Y . Ni, L. Jiang, et al., X. Guo. Hierarchical Expert Prompt for Large- Language-Model: An Approach Defeat Elite AI in TextStarCraft II for the First Time. arXiv preprint arXiv:2502.11122, 2025

    arXiv 2025

  5. [5]

    ChatDev: Communicative Agents for Software Development

    Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y ., Li, J., Yang, C., Chen, W., Su, Y ., Cong, X., Xu, J.,et al., Sun, M. ChatDev: Communicative Agents for Software Development. arXiv preprint arXiv:2307.07924, 2023

    arXiv 2023

  6. [6]

    F., An, B.,et al., Lu, Z

    Tan, W., Ding, Z., Zhang, W., Li, B., Zhou, B., Yue, J., Xia, H., Jiang, J., Zheng, L., Xu, X., Bi, Y ., Gu, P., Wang, X., Karlsson, B. F., An, B.,et al., Lu, Z. Towards General Computer Control: A 10 Zongyuan Li et. al. Multimodal Agent for Red Dead Redemption II as a Case Study. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, January 2024

    work page 2024

  7. [7]

    Y . Li, S. Liu, T. Zheng, M. Song. Parallelized Planning-Acting for Efficient LLM-based Multi-Agent Systems. arXiv preprint arXiv:2503.03505, 2025

    arXiv 2025

  8. [8]

    Z. Zeng, W. Watson, N. Cho, S. Rahimi, S. Reynolds, et al., M. Veloso. FlowMind: Automatic Workflow Generation with LLMs. In Proceedings of the Fourth ACM International Conference on AI in Finance, pages 73–81, November 2023

    work page 2023

Show all 69 references
  1. [9]

    J. Xu, W. Du, X. Liu, X. Li. LLM4Workflow: An LLM-Based Automated Workflow Model Generation Tool. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pages 2394–2398, October 2024

    2024

  2. [10]

    TradingAgents: Multi-Agents LLM Financial Trading Framework

    Xiao, Y ., Sun, E.,et al., Wang, W. TradingAgents: Multi-Agents LLM Financial Trading Framework. arXiv preprint arXiv:2412.20138, 2024

    2024 arXiv

  3. [11]

    Ghanem, B

    Ma, W., Fu, Y ., Zhang, Z., Li, G.,et al. Ghanem, B. VLMs Play StarCraft II: A Benchmark and Multimodal Decision Method. arXiv e-prints, arXiv:2503, 2025

    2025

  4. [12]

    , Anandkumar, A

    Fan, L., Wang, G., Jiang, Y ., Mandlekar, A., Yang, Y ., Zhu, H., Tang, A., Huang, D., et al. , Anandkumar, A. MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge. Advances in Neural Information Processing Systems, vol. 35, pages 18343–18362, 2022

    2022

  5. [13]

    Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf

    Xu, Y ., Wang, S., Li, P., Luo, F., Wang, X., Liu, W.,et al., Liu, Y . Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf. arXiv preprint arXiv:2309.04658, 2023

    2023 arXiv

  6. [14]

    Reflection of Episodes: Learning to Play Game from Expert and Self Experiences arXiv preprint arXiv:2502.13388, 2025

    Xu, X., Li, Z., Lu, C., Qi, R., Ni, Y ., Jiang, L., Liu, X., Zhang, X., Fang, Y ., Huang, et al., Li, Z. Reflection of Episodes: Learning to Play Game from Expert and Self Experiences arXiv preprint arXiv:2502.13388, 2025

    2025 arXiv

  7. [15]

    Y Deng, Y Yu, W Ma, Z Wang, W Zhu, J Zhao, Y Zhang SMAC-Hard: Enabling Mixed Opponent Strategy Script and Self-play on SMAC arXiv preprint arXiv:2412.17707, 2024

    2024 arXiv

  8. [16]

    Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory arXiv preprint arXiv:2305.17144, 2023

    Zhu, X., Chen, Y ., Tian, H., Tao, C., Su, W., Yang, C., Huang, G., Li, B., Lu, L., Wang, X., Qiao, Y .,et al., Dai, J . Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory arXiv preprint a...

    2023 arXiv

  9. [17]

    Zijlstra, M

    Meta Fundamental AI Research Diplomacy Team (FAIR)†, Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., et al. Zijlstra, M. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science, 378(6624), 1067-1074. 2022

    2022

  10. [18]

    Llama rider: Spurring large language models to explore the open world

    Feng, Y ., Wang, Y .,et al., Lu, Z. Llama rider: Spurring large language models to explore the open world. arXiv preprint arXiv:2310.08922, 2023

    2023 arXiv

  11. [19]

    Wang, S., Zhang, S., Zhang, J., Hu, R., Li, X., Zhang, T., et al. Hovy, E. Reinforcement learning enhanced llms: A survey. arXiv preprint arXiv:2412.10400. 2024

    2024 arXiv

  12. [20]

    Y Deng, W Ma, Y Fan, R Song, Y Zhang, H Zhang, J Zhao SMAC-R1: The Emergence of Intelligence in Decision-Making Tasks arXiv preprint arXiv:2410.16024, 2024

    2024 arXiv

  13. [21]

    V oyager: An open-ended embodied agent with large language models

    Wang, G., Xie, Y ., Jiang, Y ., Mandlekar, A., Xiao, C.,et al., Anandkumar, A. V oyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  14. [22]

    Riedmiller, M

    Mnih, V ., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D.,et al. Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. 2013

    2013 arXiv

  15. [23]

    Value-Decomposition Networks for Cooperative Multi-Agent Learning

    Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W., Zambaldi, V ., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, et al., Graepel, T. Value-Decomposition Networks for Cooperative Multi-Agent Learning. In arXiv preprint arXiv:1706.05296, 2017. URL: https://arxiv.org/abs/1706.05296

    2017 arXiv

  16. [24]

    S., Farquhar, G., Foerster,et al.J.,Whiteson, S

    Rashid, T., Samvelyan, M., De Witt, C. S., Farquhar, G., Foerster,et al.J.,Whiteson, S. Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. In Journal of Machine Learning Research, volume 21, number 178, pages 1–51, 2020

    2020

  17. [25]

    P., Hunt, J

    Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y .,et al., Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971. 2015

    2015 arXiv

  18. [26]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. 2017. 11 Zongyuan Li et. al

    2017 arXiv

  19. [27]

    Grandmaster Level in StarCraft II Using Multi-Agent Reinforcement Learning

    Vinyals, O., Babuschkin, I., Czarnecki, W., Mathieu, M., Dudzik, A., Chung, J., Choi, D., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang, A., Sifre, L., Cai, T., Agapiou, J., Jaderberg, M., Vezhnevets, A., Leblond, R., Pohlen,et al.,...

    2019

  20. [28]

    DI-star: An Open-source Reinforcement Learning Framework for StarCraft II

    DI-star Contributors. DI-star: An Open-source Reinforcement Learning Framework for StarCraft II. 2021

    2021

  21. [29]

    L., Fei-Fei, L., Hajishirzi, H., et al

    Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., et al. Hashimoto, T. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393. 2025

    2025 arXiv

  22. [30]

    Liu, R., Gao, J., Zhao, J., Zhang, K., Li, X., Qi, B., et al. Zhou, B. Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling. arXiv preprint arXiv:2502.06703. 2025

    2025 arXiv

  23. [31]

    Zuo, Y ., Zhang, K., Qu, S., Sheng, L., Zhu, X., Qi, B.,et al. Zhou, B. TTRL: Test-Time Reinforcement Learning. arXiv preprint arXiv:2504.16084. 2025

    2025 arXiv

  24. [32]

    role": "system

    Liu, J., Lin, J., Liu, Y . How much can rag help the reasoning of llm?. arXiv preprint arXiv:2410.02338. 2024. 12 Zongyuan Li et. al. Appendix A. Pseudo Code A.1 RAL Framework Algorithm 1 RAL: A Retrieval Augmented Self-Supervised Context Learning Require: LLM decision-making ...

    2024 arXiv

  25. [33]

    Always concentrating all teams' fire at one unit that (1)with highest DPS(most valuable) (2)most vulnerable (3)closest

    Concentrating firepower is always necessary, attack different unit at the same time will definitely reduce killing speed and leading to terrible result. Always concentrating all teams' fire at one unit that (1)with highest DPS(most valuable) (2)most vulnerable (3)closest. Here...

  26. [34]

    (Action Validity) Whether each action of a_t1 are in a legal form that listed in the 'Valid Actions' part of s_t1?

  27. [35]

    (Action Sequence) Whether each action of a_t1 is queued in correct sequence?

  28. [36]

    (Skills/Abilities) Is it currently **possible to use skills** and **should they be used**? What are the possible side effects of using these skills?

  29. [37]

    (Attack) Whether firepower concentrated on the most **vulnerable/valuable** enemy? What are the possible side effects of attacking these units?

  30. [38]

    Also, you will be provided with 'existing_hypothesis'

    (Move) Whether the position of moving is **valid**, **long enough** and **safe** for next several seconds? What are the possible side effects of moving to the position? After generating the 'Analysis', you need to generate 'Hypothesis' with universality and robustness, which m...

  31. [39]

    xxxxx Hypothesis: Hypothetical Strategy name: xxxxx (such as **Hold Position Attack**(concentrate fire on xxx first), **Hit and Run**(concentrate fire on xxx first), **Retreat**, **Wait for Healing**) Use xxxxx(actions) to xxxxx(purpose, to complete the task given in s_t1). Po...

  32. [41]

    (Action Validity) Whether the strategy is available (main actions of the strategy in the 'Valid Action' part)

  33. [42]

    (Strategy-Task Correlation) Whether the executed actions a_t1 helps to achieve the task of s_t1?

  34. [43]

    (Benefits) According to current event and state transition from s_t1 to s_t2, what is the benefits of this hypothetical strategy? (positive impact on the task? killed enemy unit? dealt damage? or any other benefits)

  35. [44]

    Please carefully and accurately analyse costs and benefits and compare whether the benefits outweigh the losses

    (Costs) According to current event and state transition from s_t1 to s_t2, what is the cost of this hypothetical strategy? (negative impact on the task? how many unit dead? how many health lost? gained debuff? or any other cost) After generating the 'Analysis', you need to gen...

  36. [45]

    This is a xxxxx (excellent/good/bad/terrible) hypothesis

    xxxxx Validation: Consider xxxxx. This is a xxxxx (excellent/good/bad/terrible) hypothesis. Output Format Figure B5: System prompt, example input prompt and example output prompt for generating validations. The model is asked to validate an existing policy, check the actual be...

  37. [46]

    Always concentrating all teams' fire at one unit that (1)with highest DPS(most valuable) (2)most vulnerable (3)closest

    Concentrating firepower is always necessary, attack different unit at the same time will definitely reduce killing speed and leading to terrible result. Always concentrating all teams' fire at one unit that (1)with highest DPS(most valuable) (2)most vulnerable (3)closest. Here...

  38. [47]

    How many times the hypothesis achieved **good** results? What can be learned from them?

  39. [48]

    How many times the hypothesis lead to **bad** results? What can be learned from them? After generating the 'Analysis', you need to generate 'Experience', clearly point out the actual benefits and costs of this strategy(No need for pointing out potential benefits or costs). Gen...

  40. [49]

    Its advantages include xxxx

    xxxxx Experience: xxxxx(content of the hypothesis strategy) is a xxxxx(hypothesis good/bad) hypothesis. Its advantages include xxxx. Its drawbacks include xxxx. Extra attention should be paid on xxxxx when implementing this strategy. Output Format Figure B6: System prompt, exa...

  41. [50]

    **Retreat the low-health Stalker** (10 health) to safety, so it doesn't get eliminated

  42. [51]

    **Focus fire on the Zealot** at [10, 14] to eliminate it quickly

  43. [52]

    hypothesis

    **Move Stalkers into a more advantageous position** to prevent being surrounded or taking additional damage. Actions: Team Stalker-1: <Select_Unit_Move_Screen(0x100040001, [3, 17])> # Retreat the low-health Stalker to a safer position. <Attack_Unit(0x100140001)> # Focus fire o...

  44. [53]

    **Focused Fire**: First, attack the Zealot with the lowest health to eliminate it quickly and reduce the enemy's threat

  45. [54]

    This provides a better angle for further attacks and minimizes the risk of being trapped

    **Reposition**: After eliminating the first Zealot, move to a safer position away from the edge of the map. This provides a better angle for further attacks and minimizes the risk of being trapped

  46. [55]

    experience-0

    **Safety**: Since the Stalkers are near the map edge, moving to a more secure and strategic position will allow them to continue engaging the remaining Zealot while minimizing risks. Actions: Team Stalker-1: <Attack_Unit(0x100140001)> # Attack the Zealot with health 111 (weake...

  47. [56]

    Focus fire on the Zealot with 50% health to eliminate it

  48. [57]

    Move the low-health Stalker to a safe position

  49. [58]

    , "a_t1":

    Ensure that the remaining Stalkers stay in a good position to continue combat. **Actions:** Team Stalker-1: <Attack_Unit(0x100140001)> # Focus on eliminating the Zealot with 50% health <Select_Unit_Move_Screen(0x100040001, [16, 13])> # Move low-health Stalker to a safer positi...

  50. [59]

    The attack on the Zealot (tag: 0x1000c0001) and the move to [16, 15] are both legal actions according to the valid action list

    **Action Validity**: All actions of a_t1 are valid. The attack on the Zealot (tag: 0x1000c0001) and the move to [16, 15] are both legal actions according to the valid action list

  51. [60]

    skills/abilities -> attack -> move

    **Action Sequence**: The sequence of actions in a_t1 is correct. The attack is performed before the move, which is in line with the recommended action order of "skills/abilities -> attack -> move."

  52. [61]

    The Stalkers are only performing attacks and moves, which is appropriate for the current situation

    **Skills/Abilities**: No skills or abilities are mentioned or available in this situation. The Stalkers are only performing attacks and moves, which is appropriate for the current situation

  53. [62]

    By targeting the weak Zealot, the Stalkers aim to eliminate the lowest-health unit quickly

    **Attack**: The attack was directed towards the Zealot at [11, 9] (health 51/150, 34% health), which is the most vulnerable enemy unit. By targeting the weak Zealot, the Stalkers aim to eliminate the lowest-health unit quickly. However, there is a risk that the remaining Zealo...

  54. [63]

    , "a_t1":

    **Move**: The move to [16, 15] is valid within the screen boundaries, but it is important to note that this move will bring the Stalkers further from the remaining Zealots. Although the move itself is safe, it does not provide the Stalkers with immediate access to re-engage th...

  55. [64]

    **Action Validity:** - The hypothesis suggests using **<Attack_Unit(tag)>** on the Zealot with the lowest health (Zealot at screen position [12, 14], health 75) followed by a **<Move_Screen([x, y])>** action. The suggested actions are valid within the context of the game state...

  56. [65]

    Kill as much as enemy units as possible and avoid losing units

    **Strategy-Task Correlation:** - The task is to "Kill as much as enemy units as possible and avoid losing units." The hypothesis is well-aligned with this task as it aims to quickly eliminate the weakest enemy Zealot and reposition the Stalkers to a safer location to avoid fut...

  57. [66]

    The **<Attack_Unit(tag)>** would likely result in the elimination of this Zealot, reducing the enemy threat significantly

    **Short-term Benefits:** - The event log indicates that the Zealot at position [12, 14] (health 75) has already been attacked and lost 36 health, bringing it down to 39 health (51% of its total health). The **<Attack_Unit(tag)>** would likely result in the elimination of this ...

  58. [67]

    hypothesis\

    **Short-term Costs:** - The **Zealot at [12, 15] (health 150)** is still present and can continue dealing damage, and repositioning could expose your Stalkers to a brief period of risk, as the move may slightly delay further attacks. - No units have died, and no significant he...

  59. [69]

    Concentrated Fire (Most Vulnerable Unit)

    The strategy's effectiveness was further validated by successful repositioning to safer areas, reducing exposure to risk and minimizing damage, particularly to the low-health Stalker. Experience: The "Concentrated Fire (Most Vulnerable Unit)" strategy is a **good** hypothesis....

  60. [70]

    The hypothesis achieved good results in all validations, demonstrating the benefits of targeting the most vulnerable enemy unit first to reduce enemy numbers quickly

  61. [71]

    Concentrated Fire (Most Vulnerable Unit)

    The strategy's effectiveness was further validated by successful repositioning to safer areas, reducing exposure to risk and minimizing damage, particularly to the low-health Stalker. Experience: The "Concentrated Fire (Most Vulnerable Unit)" strategy is a **good** hypothesis....

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.