Pith. sign in

REVIEW 3 major objections 6 minor 20 references

elsciRL: Integrating Language Solutions into Reinforcement Learning Problem Settings

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-generated instruction sub-goals can improve reinforcement learning agents, a new open-source library shows.

desk verdict A real but modest software library paper with an honest under-powered evaluation; the unquantified instruction-to-state mapping is the main risk, yet the paper deserves referee time as a software contribution. read the letter →

arxiv 2507.08705 v1 pith:U3RHDBIN submitted 2025-07-11 cs.AI

Philip Osborne , Danilo S. Carvalho , André Freitas This is my paper · ORCID
classification cs.AI
keywords reinforcementlearninglargelanguagemodelsinstructionfollowingrewardshapingadapterself-completinginstructionsdeepQ-networkQ-learning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

elsciRL is an open-source Python library whose stated purpose is to make language-based solutions easy to apply to reinforcement learning problems. The paper extends the Language Adapter with Self-Completing Instruction framework by adding large language model components: an adapter that turns numeric or symbolic states into text, a planner that breaks a user instruction into sub-steps, and a validator that checks whether a predicted state completes the current instruction. The central claim is that LLM-generated, self-completing instructions can improve the testing performance of a Q-learning agent and, in at least one setting, a Deep-Q Network agent, while remaining re-applicable to new applications with minimal setup. The paper presents results from three small grid-world applications and reports that instruction following changes early-training reward and can improve testing scores, while also noting that the effects are not uniform across all environments.

What carries the argument

The load-bearing mechanism is the self-completing instruction pipeline. First an LLM adapter maps each environment state into a language observation, caching one description per state. An LLM planner then breaks the user's text input into sub-instructions, and each sub-instruction is matched against the cached language observations by cosine similarity of text embeddings to predict a completion state. An LLM validator checks the match; if it rejects it, a small negative signal adjusts the match and the LLM is asked to reflect and refine the instruction, repeating until validation passes or a limit is reached. Confirmed completion states become sub-goals that provide an additional training reward, which is removed during testing, so the final evaluation reflects only the underlying environment reward.

What would settle it

Take an environment with known ground-truth completion states for a set of LLM-generated instructions, count how often the cosine-similarity match and the LLM validator pick the wrong state, and then check whether the training reward bonus fires at those wrong states; if the wrong-state rate is high while test scores still improve, the reported gains are not caused by correct instruction following.

Watch

Extended reading notes

Core claim

On the paper's own terms, the finding is that LLM adapters and LLM-based self-completing instructions can improve a reinforcement learning agent's final test performance, measured after the instruction-following bonus is removed. The strongest reported example is the Classroom environment, where the Q-learning agent's testing score improves relative to the non-LLM baseline, and the authors likewise attribute the DQN improvement to the instruction-following combination. The paper does not claim universal gains: FrozenLake and the Maze environments show mixed or negative results, and the authors observe that the DQN underperforms the tabular Q-learning agent on these simple problems. The intended contribution is therefore not a new algorithm but a reusable software layer that lets language guidance be added to reward-based environments without custom RL engineering.

Load-bearing premise

The load-bearing premise is that the unsupervised cosine-similarity match between an LLM-generated instruction and a cached language observation correctly identifies the state that completes the instruction, with the LLM validator compensating for mistakes; the paper only inspects this by eye on four simple environments and does not quantify the matching or validation error.

Editorial extensions

If this is right

  • Language guidance can be added to an existing reward-based environment without changing the test-time reward, because the instruction bonus is applied only during training.
  • The instruction-following mechanism is agent-agnostic: completion states are defined in the environment's numeric state representation, so the trained agent does not itself need to be an LLM.
  • The library is designed so that the LLM model, language encoder, planner, and validator can be swapped, enabling ablation-style evaluation of each language component with minimal setup.
  • If the reported gains reproduce, this provides a low-effort way for domain specialists to use LLM-generated sub-goals as reward shaping in small discrete-state environments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit is to measure the cosine-similarity matcher's accuracy against ground-truth completion states on harder environments; its current validation is visual inspection on four simple grids.
  • The mixed results suggest that instruction-following reward can anchor an agent at a sub-goal instead of the final goal; decaying the bonus during training or ordering sub-goals as a curriculum would test that interpretation.
  • The 'fixed agents, varied problem specification' evaluation protocol could serve as a reusable benchmark for language-as-reward-shaping outside this library.
  • Caching every state's language description for embedding matching may become costly in large or continuous state spaces; nearest-neighbour indexing or validator-proposed alternatives are natural extensions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents elsciRL, an open-source Python library that integrates language-based solutions into reinforcement learning environments. It extends the LASIF framework from Osborne (2024) by adding LLM-based adapters, an LLM planner that decomposes user instructions into sub-instructions, and an LLM validator for unsupervised instruction-to-state matching. The library includes a GUI and is evaluated on four environments (Classroom, FrozenLake, UMaze, Double-T Maze) with Q-learning and DQN agents. The central empirical claim is that LLM-generated self-completing instructions can improve agent performance, as shown in Table 3, while acknowledging that results are mixed.

Significance. If the central claim is supported, elsciRL is a useful open-source software contribution that lowers the barrier to applying LLM-based instruction following in RL. The shipped code, GUI, and reproducible experiment configurations are concrete strengths, and the paper is honest about some limitations. However, the empirical evidence as presented is not statistically robust, and the unquantified instruction-to-state matching pipeline is a load-bearing risk. The significance of the work is therefore conditional on strengthening the evaluation.

major comments (3)
  1. [Section 2.2.2 and Table 2] The cosine-similarity instruction-to-state mapping and the LLM validator meant to correct mismatches are never quantitatively evaluated. The paper justifies the match quality only by manual inspection of four simple environments (Section 4: 'due to the simplicity of the environments we can confirm that these are valid'). No data are reported on how often the LLM validator rejected a candidate match, how often the user confirmation step changed a prediction, or how accurate the cosine-similarity search is against ground truth. Because the training reward is given for reaching the predicted completion state, an incorrect match would turn measured improvements into reward-shaping artifacts rather than evidence of instruction following. The user-confirmation step mitigates this risk in the four demos, but the paper claims the library is re-appliable with minimal setup, where no such manual quality control is assumed. Please quantify matching accuracy on held-out annotated instructions, report validator rejection rates, and report the frequency and effect of user corrections.
  2. [Table 3 and Section 4] The central empirical claim that instruction following improves performance is not supported by appropriate statistical reporting. Table 3 reports only point estimates from the best of 10 training repeats, with no error bars, confidence intervals, or significance tests. The results are mixed: for example, the UMaze Q-learning score drops from 1.00 baseline to -0.10 with the LLM instruction-following combination, while the DQN score improves from -0.10 to 0.12; the Classroom Q-learning language row improves from 0.12 to 0.16, but the numeric row in the same environment shows the instruction following combination (-0.33) better than the LLM adapter alone (-0.78) yet worse than the base (-0.48). The best-of-10 selection can inflate apparent improvements, and the paper itself acknowledges that 10 repeats 'may still be insufficient for proving definitive results' (Section 4). Despite this, the abstract and introduction conclude that 'LLM approaches can be used to improve the performance.' Please report the full distribution across repeats (e.g., medians with interquartile ranges as in Agarwal et al., 2021, which is cited), and run paired significance tests or equivalent statistical comparisons.
  3. [Section 2.2.1 and 2.2.2] The observed-state cache used for unsupervised instruction completion is collected by random exploration, but the paper does not discuss state-space coverage or the behavior when the true completion state is absent from the cache. In sparse or larger mazes, random exploration may never reach the intended target cell, causing the cosine-similarity search to select a nearest but incorrect state. The paper should either ensure the cache covers the relevant state space, report the coverage rate, or describe how the pipeline handles missing completion states. Without this information, the reliability of the Table 2 matches (and the Table 3 results that depend on them) cannot be assessed outside the four specific environments shown.
minor comments (6)
  1. [Section 4] The sentence 'the instruction following approach can improve of the agent's performance' contains a grammar error; it should read 'can improve the agent's performance.'
  2. [Appendix C] Several figure captions appear to be mislabeled: Figures 13, 17, 21, and 25 are placed in the testing section but are captioned as 'NO instruction training results'; they should be 'testing results'.
  3. [Table 3] The table layout is difficult to parse because adapter-type rows (Numeric, Language) are not visually separated from the column headers, and the eight numeric columns are not clearly grouped by agent. Please restructure the table so that the Q-learning and DQN blocks, and the Base/Adapter/LLM Adapter/LLM IF Comb columns, are unambiguously labeled.
  4. [Section 1] The claim that this is the 'first attempt at a general-purpose framework' is an overstatement without a broader survey of related software; it would be safer to say that no library is known to the authors that formalizes this specific approach.
  5. [References] The model citation is split awkwardly as '(Grattafiori et al., 2024), (Meta, 2024)', and the Meta (2024) entry refers to 'Ollama/llama3.2' as a web page; please clarify which model version and inference server were actually used.
  6. [Section 2.2.2] The phrase 'Prior works of instructions or sub-goals required the completion to be defined by a human' is awkward; consider rewording for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed performance gains are measured against fixed non-LLM baselines with instruction reward removed at test time, and the instruction-to-state matching is manually verified rather than fitted to the test outcomes.

full rationale

The paper's central claim is empirical: LLM-generated, self-completing instructions can improve Q-learning/DQN performance. The comparison in Table 3 is between fixed agents trained with and without instruction following, under identical hyperparameters, and the additional instruction reward is removed in testing. The unsupervised cosine-similarity matching in Section 2.2.2 is a deterministic mapping from instruction embeddings to cached language observations; it is not fitted to the test rewards, and the resulting sub-goals are manually confirmed in Section 4 ('Due to the simplicity of the environments we can confirm that these are valid'). The repeated citation of Osborne (2024) supplies the LASIF framework and the unsupervised prediction method, but the present work re-implements and evaluates that framework on new applications rather than assuming its conclusion. No equation or fitted parameter is shown to make a 'prediction' equal to its input by construction. Weaknesses such as the unquantified matching accuracy are validity or reproducibility concerns, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's experimental claims rest on several unquantified modeling choices: the cosine similarity matching, the reliability of LLM generation and validation, fixed agent hyperparameters, and the partial observability introduced by language adapters. These are domain assumptions rather than fitted parameters, but they are load-bearing for the reported improvements.

free parameters (3)
  • Number of training and testing episodes and repeats = 10,000 training, 1,000 testing, 10 repeats each
    Chosen by hand as standard practice; acknowledged as possibly insufficient in Section 4.
  • Cosine similarity matching method
    Ad hoc choice for unsupervised instruction-state matching in Section 2.2.2.
  • LLM prompts and negative reinforcement magnitude
    The system prompt and corrective reward are ad hoc, not justified or ablated.
assumptions (4)
  • domain assumption Cosine similarity in MiniLMv6 embedding space reflects semantic closeness sufficient for instruction-state matching.
    Section 2.2.2 relies on this to self-complete instructions.
  • domain assumption LLM-generated instructions and validation are reliable enough to guide training.
    The LLM planner breaks down objectives and the LLM validator checks matches without quantified accuracy.
  • domain assumption Fixed agent hyperparameters are appropriate across all applications.
    All agent parameters are fixed and available in the repo, but not justified per environment.
  • domain assumption Language adapters preserve enough information for the agent to learn.
    Partial observability is accepted; Q-learning cannot transfer across similar states.

how reviews work

0 comments
Cite this review

Pith. "Pith review of elsciRL: Integrating Language Solutions into Reinforcement Learning Problem Settings." pith.science (2026). https://pith.science/paper/U3RHDBIN

@misc{pith2026250708705,
  author       = {Pith},
  title        = {Pith review of: elsciRL: Integrating Language Solutions into Reinforcement Learning Problem Settings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U3RHDBIN}},
  note         = {Machine review of arXiv:2507.08705}
}
read the original abstract

We present elsciRL, an open-source Python library to facilitate the application of language solutions on reinforcement learning problems. We demonstrate the potential of our software by extending the Language Adapter with Self-Completing Instruction framework defined in (Osborne, 2024) with the use of LLMs. Our approach can be re-applied to new applications with minimal setup requirements. We provide a novel GUI that allows a user to provide text input for an LLM to generate instructions which it can then self-complete. Empirical results indicate that these instructions \textit{can} improve a reinforcement learning agent's performance. Therefore, we present this work to accelerate the evaluation of language solutions on reward based environments to enable new opportunities for scientific discovery.

Figures

Figures reproduced from arXiv: 2507.08705 by the authors.

Figure 1
Figure 1. Overview of the elsciRL library, red blocks highlight the Language Adapter and Self-completing Instruction Following (LASIF) framework defined by (Osborne, 2024), blue blocks highlight the methodological contributions of this work with LLMs. meric / symbolic states, 2) an LLM planner module to generate a set of instructions given a user input to complete the task, and 3) an LLM validation pro￾cess that evaluates the… view at source ↗
Figure 2
Figure 2. GUI Application Selection Tab [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. GUI Agent Configuration Selection Tab. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: GUI Instruction Following Input Tab [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: GUI Run Experiment Tab. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 9
Figure 9. Figure 9: Double-T maze environment, start position [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 7
Figure 7. Figure 7: Gym-FrozenLake environment, start position [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Umaze environment, start position fixed at [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 10
Figure 10. Figure 10: Classroom environment Instruction training results. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Classroom environment NO instruction training results. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Classroom environment Instruction testing results. [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: Classroom environment NO instruction training results. [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 14
Figure 14. Figure 14: Gym FrozenLake environment Instruction training results. [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]
Figure 15
Figure 15. Figure 15: Gym FrozenLake environment NO instruction training results. [PITH_FULL_IMAGE:figures/full_fig_p013_15.png]
Figure 16
Figure 16. Figure 16: Gym FrozenLake environment Instruction testing results. [PITH_FULL_IMAGE:figures/full_fig_p014_16.png]
Figure 17
Figure 17. Figure 17: Gym FrozenLake environment NO instruction training results. [PITH_FULL_IMAGE:figures/full_fig_p014_17.png]
Figure 18
Figure 18. Figure 18: Umaze environment Instruction training results. [PITH_FULL_IMAGE:figures/full_fig_p015_18.png]
Figure 19
Figure 19. Figure 19: Umaze environment NO instruction training results. [PITH_FULL_IMAGE:figures/full_fig_p015_19.png]
Figure 20
Figure 20. Figure 20: Umaze environment Instruction testing results. [PITH_FULL_IMAGE:figures/full_fig_p016_20.png]
Figure 21
Figure 21. Figure 21: Umaze environment NO instruction training results. [PITH_FULL_IMAGE:figures/full_fig_p016_21.png]
Figure 22
Figure 22. Figure 22: Double-t maze environment Instruction training results. [PITH_FULL_IMAGE:figures/full_fig_p017_22.png]
Figure 23
Figure 23. Figure 23: Double-t maze environment NO instruction training results. [PITH_FULL_IMAGE:figures/full_fig_p017_23.png]
Figure 24
Figure 24. Figure 24: Double-t maze environment Instruction testing results. [PITH_FULL_IMAGE:figures/full_fig_p018_24.png]
Figure 25
Figure 25. Figure 25: Double-t maze environment NO instruction training results. [PITH_FULL_IMAGE:figures/full_fig_p018_25.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 12 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Courville, and Marc G

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. 2021. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems (NeuIPS)

    work page 2021

  4. [4]

    Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. 2021. http://jmlr.org/papers/v22/18-056.html Mushroomrl: Simplifying reinforcement learning research . Journal of Machine Learning Research, 22(131):1--5

    work page 2021

  5. [5]

    Koushik, Zhiyuan Hu, Ying Wen, and Jun Wang

    Xidong Feng, Ziyu Wan, Haotian Fu, Bo Liu, Mengyue Yang, Girish A. Koushik, Zhiyuan Hu, Ying Wen, and Jun Wang. 2024. http://arxiv.org/abs/2411.14251 Natural language reinforcement learning

    arXiv 2024

  6. [6]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and et al. 2024. http://arxiv.org/abs/2407.21783 The llama 3 herd of models

    arXiv 2024

  7. [7]

    Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. 2019. Hierarchical decision making by generating and following natural language instructions. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), pages 10025--10034

    work page 2019

  8. [8]

    Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. 2022. http://jmlr.org/papers/v23/21-1342.html Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms . Journal of Machine Learning Research, 23(274):1--18

    work page 2022

Show all 20 references
  1. [9]

    Gonzalez, Michael I

    Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. 2018. RLlib : Abstractions for distributed reinforcement learning. In International Conference on Machine Learning ( ICML )

    2018

  2. [10]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: it...

    2023

  3. [11]

    Meta. 2024. https://ollama.com/library/llama3.2 Ollama/llama3.2 . Web page

    2024

  4. [12]

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. https://doi.org/10.48550/arXiv.1312.5602 Playing Atari with Deep Reinforcement Learning

    2013 doi arXiv

  5. [13]

    Philip Osborne. 2024. Improving Real-World Reinforcement Learning by Self Completing Human Instructions on Rule Defined Language. Phd thesis, The University of Manchester, Manchester, UK

    2024

  6. [14]

    Philip Osborne, Heido N \ o mm, and Andr \'e Freitas. 2022 a . https://doi.org/10.1162/tacl_a_00495 A Survey of Text Games for Reinforcement Learning Informed by Natural Language . Transactions of the Association for Computational Linguistics, 10:873--887

    2022 doi

  7. [15]

    Philip Osborne, Kajal Singh, and Matthew E. Taylor. 2022 b . Applying Reinforcement Learning on Real-World Data with Practical Examples in Python . Synthesis Lectures on Artificial Intelligence and Machine Learning . Springer International Publishing, Cham

    2022

  8. [16]

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. 2021. http://jmlr.org/papers/v22/20-1364.html Stable-baselines3: Reliable reinforcement learning implementations . Journal of Machine Learning Research, 22(268):1--8

    2021

  9. [17]

    Tianmin Shu, Caiming Xiong, and Richard Socher. 2018. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. International Conference on Learning Representations (ICLR), 6

    2018

  10. [18]

    Mankowitz, and Shie Mannor

    Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J. Mankowitz, and Shie Mannor. 2017. A deep hierarchical approach to lifelong learning in minecraft. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, page 1553–1561. AAAI Press

    2017

  11. [19]

    Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goul \ a o, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. 2024. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032

    2024 arXiv

  12. [20]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. http://arxiv.org/abs/2002.10957 Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    2020 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.