Pith. sign in

REVIEW 4 major objections 5 minor 113 references

Pre-trained LLMs used as tutors accelerate reinforcement-learning convergence while leaving final performance largely unchanged, according to 54 experiments.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LLM tutoring modestly accelerates RL convergence on average, with advice reuse saving wall-clock time but reducing stability.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A competent, honest empirical study that combines LLM tutoring with advice reuse, but the 'significantly accelerates' claim rests on an unvalidated metric and N=3 runs with no significance tests. the 4 major comments →

arxiv 2509.08329 v1 pith:DRP6ZV6N submitted 2025-09-10 cs.LG cs.AI

Accelerating Reinforcement Learning Algorithms Convergence using Pre-trained Large Language Models as Tutors With Advice Reusing

Lukas Toral , Teddy Lazebnik This is my paper
classification cs.LG cs.AI
keywords reinforcement learninglarge language modelsLLM tutorstudent-teacher architectureadvice reuseconvergence accelerationDQNPPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a pre-trained large language model, acting as a tutor that suggests actions to a reinforcement-learning agent, can make the agent converge faster than it would by trial and error alone. Across 54 combinations of three RL algorithms, three open-source LLMs, and three game environments, the authors report that LLM-suggested actions accelerate convergence while leaving final performance roughly unchanged. They also show that storing and reusing the LLM's advice for a fixed number of revisits shortens training time further, at the cost of less stable convergence dynamics. The practical stake is large: if true, practitioners can cut RL training time in sparse-reward environments without hand-built reward shaping or curriculum design, by coupling an off-the-shelf LLM to a standard algorithm. The authors stress that the benefit is configuration-sensitive and can even reverse in some combinations.

Core claim

The paper's central claim is that a pre-trained large language model can act as a tutor in a student-teacher setup and make a reinforcement-learning agent converge faster, without changing the algorithm's eventual performance. The evidence is an empirical comparison of 54 configurations: three RL algorithms (DQN, PPO, A2C) trained on three games (Blackjack, Connect Four, Snake), each combined with one of three open-source LLMs (Llama 3.1, Vicuna, DeepSeek-R1), against a no-tutor baseline. Convergence is measured by P, the integral of the globally normalized training curve. On average, the largest tutor (DeepSeek) beat the baseline (0.6058 vs 0.5811), Vicuna matched it, and the smallest (Llam

What carries the argument

The mechanism is an LLM-in-the-loop action selector grafted onto the RL agent. At each step, with a probability that decays linearly from 1.0 to 0.1 over a fixed number of training steps, the agent asks the LLM for an action instead of following its own policy or taking a random exploration action; the LLM's reply is constrained by prompt templates to be an integer inside <action> tags. Advice reuse adds a look-up dictionary keyed by a hash of the state; the first time the LLM advises in a state, the action is stored with a budget of 3, and subsequent visits to the same state reuse the stored action until the budget is exhausted. The comparison runs on the P metric, the integral of each trai

Load-bearing premise

The results rest on a customized convergence score that normalizes every training curve by the single best and worst reward seen across all configurations in that group; if one extreme curve stretches the scale, the ranking by this score may not reflect how fast agents truly converge.

What would settle it

Recompute the Table 2 convergence scores after deleting one curve at a time from each environment-algorithm group, re-deriving the min-max normalization each time; if a tutored-versus-baseline ordering flips, the reported acceleration is an artifact of the global scale. Separately, run 30 seeds per configuration and compare final-policy rewards; if untutored agents match or beat tutored agents at the end of training, the claim of comparable optimal performance fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the results hold, LLM tutoring can be added to standard DQN, PPO, and A2C implementations as an action-sampling layer, with no change to the loss or update rule, so acceleration is achievable with modest engineering.
  • The advice-reuse budget of three uses per state cuts the number of LLM calls and wall-clock training time, most when the tutor is slow; the largest reported saving, 596 minutes, came from the slowest tutor.
  • Because the smallest LLM often underperforms the baseline, practitioners cannot assume any LLM helps; tutor scale and reasoning ability matter for the effect.
  • The acceleration is concentrated in the early, exploratory phase of training, since the probability of consulting the LLM decays from 1.0 to 0.1, implying LLM guidance substitutes for random exploration rather than improving the policy asymptote.
  • The observed sensitivity across configurations means LLM tutoring should be evaluated per task rather than adopted as a universal speed-up.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The non-significant size-performance correlation (r=0.24, p=0.14) suggests the DeepSeek advantage may come from reasoning style rather than raw parameter count; a replication holding model size constant across families would separate these.
  • A cheap extension is to make the reuse budget adaptive, for example reusing advice longer in states where the stored action has historically raised the value estimate; the paper only tests the fixed-budget variant, though it mentions Q-change and decay alternatives.
  • Because P integrates normalized training curves, the claim of comparable optimal performance is about area under the curve, not final return; a final-policy evaluation on more seeds could change how the result reads in applied settings.
  • In simple environments where LLM response time dominates, a distilled small tutor fine-tuned on the large model's advice could preserve acceleration at a fraction of the wall-clock cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a student-teacher architecture in which a pre-trained open-source LLM (Llama, Vicuna, DeepSeek) acts as a tutor for an RL agent (DQN, PPO, A2C), with optional reuse of previously generated advice. The RL agent probabilistically consults the LLM during training, with the consultation probability decaying over time; advice is stored in a hash table and reused up to a fixed budget. The authors report experiments on Blackjack, Connect Four, and Snake across 54 configurations and introduce a normalized area-under-curve metric P (Eq. 2) to measure convergence speed. They claim that LLM tutoring significantly accelerates convergence while preserving final performance, and that advice reuse further reduces training time at the cost of stability. The paper includes a limitations section acknowledging the small number of repetitions and lack of human evaluation of advice quality.

Significance. If the empirical claims were firmly established, the work would be a useful practical contribution: it shows a simple, open-source, zero-shot way to inject LLM knowledge into standard RL algorithms, and it explicitly studies a reuse mechanism that is under-explored in the LLM-as-teacher literature. The 54-configuration sweep across three environments, three RL algorithms, and three LLMs is a genuine strength, as is the candid acknowledgment of limitations. However, the current evidence does not yet support the strength of the abstract's 'significantly accelerates' claim: the central metric P is introduced and used without validation, the statistical basis is thin (N=3, no error bars or tests), and the reuse 'time saved' figure is an arithmetic identity rather than an independently measured quantity. The paper's contribution is therefore plausible but not yet demonstrated at the level claimed.

major comments (4)
  1. [§3.3, Eq. (2)] All quantitative conclusions in Section 4 and 5 rest on the convergence metric P, defined as the area under a training curve normalized by the global minimum and maximum across all configurations in the same environment-algorithm group. This metric is introduced in this paper and is never validated against final-policy performance, average return, or any established convergence measure. Because the normalization uses global extrema, a single outlier curve (e.g., one early high-reward run) can compress the other curves and change the ordering of P values across configurations. The authors should validate P against at least one independent performance measure, and report per-seed P values or an alternative normalization that is robust to outliers.
  2. [Table 2 and abstract] The abstract claims that 'LLM tutoring significantly accelerates RL convergence,' but Table 2 reports only a single P value per configuration, with N=3 repetitions and no standard deviations, confidence intervals, or significance tests. Several configurations in Table 2 show the LLM-tutored run below the baseline (e.g., Blackjack/PPO/LLaMA 0.4874 vs baseline 0.5345; Connect Four/A2C/Vicuna 0.5252 vs baseline 0.5454; Snake/PPO/Vicuna 0.4551 vs baseline 0.5557). The aggregate average improvement is therefore sensitive to the chosen metric and the averaging method. To support 'significantly,' the authors need to report seed-level variance, perform paired comparisons or effect-size estimates, and state how many of the 18 comparisons improve and by how much.
  3. [Table 3 and §5] The 'time saved' in Table 3 is computed as the number of advice reuses multiplied by the LLM's average response time. This is true by construction, not an independent measurement of wall-clock training time. It does not account for the time spent generating the initial advice, the overhead of LLM prompting and parsing, or any effect of reuse on the number of training steps. In addition, the claim that reuse yields a 'further improvement' is not consistent across Table 2: e.g., Snake/DQN with Vicuna drops from 0.5005 to 0.4379, and Blackjack/DQN with DeepSeek drops from 0.6211 to 0.6073. The authors should either report actual end-to-end training times or explicitly state that Table 3 is a back-of-the-envelope estimate of the maximum possible saving, not a measured training-duration reduction.
  4. [§3.2.3] The decay period θ for the LLM consultation probability is manually tuned per environment (3000/15000 steps for Blackjack, 1000/10000 for Connect Four, 1000/8000 for Snake), with no sensitivity analysis. The main claim is therefore conditional on these hand-picked values. Since θ is a free parameter of the proposed method, the authors should either show that results are robust to reasonable variations in θ or explicitly discuss how sensitive conclusions are to this choice.
minor comments (5)
  1. [Listing 1] Typo in the caption: 'environemnt' should be 'environment'.
  2. [§5] The closing paragraph says LLM tutoring 'increase[s] convergence time,' which appears to be the opposite of the paper's claim; likely should be 'decrease' or 'accelerate.'
  3. [Fig. 2 / Fig. 3] The text says 'Fig. 3 illustrates the three environments,' but the environments are shown in Fig. 2; Fig. 3 shows learning curves. Please renumber or cross-reference correctly.
  4. [Algorithm 1] The pseudocode does not define the hash function, the dictionary O's initialization, or the structure of the action object a. For reproducibility, these details should be specified.
  5. [Eq. (2)] The notation is slightly confusing: p is used both as a point in a vector and as the normalized value, and the formula '∀p∈p̂' is not well-formed. Please rewrite with distinct symbols for raw reward, normalized reward, and the integral.

Circularity Check

1 steps flagged

Reuse time-saving is an arithmetic identity (reuses × response time); the main LLM-tutoring acceleration comparison is otherwise an independent empirical measurement.

specific steps
  1. self definitional [Section 4 Results, Table 3 and accompanying paragraph]
    "Table 3 presents an overview of the time saved in minutes during the training of RL agents when reusing advice generated by the selected LLMs across the three selected environments: Blackjack, Connect Four, and Snake. The table includes the average number of advice reuses, the LLM’s average response time in seconds, and the corresponding training time saved."

    The 'time saved' column is not an independently measured wall-clock difference. Multiplying the reported reuse counts by the reported average response times reproduces the saved-time entries exactly (e.g., 901 × 7.33 s = 110 min for Blackjack/Vicuna; 172 × 208 s = 596 min for Connect Four/DeepSeek). Thus the paper's conclusion that 'the advice reuse mechanism shows a further improvement in training duration' follows by construction from the definition of the metric: any positive reuse count yields positive 'time saved'. It is an arithmetic identity, not an empirical result.

full rationale

The central claim—that LLM tutoring accelerates RL convergence—is an empirical comparison of reward curves, not a derivation. The convergence metric P (Eq. 2) is a normalization of observed training rewards; while it is unvalidated and potentially outlier-sensitive, it is not circular because it is not fitted to the conclusions nor defined in terms of them. The LLM tutoring results are therefore an independent empirical measurement. The one genuinely circular element is the reuse benefit: Table 3's 'time saved' is exactly the product of the average number of reuses and the average LLM response time, so the reported 'further improvement in training duration' is true by construction rather than an independently measured saving. This affects a secondary claim, not the paper's main acceleration result. The paper contains no load-bearing self-citations; the LLM-integration and advice-reuse mechanisms are adapted from external prior works [25,80,84]. The authors' own limitations (N=3, unverified advice quality) weaken reliability but do not constitute circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The claimed gains rest on hand-tuned consultation schedules, a reuse budget taken from prior work, and an unvalidated convergence metric. The paper introduces no new theoretical entities. The LLM tutor is a pre-existing tool and the reuse mechanism is adapted from [84].

free parameters (2)
  • LLM consult probability decay period (per environment) = 3000 steps (Blackjack, 15000 total), 1000 (Connect Four, 10000 total), 1000 (Snake, 8000 total)
    Section 3.2.3: 'These numbers were picked following a manual trial-and-error process.' The decay schedule determines how long LLM advice is injected and affects the measured convergence benefit.
  • Advice reuse budget = 3
    Set 'consistent with the methodology in [84]' (§3.1.3). This budget determines how many times advice is reused and is a tuning choice that affects the reuse benefit and instability.
axioms (4)
  • domain assumption The state hashing in Algorithm 1 maps identical states to the same key, so reused advice remains applicable.
    Algorithm 1 lines 2-9; if the environment is stochastic or the state representation is lossy, advice may become stale.
  • domain assumption LLM zero-shot advice is reliable enough to guide learning without human verification.
    Discussion: 'the quality of LLM advice was not manually or systematically evaluated by human experts.'
  • ad hoc to paper P in Eq. (2), computed with global min/max normalization, is a faithful measure of convergence speed across configurations.
    Section 3.3; the metric is newly proposed and not validated against any external benchmark or statistical test.
  • standard math RL algorithms (DQN, PPO, A2C) converge to optimal policies in the three Gymnasium environments under the given hyperparameters.
    Standard convergence results are cited (e.g., [43]), but the finite training horizon may not reach optimality.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Accelerating Reinforcement Learning Algorithms Convergence using Pre-trained Large Language Models as Tutors With Advice Reusing." pith.science (2026). https://pith.science/paper/DRP6ZV6N

@misc{pith2026250908329,
  author       = {Pith},
  title        = {Pith review of: Accelerating Reinforcement Learning Algorithms Convergence using Pre-trained Large Language Models as Tutors With Advice Reusing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DRP6ZV6N}},
  note         = {Machine review of arXiv:2509.08329}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reinforcement Learning (RL) algorithms often require long training to become useful, especially in complex environments with sparse rewards. While techniques like reward shaping and curriculum learning exist to accelerate training, these are often extremely specific and require the developer's professionalism and dedicated expertise in the problem's domain. Tackling this challenge, in this study, we explore the effectiveness of pre-trained Large Language Models (LLMs) as tutors in a student-teacher architecture with RL algorithms, hypothesizing that LLM-generated guidance allows for faster convergence. In particular, we explore the effectiveness of reusing the LLM's advice on the RL's convergence dynamics. Through an extensive empirical examination, which included 54 configurations, varying the RL algorithm (DQN, PPO, A2C), LLM tutor (Llama, Vicuna, DeepSeek), and environment (Blackjack, Snake, Connect Four), our results demonstrate that LLM tutoring significantly accelerates RL convergence while maintaining comparable optimal performance. Furthermore, the advice reuse mechanism shows a further improvement in training duration but also results in less stable convergence dynamics. Our findings suggest that LLM tutoring generally improves convergence, and its effectiveness is sensitive to the specific task, RL algorithm, and LLM model combination.

Figures

Figures reproduced from arXiv: 2509.08329 by Lukas Toral, Teddy Lazebnik.

Figure 1
Figure 1. Figure 1: A schematic view of student-teacher structure for RL and LLM. When the RL agent selects an action to [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visual presentation of all three selected RL environments. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The normalized collected reward over training for all three RL algorithms with their respective LLM [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

113 extracted references · 35 canonical work pages · 8 internal anchors

  1. [1]

    Leveraging more of biology in evolutionary reinforcement learning

    Bruno Gašperov, Marko Ðurasevi ´c, and Domagoj Jakobovic. “Leveraging more of biology in evolutionary reinforcement learning”. In:International conference on the applications of evolutionary computation (Part of EvoStar). Springer. 2024, pp. 91–114

    2024

  2. [2]

    Reinforcement learning: A survey

    Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. “Reinforcement learning: A survey”. In:Journal of artificial intelligence research4 (1996), pp. 237–285

    1996

  3. [3]

    Model-based reinforcement learning: A survey

    Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. “Model-based reinforcement learning: A survey”. In:Foundations and Trends® in Machine Learning16.1 (2023), pp. 1–118

    2023

  4. [5]

    Implementation of Q-Learning algorithm for solving maze prob- lem

    D Osmankovi ´c and Samim Konjicija. “Implementation of Q-Learning algorithm for solving maze prob- lem”. In:2011 proceedings of the 34th international convention MIPRO. IEEE. 2011, pp. 1619–1622

    2011

  5. [6]

    Sequence learning: From recognition and prediction to sequential decision making

    Ron Sun and C Lee Giles. “Sequence learning: From recognition and prediction to sequential decision making”. In:IEEE Intelligent Systems16.4 (2001), pp. 67–70. Draft: September 21, 2025 17

    2001

  6. [7]

    Deep reinforcement learning for sequence-to-sequence models

    Yaser Keneshloo, Tian Shi, Naren Ramakrishnan, and Chandan K Reddy. “Deep reinforcement learning for sequence-to-sequence models”. In:IEEE transactions on neural networks and learning systems31.7 (2019), pp. 2469–2489

    2019

  7. [9]

    Adaptive look-ahead economic dispatch based on deep reinforcement learning

    Xinyue Wang, Haiwang Zhong, Guanglun Zhang, Guangchun Ruan, Yiliu He, and Zekuan Yu. “Adaptive look-ahead economic dispatch based on deep reinforcement learning”. In:Applied Energy353 (2024), p. 122121

    2024

  8. [10]

    Recent developments of game theory and reinforce- ment learning approaches: A systematic review

    Garima Jain, Arun Kumar, and Shahid Ahmad Bhat. “Recent developments of game theory and reinforce- ment learning approaches: A systematic review”. In:IEEE Access12 (2024), pp. 9999–10011

    2024

  9. [11]

    Recent advances in reinforcement learning-based autonomous driving behavior planning: A survey

    Jingda Wu, Chao Huang, Hailong Huang, Chen Lv, Yuntong Wang, and Fei-Yue Wang. “Recent advances in reinforcement learning-based autonomous driving behavior planning: A survey”. In:Transportation Research Part C: Emerging Technologies164 (2024), p. 104654

    2024

  10. [12]

    Rein- forcement learning for autonomous process control in industry 4.0: Advantages and challenges

    Nuria Nievas, Adela Pagès-Bernaus, Francesc Bonada, Lluís Echeverria, and Xavier Domingo. “Rein- forcement learning for autonomous process control in industry 4.0: Advantages and challenges”. In:Ap- plied Artificial Intelligence38.1 (2024), p. 2383101

    2024

  11. [13]

    A review of safe reinforcement learning: Methods, theories and applications

    Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. “A review of safe reinforcement learning: Methods, theories and applications”. In:IEEE Transactions on Pattern Analysis and Machine Intelligence(2024)

    2024

  12. [14]

    Distributed deep reinforcement learning based gradient quantization for federated learning enabled vehi- cle edge computing

    Cui Zhang, Wenjun Zhang, Qiong Wu, Pingyi Fan, Qiang Fan, Jiangzhou Wang, and Khaled B Letaief. “Distributed deep reinforcement learning based gradient quantization for federated learning enabled vehi- cle edge computing”. In:IEEE Internet of Things Journal(2024)

    2024

  13. [15]

    A review of research on reinforcement learning algorithms for multi-agents

    Kai Hu, Mingyang Li, Zhiqiang Song, Keer Xu, Qingfeng Xia, Ning Sun, Peng Zhou, and Min Xia. “A review of research on reinforcement learning algorithms for multi-agents”. In:Neurocomputing(2024), p. 128068

    2024

  14. [16]

    Toward enhanced reinforcement learning-based resource management via digital twin: Opportunities, applications, and challenges

    Nan Cheng, Xiucheng Wang, Zan Li, Zhisheng Yin, Tom Luan, and Xuemin Sherman Shen. “Toward enhanced reinforcement learning-based resource management via digital twin: Opportunities, applications, and challenges”. In:IEEE Network(2024)

    2024

  15. [17]

    Human-level control through deep reinforcement learning

    V olodymyr Mnih et al. “Human-level control through deep reinforcement learning”. In:nature518.7540 (2015), pp. 529–533

    2015

  16. [18]

    Evolu- tionary reinforcement learning: a systematic review and future directions

    Yuanguo Lin, Fan Lin, Guorong Cai, Hong Chen, Linxin Zou, Yunxuan Liu, and Pengcheng Wu. “Evolu- tionary reinforcement learning: a systematic review and future directions”. In:Mathematics13.5 (2025), p. 833

    2025

  17. [19]

    Rl-pruner: Structured pruning using reinforcement learning for cnn compression and acceleration

    Boyao Wang and V olodymyr Kindratenko. “Rl-pruner: Structured pruning using reinforcement learning for cnn compression and acceleration”. In:arXiv preprint arXiv:2411.06463(2024)

    Pith/arXiv arXiv 2024

  18. [20]

    A Reinforcement Learning Training Acceleration Method Based on Knowledge Distillation

    Zhiwen Wang, Rujian Zhang, Yanke Xu, and Qing Fei. “A Reinforcement Learning Training Acceleration Method Based on Knowledge Distillation”. In:2024 43rd Chinese Control Conference (CCC). IEEE. 2024, pp. 8939–8944. Draft: September 21, 2025 18

    2024

  19. [21]

    Plan-based reward shaping for reinforcement learning

    Marek Grzes and Daniel Kudenko. “Plan-based reward shaping for reinforcement learning”. In:2008 4th International IEEE Conference Intelligent Systems. V ol. 2. IEEE. 2008, pp. 10–22

    2008

  20. [22]

    Cur- riculum learning for reinforcement learning domains: A framework and survey

    Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. “Cur- riculum learning for reinforcement learning domains: A framework and survey”. In:Journal of Machine Learning Research21.181 (2020), pp. 1–50

    2020

  21. [23]

    Learning for a robot: Deep reinforcement learning, imitation learning, transfer learning

    Jiang Hua, Liangcai Zeng, Gongfa Li, and Zhaojie Ju. “Learning for a robot: Deep reinforcement learning, imitation learning, transfer learning”. In:Sensors21.4 (2021), p. 1278

    2021

  22. [24]

    Meta-learning in reinforcement learning

    Nicolas Schweighofer and Kenji Doya. “Meta-learning in reinforcement learning”. In:Neural Networks 16.1 (2003), pp. 5–9

    2003

  23. [25]

    How Can LLM Guide RL? A Value-Based Approach

    Shenao Zhang, Sirui Zheng, Shuqi Ke, Zhihan Liu, Wanxin Jin, Jianbo Yuan, Yingxiang Yang, Hongxia Yang, and Zhaoran Wang. “How can llm guide rl? a value-based approach”. In:arXiv preprint arXiv:2402.16181 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  24. [26]

    Gpt-4 technical report

    Josh Achiam et al. “Gpt-4 technical report”. In:arXiv preprint arXiv:2303.08774(2023)

    Pith/arXiv arXiv 2023

  25. [27]

    DeepSeek-AI et al.DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learn- ing. 2025. arXiv:2501.12948 [cs.CL].URL:https://arxiv.org/abs/2501.12948

    Pith/arXiv arXiv 2025

  26. [28]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning

    Tian Xie et al. “Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning”. In:arXiv preprint arXiv:2502.14768(2025)

    Pith/arXiv arXiv 2025

  27. [29]

    Teacher-student archi- tecture for knowledge distillation: A survey

    Chengming Hu, Xuan Li, Dan Liu, Haolun Wu, Xi Chen, Ju Wang, and Xue Liu. “Teacher-student archi- tecture for knowledge distillation: A survey”. In:arXiv preprint arXiv:2308.04268(2023)

    Pith/arXiv arXiv 2023

  28. [30]

    Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMs

    Bahar Radmehr, Adish Singla, and Tanja Käser. “Towards generalizable agents in text-based educational environments: A study of integrating rl with llms”. In:arXiv preprint arXiv:2404.18978(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  29. [31]

    Intelligent Control of Closed-Loop Sedation in Simulated ICU Patients

    Brett Moore, Eric Sinzinger, Todd Quasny, and Larry Pyeatt. “Intelligent Control of Closed-Loop Sedation in Simulated ICU Patients”. In:Proceedings of the Seventeenth International Florida Artificial Intelligence Research Society Conference, FLAIRS 20041 (July 2004)

    2004

  30. [32]

    Xiao-Yang Liu, Hongyang Yang, Qian Chen, Runjia Zhang, Liuqing Yang, Bowen Xiao, and Christina Dan Wang.FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance. 2022. arXiv:2011 . 09607 [q-fin.TR].URL:https : / / arxiv . org / abs / 2011 . 09607

    2022

  31. [33]

    Laura Schultz and Vadim Sokolov.Deep Reinforcement Learning for Dynamic Urban Transportation Problems. 2018. arXiv:1806 . 05310 [stat.ML].URL:https : / / arxiv . org / abs / 1806 . 05310

    2018

  32. [34]

    A reinforcement learning approach to obtain treatment strategies in sequential medi- cal decision problems

    Radhika Poolla. “A reinforcement learning approach to obtain treatment strategies in sequential medi- cal decision problems”. In: 2003.URL:https : / / api . semanticscholar . org / CorpusID : 109170984

    2003

  33. [35]

    Rein- forcement Learning for Closed-Loop Propofol Anesthesia: A Study in Human V olunteers

    Brett Moore, Larry Pyeatt, Vivekanand Kulkarni, P. Panousis, Kevin Padrez, and Anthony Doufas. “Rein- forcement Learning for Closed-Loop Propofol Anesthesia: A Study in Human V olunteers”. In:Journal of Machine Learning Research15 (Feb. 2014), pp. 655–696. Draft: September 21, 2025 19

    2014

  34. [36]

    7. Using Reinforcement Learning in the Algorithmic Trading Problem

    Evgeny Ponomarev, Ivan V . Oseledets, Ivan V . Oseledets, and Andrzej Cichocki. “7. Using Reinforcement Learning in the Algorithmic Trading Problem”. In:Journal of Communications Technology and Electron- ics(2019).DOI:10.1134/S1064226919120131

    work page doi:10.1134/s1064226919120131 2019

  35. [37]

    Deep Reinforcement Learning for Optimizing Finance Portfolio Man- agement

    Yuh-Jong Hu and Shang-Jen Lin. “Deep Reinforcement Learning for Optimizing Finance Portfolio Man- agement”. In:2019 Amity International Conference on Artificial Intelligence (AICAI). 2019, pp. 14–20. DOI:10.1109/AICAI.2019.8701368

    work page arXiv 2019

  36. [38]

    Deep Reinforcement Learning for Solving the Vehicle Routing Problem

    MohammadReza Nazari, Afshin Oroojlooy, Lawrence V . Snyder, and Martin Takác. “Deep Reinforcement Learning for Solving the Vehicle Routing Problem”. In:CoRRabs/1802.04240 (2018). arXiv:1802 . 04240.URL:http://arxiv.org/abs/1802.04240

    Pith/arXiv arXiv 2018

  37. [39]

    Reinforcement Learning Boat Autopilot: A Sample-efficient and Model Predictive Control based Approach

    Yunduan Cui, Shigeki Osaki, and Takamitsu Matsubara. “Reinforcement Learning Boat Autopilot: A Sample-efficient and Model Predictive Control based Approach”. In:2019 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS). 2019, pp. 2868–2875.DOI:10.1109/IROS40897. 2019.8967630

    work page arXiv 2019

  38. [40]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. Cambridge, MA, USA: A Bradford Book, 2018.ISBN: 0262039249

    2018

  39. [41]

    Reinforcement learning in artificial and biological systems

    Emre O Neftci and Bruno B Averbeck. “Reinforcement learning in artificial and biological systems”. In: Nature Machine Intelligence1.3 (2019), pp. 133–143

    2019

  40. [42]

    Introduction to reinforcement learning

    Zihan Ding, Yanhua Huang, Hang Yuan, and Hao Dong. “Introduction to reinforcement learning”. In: Deep reinforcement learning: fundamentals, research and applications(2020), pp. 47–123

    2020

  41. [43]

    A generalized reinforcement-learning model: Convergence and appli- cations

    M. L. Littman and C. Szepesvári. “A generalized reinforcement-learning model: Convergence and appli- cations”. In:ICML. V ol. 96. 1996, pp. 310–318

    1996

  42. [44]

    Q-learning

    Christopher J. C. H. Watkins and Peter Dayan. “Q-learning”. In:Machine Learning8.3 (May 1992), pp. 279–292.ISSN: 1573-0565.DOI:10 . 1007 / BF00992698.URL:https : / / doi . org / 10 . 1007/BF00992698

    1992

  43. [45]

    Value-free reinforcement learning: policy optimization as a minimal model of operant behavior

    Daniel Bennett, Yael Niv, and Angela J Langdon. “Value-free reinforcement learning: policy optimization as a minimal model of operant behavior”. en. In:Curr Opin Behav Sci41 (May 2021), pp. 114–121

    2021

  44. [46]

    Actor-Critic Reinforcement Learning and Ap- plication in Developing Computer-Vision-Based Interface Tracking

    Oguzhan Dogru, Kirubakaran Velswamy, and Biao Huang. “Actor-Critic Reinforcement Learning and Ap- plication in Developing Computer-Vision-Based Interface Tracking”. In:Engineering7.9 (2021), pp. 1248– 1261

    work page 2021

  45. [47]

    OpenAI. Nov. 2022.URL:https://openai.com/index/chatgpt

    work page 2022

  46. [48]

    Gemini Team et al.Gemini: A Family of Highly Capable Multimodal Models. 2024. arXiv:2312.11805 [cs.CL].URL:https://arxiv.org/abs/2312.11805

    Pith/arXiv arXiv 2024

  47. [49]

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao.Large Language Models: A Survey. 2025. arXiv:2402 . 06196 [cs.CL].URL: https://arxiv.org/abs/2402.06196

    Pith/arXiv arXiv 2025

  48. [50]

    Wayne Xin Zhao et al.A Survey of Large Language Models. 2025. arXiv:2303.18223 [cs.CL].URL: https://arxiv.org/abs/2303.18223. Draft: September 21, 2025 20

    Pith/arXiv arXiv 2025

  49. [51]

    Large language models surpass human experts in predicting neuroscience results

    Xiaoliang Luo et al. “Large language models surpass human experts in predicting neuroscience results”. In:Nature Human Behaviour9.2 (Nov. 2024), pp. 305–315.ISSN: 2397-3374.DOI:10.1038/s41562- 024-02046-9.URL:http://dx.doi.org/10.1038/s41562-024-02046-9

    doi:10.1038/s41562- 2024

  50. [52]

    arXiv:2002.08910 [cs.CL].URL:https://arxiv.org/abs/ 2002.08910

    Adam Roberts, Colin Raffel, and Noam Shazeer.How Much Knowledge Can You Pack Into the Parameters of a Language Model?2020. arXiv:2002.08910 [cs.CL].URL:https://arxiv.org/abs/ 2002.08910

    Pith/arXiv arXiv 2020

  51. [53]

    Jeff Da, Ronan Le Bras, Ximing Lu, Yejin Choi, and Antoine Bosselut.Analyzing Commonsense Emer- gence in Few-shot Knowledge Models. 2021. arXiv:2101.00297 [cs.CL].URL:https://arxiv. org/abs/2101.00297

    Pith/arXiv arXiv 2021

  52. [54]

    Wang-Cheng Kang, Jianmo Ni, Nikhil Mehta, Maheswaran Sathiamoorthy, Lichan Hong, Ed Chi, and Derek Zhiyuan Cheng.Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Pre- diction. 2023. arXiv:2305.06474 [cs.IR].URL:https://arxiv.org/abs/2305.06474

    Pith/arXiv arXiv 2023

  53. [55]

    TALLRec: An Effec- tive and Efficient Tuning Framework to Align Large Language Model with Recommendation

    Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. “TALLRec: An Effec- tive and Efficient Tuning Framework to Align Large Language Model with Recommendation”. In:Pro- ceedings of the 17th ACM Conference on Recommender Systems. RecSys ’23. ACM, Sept. 2023, pp. 1007– 1014.DOI:10.1145/3604915.3608857.URL:http://dx.doi.org/10.1145/36...

    arXiv 2023

  54. [56]

    LLM-Rec: Personalized Recommendation via Prompting Large Language Models

    Hanjia Lyu et al. “LLM-Rec: Personalized Recommendation via Prompting Large Language Models”. In:Findings of the Association for Computational Linguistics: NAACL 2024. Ed. by Kevin Duh, He- lena Gomez, and Steven Bethard. Mexico City, Mexico: Association for Computational Linguistics, June 2024, pp. 583–612.DOI:10 . 18653 / v1 / 2024 . findings - naacl . ...

    work page 2024

  55. [57]

    Review-driven Personalized Preference Reasoning with Large Language Models for Recommendation

    Jieyong Kim, Hyunseo Kim, Hyunjin Cho, SeongKu Kang, Buru Chang, Jinyoung Yeo, and Dongha Lee. Review-driven Personalized Preference Reasoning with Large Language Models for Recommendation

    work page

  56. [58]

    Wordcraft: Story Writing With Large Lan- guage Models

    Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. “Wordcraft: Story Writing With Large Lan- guage Models”. In:Proceedings of the 27th International Conference on Intelligent User Interfaces. IUI ’22. Helsinki, Finland: Association for Computing Machinery, 2022, pp. 841–852.ISBN: 9781450391443. DOI:10.1145/3490099.3511105.URL:https://doi.org/10.1145/...

    arXiv 2022

  57. [59]

    A comprehensive survey on integrating large language models with knowledge-based methods

    Wenli Yang, Lilian Some, Michael Bain, and Byeong Kang. “A comprehensive survey on integrating large language models with knowledge-based methods”. In:Knowledge-Based Systems318 (2025), p. 113503. ISSN: 0950-7051.DOI:https://doi.org/10.1016/j.knosys.2025.113503.URL:https: //www.sciencedirect.com/science/article/pii/S0950705125005490

    work page arXiv 2025

  58. [60]

    Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh T. N. Nguyen, Lauren T. May, Geoffrey I. Webb, and Shirui Pan.Large Language Models for Scientific Synthesis, Inference and Explanation. 2023. arXiv: 2310.07984 [cs.AI].URL:https://arxiv.org/abs/2310.07984

    Pith/arXiv arXiv 2023

  59. [61]

    Quim Motger, Alessio Miaschi, Felice Dell’Orletta, Xavier Franch, and Jordi Marco.Leveraging Encoder- only Large Language Models for Mobile App Review Feature Extraction. 2025. arXiv:2408 . 01063 [cs.CL].URL:https://arxiv.org/abs/2408.01063. Draft: September 21, 2025 21

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  60. [62]

    Bosheng Ding et al.Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges. 2024. arXiv:2403 . 02990 [cs.CL].URL:https : / / arxiv . org / abs/2403.02990

    Pith/arXiv arXiv 2024

  61. [63]

    Haochen Zhang, Yuyang Dong, Chuan Xiao, and Masafumi Oyamada.Large Language Models as Data Preprocessors. 2024. arXiv:2308.16361 [cs.AI].URL:https://arxiv.org/abs/2308. 16361

    Pith/arXiv arXiv 2024

  62. [64]

    DiarizationLM: Speaker Diarization Post-Processing with Large Language Models

    Quan Wang, Yiling Huang, Guanlong Zhao, Evan Clark, Wei Xia, and Hank Liao. “DiarizationLM: Speaker Diarization Post-Processing with Large Language Models”. In:Interspeech 2024. ISCA, Sept. 2024, pp. 3754–3758

    work page 2024

  63. [65]

    Data Augmentation for Intent Classification with Off-the-shelf Large Language Models

    Gaurav Sahu, Pau Rodriguez, Issam Laradji, Parmida Atighehchian, David Vazquez, and Dzmitry Bah- danau. “Data Augmentation for Intent Classification with Off-the-shelf Large Language Models”. In:Pro- ceedings of the 4th Workshop on NLP for Conversational AI. Ed. by Bing Liu, Alexandros Papangelis, Stefan Ultes, Abhinav Rastogi, Yun-Nung Chen, Georgios Spi...

    doi:10.18653/v1/ 2022

  64. [66]

    When and how to paraphrase for named entity recognition?

    Saket Sharma, Aviral Joshi, Yiyun Zhao, Namrata Mukhija, Hanoz Bhathena, Prateek Singh, and Sashank Santhanam. “When and how to paraphrase for named entity recognition?” In:Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ed. by Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki. Toronto, Ca...

    doi:10.18653/v1/2023.acl- 2023

  65. [67]

    Does Collaborative Human-LM Dialogue Generation Help Information Extraction from Human Dialogues?

    Bo-Ru Lu et al.Does Collaborative Human-LM Dialogue Generation Help Information Extraction from Human Dialogues?2024. arXiv:2307 . 07047 [cs.CL].URL:https : / / arxiv . org / abs / 2307.07047

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  66. [68]

    Aaron Grattafiori et al.The Llama 3 Herd of Models. 2024. arXiv:2407 . 21783 [cs.AI].URL: https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  67. [69]

    Wei-Lin Chiang et al.Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. Mar. 2023.URL:https://lmsys.org/blog/2023-03-30-vicuna/

    work page 2023

  68. [70]

    Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models

    Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Zhenhailong Wang, and Heng Ji. “Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models”. In:Find- ings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Lin- guistics, 2023, pp. 9070–9084.DOI:10.18653/v1/2023.findings-em...

    work page doi:10.18653/v1/2023.findings-emnlp.608.url:http: 2023

  69. [71]

    Saem Hasan and Sanju Basak.Open-Source AI-Powered Optimization in Scalene: Advancing Python Per- formance Profiling with DeepSeek-R1 and LLaMA 3.2. 2025. arXiv:2502 . 10299 [cs.PL].URL: https://arxiv.org/abs/2502.10299

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  70. [72]

    Shuhe Wang et al.Reinforcement Learning Enhanced LLMs: A Survey. 2025. arXiv:2412 . 10400 [cs.CL].URL:https://arxiv.org/abs/2412.10400. Draft: September 21, 2025 22

    Pith/arXiv arXiv 2025

  71. [73]

    Zae Myung Kim, Anand Ramachandran, Farideh Tavazoee, Joo-Kyung Kim, Oleg Rokhlenko, and Dongyeop Kang.Align to Structure: Aligning Large Language Models with Structural Information. 2025. arXiv: 2504.03622 [cs.CL].URL:https://arxiv.org/abs/2504.03622

    arXiv 2025

  72. [74]

    Minbeom Kim, Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, and Kyomin Jung.Critic- Guided Decoding for Controlled Text Generation. 2022. arXiv:2212.10938 [cs.CL].URL:https: //arxiv.org/abs/2212.10938

    Pith/arXiv arXiv 2022

  73. [75]

    Reward modeling for mitigating toxicity in transformer- based language models

    Farshid Faal, Ketra Schmitt, and Jia Yuan Yu. “Reward modeling for mitigating toxicity in transformer- based language models”. In:Applied Intelligence53.7 (July 2022), pp. 8421–8435.ISSN: 1573-7497.DOI: 10.1007/s10489- 022- 03944- z.URL:http://dx.doi.org/10.1007/s10489- 022- 03944-z

    doi:10.1007/s10489- 2022

  74. [76]

    Safe RLHF: Safe Reinforcement Learning from Human Feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe Reinforcement Learning from Human Feedback. 2023. arXiv:2310.12773 [cs.AI]. URL:https://arxiv.org/abs/2310.12773

    Pith/arXiv arXiv 2023

  75. [77]

    Reflexion: Language Agents with Verbal Reinforcement Learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning. 2023. arXiv:2303.11366 [cs.AI]. URL:https://arxiv.org/abs/2303.11366

    Pith/arXiv arXiv 2023

  76. [78]

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi.CodeRL: Master- ing Code Generation through Pretrained Models and Deep Reinforcement Learning. 2022. arXiv:2207. 01780 [cs.LG].URL:https://arxiv.org/abs/2207.01780

    Pith/arXiv arXiv 2022

  77. [79]

    Yuntao Bai et al.Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. 2022. arXiv:2204.05862 [cs.CL].URL:https://arxiv.org/abs/2204.05862

    Pith/arXiv arXiv 2022

  78. [80]

    Zihao Zhou, Bin Hu, Chenyang Zhao, Pu Zhang, and Bin Liu.Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents. 2024. arXiv:2311 . 13373 [cs.AI].URL:https : //arxiv.org/abs/2311.13373

    Pith/arXiv arXiv 2024

  79. [81]

    2025.URL:https://openreview.net/forum? id=6y00rooi7i

    Runhan Yang, Jieao Shi, Mengqi SU, and Dongruo Zhou.Leveraging Imitation Learning and LLMs for Efficient Hierarchical Reinforcement Learning. 2025.URL:https://openreview.net/forum? id=6y00rooi7i

    work page 2025

  80. [82]

    Aoyu Pang, Maonan Wang, Man-On Pun, Chung Shue Chen, and Xi Xiong.iLLM-TSC: Integration rein- forcement learning and large language model for traffic signal control policy improvement. 2024. arXiv: 2407.06025 [cs.AI].URL:https://arxiv.org/abs/2407.06025

    Pith/arXiv arXiv 2024

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.