Pith. sign in

REVIEW 4 major objections 5 minor 64 references

Enhancing Decision-Making of Large Language Models via Actor-Critic

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a 7B/8B open-source LLM, paired with a critic built from its own GOOD/BAD token logits and a gradient-free KL-constrained policy update, outperforms GPT-4+ReAct on most evaluated multi-step decision tasks.

desk verdict A useful and mostly sound method for improving LLM agents, but the headline comparison against GPT-4+ReAct is confounded by unmatched fine-tuning. read the letter →

arxiv 2506.06376 v1 pith:BATOEOY5 submitted 2025-06-04 cs.CL cs.AI

Heng Dong , Kefei Duan , Chongjie Zhang This is my paper
classification cs.CLcs.AI
keywords largelanguagemodelsactor-critictokenlogitsgradient-freepolicyoptimizationsparse-rewarddecision-makingALFWorldBabyAI-TextWebShop
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a small open-source LLM can be turned into a stronger sequential decision-maker by pairing its own action prior with a critic built from its internal success/failure beliefs, then reweighting candidate actions without any gradient updates. The method, LAC, computes a Q-value for each candidate action as the log ratio of the model's probabilities for GOOD and BAD tokens after imagining a few future steps, and sets the new action distribution proportional to prior probability times exp(alpha times Q). On ALFWorld, BabyAI-Text, and WebShop, LAC with 7B/8B models outperforms existing LLM planning baselines and, in most settings, beats GPT-4+ReAct. If correct, this shows long-term action evaluation can be extracted cheaply from an LLM's token probabilities and folded back into its policy without training new policy parameters.

What carries the argument

The load-bearing object is the identity in Equation (3): $Q_{\mathrm{LLM}}(g,h_t,a_t,u_t)=\log[P(\mathrm{GOOD}\mid \cdot)/P(\mathrm{BAD}\mid \cdot)]$, connected to success probability by the logistic relation in Equation (1). The critic reads the probabilities the model assigns to paired positive/negative tokens after a forward model $f_{\mathrm{LLM}}$ rolls out a few future steps and a reflection judges the step. The policy improvement step in Equation (6), $\pi_{\mathrm{new}}(a_t\mid g,h_t)\propto \pi_{\mathrm{LLM}}(a_t\mid g,h_t)\exp(\alpha Q_{\mathrm{LLM}}(g,h_t,a_t,u_t))$, is the closed-form solution of the KL-constrained objective in Equation (4), with $\alpha$ controlling how far the new policy may deviate from the original one.

What would settle it

Compare LAC's Q-values against empirical success rates on a held-out set of ALFWorld or WebShop states: if actions rated higher by $Q_{\mathrm{LLM}}$ are not more likely to succeed, or if the GOOD/BAD probabilities are systematically overconfident, then the critic is not estimating success probability and the reported gains would not transfer to new tasks.

Watch

Extended reading notes

Core claim

The central discovery is that the log-odds of a model's own GOOD versus BAD token probabilities, taken after a short imagined rollout and a reflection, act as a usable Q-function for sparse-reward goal tasks, and that exponentiating these Q-values onto the prior action distribution yields a policy that outperforms both the prior policy and the critic alone.

Load-bearing premise

The whole method rests on the assumption that the log-ratio of the LLM's GOOD and BAD token probabilities, read after an imagined rollout, is a faithful estimate of how likely a candidate action is to lead to task success; if the model's self-evaluations are miscalibrated, the exponential reweighting could amplify its biases rather than correct them.

Editorial extensions

If this is right

  • Seven- and eight-billion-parameter open models can match or exceed the decision-making performance of GPT-4+ReAct on ALFWorld, BabyAI-Text, and WebShop, so strong task performance no longer requires a closed frontier model.
  • Because the policy update is gradient-free and only reweights sampled candidates, the method avoids expensive per-task fine-tuning of the policy; only the reflection generator is fine-tuned on a few trajectories.
  • The ablations support that all three components—future rollouts, reflections, and critic-guided reweighting—contribute, since removing any of them lowers success rates.
  • Statistical analyses link Q-values to task progression and show the improved policy follows the more confident of prior and critic, suggesting the mechanism is confidence-weighted agreement rather than blind score chasing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test, not run in the paper, is whether the same GOOD/BAD logit critic stays calibrated in longer-horizon or partially observable environments; the paper's Crafter result is preliminary and limited to one model.
  • If the critic's calibration holds, the same reweighting recipe could be applied to other sparse-reward domains without collecting reward labels, since the success signal is read from the model's own tokens.
  • The paper samples only top-5 candidate actions and acknowledges this may miss the true argmax; a distributional or temperature-widened sampling variant could make the method more robust in open-ended action spaces.
  • The confidence analysis suggests an explicit confidence-weighted arbitration rule could be studied as a standalone component, separating when to trust prior versus critic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces LAC (LLM-based Actor-Critic), a framework in which an LLM serves as both actor (generating candidate actions) and critic (evaluating actions via the log-odds of GOOD/BAD token probabilities after imagined rollouts), with policy improvement performed by a closed-form KL-constrained reweighting. Experiments on ALFWorld, BabyAI-Text, and WebShop claim state-of-the-art results using 7B/8B LLMs, including outperforming GPT-4+ReAct in most settings.

Significance. The proposed gradient-free actor-critic update is simple and computationally attractive, and the paper provides extensive ablations, hyperparameter analyses, and public code. If the issues below are remedied, the method could be a useful contribution to LLM-based decision-making. However, the headline result is confounded by task-specific LoRA fine-tuning, and the formal objective in Section 4.2 does not match the implemented algorithm, so the paper's central 'principled and scalable' claim needs revision.

major comments (4)
  1. [Abstract; Section 5.2; Table 11] The claim in the Abstract and Section 5.2 that 'LAC outperforms all other baselines in both ALFWorld and BabyAI-Text, and is even better than GPT-4+ReAct in most settings' is only supported when LAC includes LoRA fine-tuning on 18 task-specific trajectories (Appendix B.2), whereas GPT-4+ReAct is not fine-tuned. Table 11 shows that LAC without fine-tuning achieves ALFWorld success rates of 0.39 (CodeLlama-7B), 0.59 (Gemma-7B), 0.71 (Llama-3-8B), and 0.57 (Mistral-7B), which are below the GPT-4+ReAct level in the same benchmark (about 0.66 in Figure 8) for three of the four models. This unmatched comparison makes the headline overstatement; please qualify the claim as applying to the fine-tuned version of LAC or present a matched comparison in the main figures.
  2. [Section 4.2, Eqs. (4)-(6); Appendix B.1] The objective in Eq. (4) contains an expectation over u_i^t ~ f_LLM(g,h_t,a_i^t), but the closed-form solution in Eq. (5) uses exp(α Q_LLM(g,h_t,a_i^t,u_i^t)) for a specific sampled rollout u_i^t. Appendix B.1 drops the expectation over u and solves a different objective. The correct solution to Eq. (4) is π_new(a) ∝ π_LLM(a) exp(α E_{u~f(a)} Q(a,u)). Either Q should be redefined as the marginal value after integrating out rollout uncertainty, or the algorithm should be stated as a single-sample Monte Carlo approximation with a justification. As written, the derivation does not match the algorithm.
  3. [Section 5.2; Appendix C.3] The paper sets temperature to 0 and states that generation is deterministic (Section 5.2), yet Algorithm 1 samples n=5 candidate actions from π_LLM and Appendix C.3 says it 'samples the top 5 candidate actions'. Greedy decoding with temperature 0 yields one deterministic action sequence, so the mechanism for obtaining five distinct candidates is unspecified. Please clarify whether candidates are obtained via top-k token selection, beam search, or stochastic sampling. If stochastic sampling is used, the 'no error bar' justification is invalid and multiple seeds should be reported.
  4. [Section 4.1.2, Eq. (3); Section 5.5] The construction of Q as log(P(GOOD)/P(BAD)) in Eq. (3) is an assumption that LLM token probabilities correspond to the success probability in Eq. (1). The statistical analysis in Table 1 shows a correlation between Q-values and timesteps within successful/failed trajectories, but correlation does not establish calibration. Because Eq. (6) exponentiates Q, systematic miscalibration (e.g., overconfidence) is amplified rather than corrected. Please report calibration of the predicted success probabilities against actual outcomes, or at least show that the ranking-based performance is invariant to monotone transformations of Q.
minor comments (5)
  1. [Introduction; Related Work] The citation 'ichter et al., 2023' should be 'Ichter et al., 2023'.
  2. [Section 5.5, Table 2] The text says that when the prior and Q disagree, the improved policy 'tends to trust the more confident model', but in the row 'Prior agrees, Q disagrees' the prior's confidence (0.34) exceeds the improved policy's confidence (0.28), so the improved policy does not fully follow the more confident model. Please reconcile the interpretation with the numbers.
  3. [Section 4.1.2] The notation P(y_{w,l}|...) is unconventional; please write P(y_w|...) and P(y_l|...) explicitly.
  4. [Figure 8] The bars for the GPT-4 family and the fine-tuning baselines are crowded and their values are hard to read; consider adding a table with exact numbers.
  5. [Abstract] The phrase 'even outperforming baseline methods employing GPT-4 in complex tasks' is vague; specify which benchmark and which base model.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the policy-update formula is independently derived and the headline results are evaluated against external benchmarks; the actor/critic same-model concern is an assumption, not a by-construction equivalence.

full rationale

LAC's policy update (Eq. 6) is the closed-form solution to the KL-constrained objective (Eq. 4), and the derivation in Appendix B.1 is self-contained; it reduces the objective to a KL divergence against an explicitly constructed distribution and does not assume the conclusion. The Q-estimator (Eqs. 2-3) is a definitional log-odds transformation of GOOD/BAD token probabilities, not a fitted prediction of the benchmark outcomes; whether those log-odds faithfully estimate true success probability is a calibration assumption that is empirically testable, not a by-construction equivalence. The critic and actor are the same base model, but the critic conditions on imagined rollouts and reflections, so the final action is not identical to the actor's prior by construction. The appendix's Table 11 shows that fine-tuning contributes substantially to the margin over GPT-4+ReAct, which is a legitimate experimental-design concern about matched comparisons, but it is not a circularity in the derivation chain. The only self-citation (Dong et al., 2022) appears in related work and is not load-bearing. Benchmark results on ALFWorld, BabyAI-Text, WebShop, and Crafter are external, so the central claims are not self-confirming. Overall, no significant circularity is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method relies on the unvalidated assumption that LLM token logits are calibrated success probabilities, and on the accuracy of imagined rollouts. The policy update formula is standard math. The only tuned numerical parameter is alpha, with hand-chosen hyperparameters for candidate count and rollout depth.

free parameters (4)
  • alpha (KL penalty coefficient) = ALFWorld: 1; BabyAI-Text: per-model grid search (e.g., 2 for Gemma-7B, 10 for Mistral-7B); WebShop: grid search over…
    Controls trade-off between LLM prior and critic. Selected via grid search on validation tasks (Appendix A.11).
  • candidate action count n = 5
    Chosen by hand as trade-off between coverage and compute.
  • maximum rollout steps = 4
    Chosen by hand; the model predicts until GOOD/BAD or 4 steps.
  • fine-tuning trajectory count = 18 per benchmark
    Composed of 2 successful and 1 failed trajectory per task type; choice is ad hoc but motivated by small-data regime.
assumptions (4)
  • domain assumption The LLM's token probabilities for GOOD/BAD tokens correspond to the true success/failure probabilities of the task.
    Invoked in Sec 4.1.2 (Eq. 2-3) without external calibration; the entire critic rests on this.
  • ad hoc to paper The logistic link Q = log(P(yw)/P(yl)) is an appropriate transformation of success probability.
    Equation (1) is introduced as 'simple and effective' with no derivation; it's a chosen modeling assumption.
  • domain assumption The forward model fLLM produces plausible future trajectories for evaluating actions.
    Used in rollouts in Sec 4.1; if rollouts are inaccurate, Q-values mislead.
  • standard math The KL-constrained optimization has the closed-form solution pi proportional to pi_LLM exp(alpha Q).
    Standard result (see Rafailov et al. 2024), derived in Appendix B.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Decision-Making of Large Language Models via Actor-Critic." pith.science (2026). https://pith.science/paper/BATOEOY5

@misc{pith2026250606376,
  author       = {Pith},
  title        = {Pith review of: Enhancing Decision-Making of Large Language Models via Actor-Critic},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BATOEOY5}},
  note         = {Machine review of arXiv:2506.06376}
}
read the original abstract

Large Language Models (LLMs) have achieved remarkable advancements in natural language processing tasks, yet they encounter challenges in complex decision-making scenarios that require long-term reasoning and alignment with high-level objectives. Existing methods either rely on short-term auto-regressive action generation or face limitations in accurately simulating rollouts and assessing outcomes, leading to sub-optimal decisions. This paper introduces a novel LLM-based Actor-Critic framework, termed LAC, that effectively improves LLM policies with long-term action evaluations in a principled and scalable way. Our approach addresses two key challenges: (1) extracting robust action evaluations by computing Q-values via token logits associated with positive/negative outcomes, enhanced by future trajectory rollouts and reasoning; and (2) enabling efficient policy improvement through a gradient-free mechanism. Experiments across diverse environments -- including high-level decision-making (ALFWorld), low-level action spaces (BabyAI-Text), and large action spaces (WebShop) -- demonstrate the framework's generality and superiority over state-of-the-art methods. Notably, our approach achieves competitive performance using 7B/8B parameter LLMs, even outperforming baseline methods employing GPT-4 in complex tasks. These results underscore the potential of integrating structured policy optimization with LLMs' intrinsic knowledge to advance decision-making capabilities in multi-step environments.

Figures

Figures reproduced from arXiv: 2506.06376 by the authors.

Figure 1
Figure 1. Framework of our LAC. At each time step, LAC optimizes the policy via two steps: (1) the critic QLLM evaluates each candidate action sampled from the policy πLLM; (2) the policy πLLM is optimized according to the action evaluations using a gradient-free method. based Actor-Critic (LAC) approach that leverages action evaluations to optimize the LLM’s prior policy under con￾straints, facilitating the effective integra… view at source ↗
Figure 2
Figure 2. Performance of our LAC compared with various baselines in benchmarks ALFWorld and BabyAI-Text. probabilities of LLMs for those special tokens reflect LLMs’ internal belief in success/failure after taking action a i t . We let the generated probabilities of “GOOD” and “BAD” rep￾resent P(yw|g, ht, ai t ) and P(yl |g, ht, ai t ) respectively. Fi￾nally, using Equation (2), we can calculate QLLM(g, ht, ai t ) for action … view at source ↗
Figure 3
Figure 3. Performance of LAC in benchmark WebShop Peng et al., 2019; Jain et al., 2013; Peters & Schaal, 2007), we can show that the optimal solution to the KL-constrained maximization objective in Equation (4) takes the following form: πnew(a i t |g, ht) = 1 Z(g, ht) πLLM(a i t |g, ht) · exp (αQLLM(g, ht, ai t , ui t )), (5) where Z(g, ht) = P a i t πLLM(a i t |g, ht) exp(αQLLM(g, ht, a i t , ui t )) is the partition functio… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Ablation studies in benchmarks ALFWorld and BabyAI-Text. the highest predicted cumulative rewards. (4) RAFA (Liu et al., 2023) evaluates each action candidate by tree-search and selects the action that may complete the most sub-goals. (5) LATS (Zhou et al., 2024a) comb…
Figure 5
Figure 5. Figure 5: Computational cost analysis of LAC and baselines. the robustness of our method in handling more complex and open-ended action spaces. LAC’s superior performance stems from its effective inte￾gration of the decision-making information contained in the LLM prior with the…
Figure 6
Figure 6. Figure 6: Performance of our LAC compared with various baselines in all tasks from BabyAI-Text. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Performance of LAC when using different definition of critic QLLM (1) prior policy w/ reflection: We remove all components of LAC, leaving the policy and the reflection unchanged. Specifically, at each step, after observing the action results, the reflection first gene…
Figure 8
Figure 8. Figure 8: Performance of our LAC and LAC’s variants compared with various baselines in benchmark ALFWorld. In this subsection, we compare LAC with more baselines including some traditional RL methods implemented using LLMs on ALFWorld (Shridhar et al., 2021). The comparison is s…
Figure 9
Figure 9. Figure 9: Computational cost analysis of our LAC compared with various baselines in benchmarks ALFWorld. Though LAC may have a higher computational cost per step due to the extra inference procedure of critics and the forward model, the total cost of LAC is still lower than most…
Figure 10
Figure 10. Figure 10: An illustrative explanation of our method LAC in ALFWorld. The histogram on the right shows the action probabilities of different methods. While LLM’s prior policy (πLLM) and critic, as well as LAC w/o reflection, make mistakes at different time steps, LAC (ours) can …
Figure 11
Figure 11. Figure 11: An illustrative explanation of our method LAC in BabyAI-Text. The histogram on the right shows the action probabilities of different methods. While LLM’s prior policy (πLLM) and critic, as well as LAC w/o reflection, make mistakes at different time steps, LAC (ours) c…
Figure 12
Figure 12. Figure 12: Analysis regarding the fine-tuning process of our LAC. (a) Influence of the fine-tuning data size. Larger data sizes (27, 36 trajectories) generally bring higher performance, but small data sizes (18 and even 9 trajectories) are already enough for our method to achiev…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 33 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    G., Sutton, R

    Barto, A. G., Sutton, R. S., and Watkins, C. Learning and sequential decision making, volume 89. University of Massachusetts Amherst, MA, 1989

    work page 1989

  3. [3]

    L., and Singh, S

    Brooks, E., Walls, L., Lewis, R. L., and Singh, S. Large language models can implement policy iteration. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  4. [4]

    The computational complexity of propositional strips planning

    Bylander, T. The computational complexity of propositional strips planning. Artificial Intelligence, 69 0 (1-2): 0 165--204, 1994

    work page 1994

  5. [5]

    Grounding large language models in interactive environments with online reinforcement learning

    Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in interactive environments with online reinforcement learning. In International Conference on Machine Learning, pp.\ 3676--3713. PMLR, 2023 a

    work page 2023

  6. [6]

    Grounding large language models in interactive environments with online reinforcement learning, 2023 b

    Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in interactive environments with online reinforcement learning, 2023 b

    work page 2023

  7. [7]

    Decision transformer: Reinforcement learning via sequence modeling

    Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 0 15084--15097, 2021

    2021

  8. [8]

    H., and Bengio, Y

    Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:1810.08272, 2018

    arXiv 2018

Show all 64 references
  1. [9]

    Efficient selectivity and backup operators in monte-carlo tree search

    Coulom, R. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pp.\ 72--83. Springer, 2006

    2006

  2. [10]

    G., Xia, T., Mao, H., et al

    Cuadron, A., Li, D., Ma, W., Wang, X., Wang, Y., Zhuang, S., Liu, S., Schroeder, L. G., Xia, T., Mao, H., et al. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.08235, 2025

    2025 arXiv

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    2025 arXiv

  4. [12]

    Task and motion planning with large language models for object rearrangement

    Ding, Y., Zhang, X., Paxton, C., and Zhang, S. Task and motion planning with large language models for object rearrangement. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.\ 2086--2092. IEEE, 2023

    2023

  5. [13]

    Low-rank modular reinforcement learning via muscle synergy

    Dong, H., Wang, T., Liu, J., and Zhang, C. Low-rank modular reinforcement learning via muscle synergy. Advances in Neural Information Processing Systems, 35: 0 19861--19873, 2022

    2022

  6. [14]

    Preact: Predicting future in react enhances agent's planning ability

    Fu, D., Huang, J., Lu, S., Dong, G., Wang, Y., He, K., and Xu, W. Preact: Predicting future in react enhances agent's planning ability. arXiv preprint arXiv:2402.11534, 2024

    2024 arXiv

  7. [15]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.\ 1587--1596. PMLR, 2018

    2018

  8. [16]

    Aligning language models with preferences through f -divergence minimization

    Go, D., Korbak, T., Kruszewski, G., Rozen, J., Ryu, N., and Dymetman, M. Aligning language models with preferences through f -divergence minimization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th Internatio...

    2023

  9. [17]

    Benchmarking the spectrum of agent capabilities

    Hafner, D. Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=1W0z96MFEoH

    2022

  10. [18]

    Reasoning with language model is planning with world model

    Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., and Hu, Z. Reasoning with language model is planning with world model. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 8154--8173,...

    2023 doi

  11. [19]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021

    2021

  12. [20]

    and Chang, K

    Huang, J. and Chang, K. C.-C. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403, 2022

    2022 arXiv

  13. [21]

    One policy to control them all: Shared modular policies for agent-agnostic control

    Huang, W., Mordatch, I., and Pathak, D. One policy to control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning, pp.\ 4455--4464. PMLR, 2020

    2020

  14. [22]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents

    Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pp.\ 9118--9147. PMLR, 2022 a

    2022

  15. [23]

    Inner monologue: Embodied reasoning through planning with language models

    Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022 b

    2022 arXiv

  16. [24]

    ichter, b., Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., Kalashnikov, D., Levine, S., Lu, Y., Parada, C., Rao, K., Sermanet, P., Toshev, A. T., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Yan, M., Brown, N., Ahn...

    2023

  17. [25]

    Learning trajectory preferences for manipulators via iterative improvement

    Jain, A., Wojcik, B., Joachims, T., and Saxena, A. Learning trajectory preferences for manipulators via iterative improvement. Advances in neural information processing systems, 26, 2013

    2013

  18. [26]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  19. [27]

    A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods

    Jin, H., Zhang, Y., Meng, D., Wang, J., and Tan, J. A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods. arXiv preprint arXiv:2403.02901, 2024

    2024

  20. [28]

    Jordan, M. I. et al. Why the logistic function? a tutorial discussion on probabilities and neural networks, 1995

    1995

  21. [29]

    and Szepesv \'a ri, C

    Kocsis, L. and Szepesv \'a ri, C. Bandit based monte-carlo planning. In European conference on machine learning, pp.\ 282--293. Springer, 2006

    2006

  22. [30]

    Y., McAleer, S., Fried, D., and Salakhutdinov, R

    Koh, J. Y., McAleer, S., Fried, D., and Salakhutdinov, R. Tree search for language model agents. arXiv preprint arXiv:2407.01476, 2024

    2024

  23. [31]

    S., Reid, M., Matsuo, Y., and Iwasawa, Y

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 0 22199--22213, 2022

    2022

  24. [32]

    X., Nie, J.-Y., and Wen, J.-R

    Li, J., Tang, T., Zhao, W. X., Nie, J.-Y., and Wen, J.-R. Pre-trained language models for text generation: A survey. ACM Comput. Surv., 56 0 (9), April 2024 a . ISSN 0360-0300. doi:10.1145/3649449. URL https://doi.org/10.1145/3649449

    2024 doi

  25. [33]

    Unigen: A unified generative framework for retrieval and question answering with large language models

    Li, X., Zhou, Y., and Dou, Z. Unigen: A unified generative framework for retrieval and question answering with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 8688--8696, 2024 b

    2024

  26. [34]

    Littman, M. L. Algorithms for sequential decision-making. Brown University, 1996

    1996

  27. [35]

    Reason for future, act for now: A principled framework for autonomous llm agents with provable sample efficiency

    Liu, Z., Hu, H., Zhang, S., Guo, H., Ke, S., Liu, B., and Wang, Z. Reason for future, act for now: A principled framework for autonomous llm agents with provable sample efficiency. arXiv preprint arXiv:2309.17382, 2023

    2023 arXiv

  28. [36]

    McCarthy, J. et al. Situations, actions, and causal laws. Comtex Scientific, 1963

    1963

  29. [37]

    Meta llama 3

    Meta. Meta llama 3. https://llama.meta.com/llama3/, 2024 a

    2024

  30. [38]

    Meta llama 3.1

    Meta. Meta llama 3.1. https://ai.meta.com/blog/meta-llama-3-1/, 2024 b

    2024

  31. [39]

    Augmented language models: a survey

    Mialon, G., Dessi, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Roziere, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., Grave, E., LeCun, Y., and Scialom, T. Augmented language models: a survey. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. ...

    2023

  32. [40]

    N., Parker-Holder, J., and Rockt \"a schel, T

    Paglieri, D., Cupia , B., Coward, S., Piterbarg, U., Wo czyk, M., Khan, A., Pignatelli, E., Kuci \'n ski, ., Pinto, L., Fergus, R., Foerster, J. N., Parker-Holder, J., and Rockt \"a schel, T. Benchmarking agentic llm and vlm reasoning on games. arXiv preprint arXiv:2411.13543, 2024

    2024 arXiv

  33. [41]

    B., Kumar, A., Zhang, G., and Levine, S

    Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019

    1910 arXiv

  34. [42]

    and Schaal, S

    Peters, J. and Schaal, S. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp.\ 745--750, 2007

    2007

  35. [43]

    Why think step by step? reasoning emerges from the locality of experience

    Prystawski, B., Li, M., and Goodman, N. Why think step by step? reasoning emerges from the locality of experience. Advances in Neural Information Processing Systems, 36, 2024

    2024

  36. [44]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  37. [45]

    E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al

    Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  38. [46]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  39. [47]

    Reflexion: Language agents with verbal reinforcement learning

    Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  40. [48]

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks

    Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10740...

    2020

  41. [49]

    Alfworld: Aligning text and embodied environments for interactive learning, 2021

    Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. Alfworld: Aligning text and embodied environments for interactive learning, 2021

    2021

  42. [50]

    Progprompt: Generating situated robot task plans using large language models

    Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 11523--11530. ...

    2023

  43. [51]

    D., and Toshev, A

    Szot, A., Schwarzer, M., Agrawal, H., Mazoure, B., Metcalf, R., Talbott, W., Mackraz, N., Hjelm, R. D., and Toshev, A. T. Large language models as generalizable policies for embodied tasks. In The Twelfth International Conference on Learning Representations, 2023

    2023

  44. [52]

    True knowledge comes from practice: Aligning large language models with embodied environments via reinforcement learning

    Tan, W., Zhang, W., Liu, S., Zheng, L., Wang, X., and An, B. True knowledge comes from practice: Aligning large language models with embodied environments via reinforcement learning. In ICLR, 2024

    2024

  45. [53]

    S., Love, J., et al

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi \`e re, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  46. [54]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  47. [55]

    Watkins, C. J. and Dayan, P. Q-learning. Machine learning, 8: 0 279--292, 1992

    1992

  48. [56]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  49. [57]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Yao, S., Chen, H., Yang, J., and Narasimhan, K. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35: 0 20744--20757, 2022

    2022

  50. [58]

    React: Synergizing reasoning and acting in language models

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  51. [59]

    C., Liu, Z., Feng, Y., Xue, L., Rithesh, R., Chen, Z., Zhang, J., Arpit, D., et al

    Yao, W., Heinecke, S., Niebles, J. C., Liu, Z., Feng, Y., Xue, L., Rithesh, R., Chen, Z., Zhang, J., Arpit, D., et al. Retroformer: Retrospective large language agents with policy gradient optimization. In The Twelfth International Conference on Learning Representations

  52. [60]

    Controlling large language model-based agents for large-scale decision-making: An actor-critic approach

    Zhang, B., Mao, H., Ruan, J., Wen, Y., Li, Y., Zhang, S., Xu, Z., Li, D., Li, Z., Zhao, R., et al. Controlling large language model-based agents for large-scale decision-making: An actor-critic approach. arXiv preprint arXiv:2311.13884, 2023

    2023 arXiv

  53. [61]

    Large language models are semi-parametric reinforcement learning agents

    Zhang, D., Chen, L., Zhang, S., Xu, H., Zhao, Z., and Yu, K. Large language models are semi-parametric reinforcement learning agents. Advances in Neural Information Processing Systems, 36, 2024

    2024

  54. [62]

    Language agent tree search unifies reasoning, acting, and planning in language models

    Zhou, A., Yan, K., Shlapentokh-Rothman, M., Wang, H., and Wang, Y.-X. Language agent tree search unifies reasoning, acting, and planning in language models. In Forty-first International Conference on Machine Learning, 2024 a

    2024

  55. [63]

    Archer: Training language model agents via hierarchical multi-turn rl

    Zhou, Y., Zanette, A., Pan, J., Levine, S., and Kumar, A. Archer: Training language model agents via hierarchical multi-turn rl. In Forty-first International Conference on Machine Learning, 2024 b

    2024

  56. [64]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.