Pith. sign in

REVIEW 4 major objections 6 minor 27 references

AgentSLABench argues that agent evaluation must measure resource consumption alongside correctness, and reports that task-specialized agents dominate general baselines when budgets are enforced.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

AgentSLABench adds resource budgets and cost/latency/memory profiles to AI agent evaluation, but its headline results rest on tiny samples and an internal inconsistency.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The framework is a useful starting point for resource-aware agent evaluation, but the headline empirical claims are not supported as reported; the infrastructure deserves refereeing, the numbers do not. the 4 major comments →

arxiv 2608.00805 v1 pith:7IBOVWD3 submitted 2026-08-01 cs.AI

AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints

Meher Bhaskar Madiraju , Meher Sai Preetam Madiraju This is my paper
classification cs.AI
keywords agent evaluationresource-aware profilingefficiency-adjusted success rateLLM agentsbenchmarkingproduction constraintsDocker isolation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that accuracy-only benchmarks misjudge AI agents for production, because they ignore latency, cost, memory, and network constraints. It presents AgentSLABench, a framework that profiles each agent episode the way a systems profiler measures code, with correctness as one dimension alongside resource use under declared budgets. The empirical claim is that task-specialized agents succeed within budgets where general reasoning agents fail completely on most domain tasks. The paper introduces EASR to make "success at any cost" score as failure. If the results hold, they imply accuracy benchmarks should be replaced by resource-aware profiles for deployment decisions.

Core claim

The paper claims that an agent's value in production is not its accuracy but its correctness per unit of declared resource budget. It builds a profiling-style benchmark, AgentSLABench, treating each agent episode like a function call in a systems profiler, recording success, latency, cost, peak memory, CPU time, network calls, and safety violations under Docker-enforced budgets. On five core tasks, the paper reports that task-specialist agents achieve 100% success on fact_qa, web_shopping, and travel planning and 66.7–83.3% on retail substitution and code generation, while general baselines (ReAct, PlanAndSolve, Reflexion, CoT, Random) score 0% on four of five domain tasks. It proposes EASR,

What carries the argument

The central object is the per-episode profile plus the EASR formula: EASR = success × min(1, budget_lat/actual_lat) × min(1, budget_cost/actual_cost) × min(1, budget_mem/actual_mem). The profile records six dimensions per episode (correctness, latency, cost, compute, network, safety) inside Docker-enforced CPU, memory, time, and network budgets; EASR converts that profile into a single deployment-gate score that zeroes out any episode exceeding a budget.

Load-bearing premise

An LLM judge's verdicts are treated as ground truth for task success, and the reported human agreement is simulated rather than measured against real people.

What would settle it

Take 50 episodes the judge marked successful and 50 it marked failed, have independent human experts label them, and compare; if agreement falls well below the claimed 100%, the reported success rates, including the 0% vs 66–100% gap, are not established.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Accuracy-only benchmarks can label an agent 90% successful when it violates every production SLA; EASR would score it 0 for over-budget episodes.
  • Teams can use per-episode profiles as a pre-deployment gate: an agent that passes accuracy but exceeds latency, cost, or memory budgets is rejected or re-tuned.
  • Specialized agents within budgets succeed where general reasoning agents get 0% on four of five domain tasks, implying domain specialization is necessary for production agents.
  • The per-step resource breakdown (e.g., the travel planner's peak at constraint validation) points to concrete optimizations such as caching hotel queries.
  • EASR creates a Pareto frontier for success versus efficiency, enabling multi-objective trade-offs rather than single-score leaderboards.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test the LLM judge against real human labels; if a human-annotated sample were scored, the headline gaps (0% vs 66–100%) could shrink or invert, since all success rates flow through that judge.
  • The EASR multiplicative caps could be extended to network and safety dimensions, and to P99 latency instead of median, which would tighten the deployment-gate interpretation.
  • A cheap testable extension is to rerun the protocol with a second judge model and with N ≥ 10 seeds; the paper's own bootstrap CIs show the 66.7% and 83.3% results are statistically indistinguishable from 0% or 100% at N = 3.
  • The specialization-gap claim could be stress-tested by giving general baselines the same domain schemas and tools the specialized agents use, isolating whether the gap is agent architecture or simply access to domain logic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes AgentSLABench, a Docker-based framework for profiling LLM agents under declared resource budgets, measuring correctness alongside latency, cost, memory, CPU time, network usage, and safety violations. It contributes a profile-oriented output format and the Efficiency-Adjusted Success Rate (EASR) metric. The authors profile five general-purpose baselines (ReAct, PlanAndSolve, Reflexion, CoT, Random) and four task-specialized agents across five core and eleven extended tasks. The headline finding is that specialized agents achieve 100% success on 3/5 core tasks (fact_qa, web_shopping, travel_planning) and 66.7–83.3% on the remaining two, while general baselines fail entirely (0%) on 4/5 domain tasks. The paper also reports resource profiles, per-step breakdowns, failure mode analyses, bootstrap confidence intervals, and significance tests.

Significance. If the empirical claims were supported, AgentSLABench would be a useful step toward production-aware agent evaluation, and the profiling metaphor (extending systems profilers to agents) is conceptually appealing. The released infrastructure, sealed test sets, versioned containers, and per-episode JSONL output are strengths that would facilitate reproducibility. However, the headline empirical contribution is not currently established: success rates are assigned by an LLM judge whose agreement with humans is described as 'simulated' rather than measured; the number of seeds is only three, yielding confidence intervals of [0,1] for the key comparisons; and the specialized agents are hand-coded by the same authors who defined the tasks, making the observed specialization gap partly by construction. These are load-bearing issues for the central claim, not presentation concerns.

major comments (4)
  1. [IV.G, Tables III–VIII] The judge calibration section states: 'Human spot-check on 20 samples shows 100% simulated agreement.' This does not establish agreement with real human labels; it only reports a simulation. All success rates in the paper—including the central 0% versus 66.7–100% comparison—are produced by the Nemotron Ultra judge, which applies task-specific metrics such as price_optimality, itinerary_quality, and test_pass_rate. Without a real human agreement study (e.g., per-task Cohen's kappa on a sizable sample), a judge biased against general agents could invert the headline result. This is a load-bearing validity threat and must be fixed with genuine human validation, not a simulated spot-check.
  2. [Abstract, Conclusion vs. Tables III/IV/V/VIII] The abstract and conclusion claim 100% success on travel_planning, but Tables III, IV, V, and VIII all report 83.3% for TravelPlanAgent. This internal inconsistency appears at the most visible level of the paper and directly affects the headline claim. The authors must correct the numbers and ensure that every table, abstract, and conclusion are consistent.
  3. [V.A, Table VIII, Table IX] Bootstrap 95% CIs with N=3 seeds are [0.0, 1.0] for retail, code_gen, and travel. Despite this, Table IX marks web_shop and travel as 'significant' at p<0.05. With three episodes per condition, the effective sample size is far too small to support either the confidence intervals or the significance claims, as the paper itself acknowledges in the limitation note. The abstract's strong assertions (e.g., 'general baselines fail entirely') are not supported by the reported statistics. At minimum, the authors need N≥10 seeds, a pre-registered analysis, and a reporting format that does not present [0,1] intervals as evidence of a stable success rate.
  4. [III.D, I (Contribution 5)] The task-specialized agents (RetailAgent, WebShoppingAgent, TravelPlanningAgent, CodeGenAgent) are hand-coded by the same authors who designed the tasks, using domain-specific heuristics and rules. Evaluating these agents on the authors' own tasks makes the observed 'specialization gap' partly by construction: the agents may simply encode the task solutions. This circularity is not addressed in the paper. The framework itself is still useful, but the empirical finding that specialized agents outperform general baselines needs validation on tasks and agents developed independently, or at least a held-out task split that was not used during agent design.
minor comments (6)
  1. [Throughout] The name is used inconsistently: 'AgentSLABench' in the title and abstract, 'AGENTSLABENCH' in many section headings. Please standardize.
  2. [Table II] Profile dimensions include metrics such as acceptance_rate, test_pass_rate, and margin_retention_pct but these are not defined in the text. Provide precise definitions so the judge's scoring criteria are reproducible.
  3. [IV.G] The phrase '100% simulated agreement' is ambiguous. If it means self-consistency of the judge or agreement with a simulated label set, state that explicitly. If it is intended to be a human agreement study, it is not one.
  4. [VII] The reproducibility section mentions 'human_judge_spotcheck_*.jsonl' artifacts. If these contain only simulated labels, please label them as such in the artifact directory; otherwise, provide evidence of real human annotation.
  5. [V.A / Table VIII] The column 'Avg Rew' is not defined. Is this the task reward, or a judge-assigned reward? Clarify the scale and how it relates to success.
  6. [Section I, Eq. (1)] The statement that EASR 'reveals' that high accuracy at unbounded cost is not production-viable is a definitional consequence of Equation (1), not an empirical discovery. Consider rephrasing to avoid implying that the metric itself is a finding.

Circularity Check

1 steps flagged

Minor definitional circularity in the EASR framing; the central success-rate comparison is empirical and not circular, though the LLM-judge validation is weak.

specific steps
  1. self definitional [Abstract; Eq. (1); Section VI.B]
    "Crucially, we report the Efficiency-Adjusted Success Rate (EASR) — success weighted by resource consumption relative to declared budgets — revealing that high accuracy at unbounded cost is not production-viable."

    EASR is defined in Eq. (1) as success multiplied by min(1, budget_lat/actual_lat) × min(1, budget_cost/actual_cost) × min(1, budget_mem/actual_mem). The property that over-budget success is penalized is therefore built into the metric by definition, not discovered from measurements. Section VI.B states 'EASR formalizes this: it is 0 for any over-budget episode, regardless of correctness,' which is a restatement (and actually a strengthening) of the definitional choice. The paper presents this as a finding ('revealing that high accuracy at unbounded cost is not production-viable'), but no data are needed to reach that conclusion once EASR is defined this way.

full rationale

Most of AgentSLABench's derivation chain is self-contained and not circular. The headline success rates in Tables III-VIII are measured outcomes, not fitted or definitionally forced: the specialized agents are hand-coded for the tasks, but the general baselines (ReAct, PlanAndSolve, Reflexion, CoT, Random) are independent, off-the-shelf methods that could in principle have succeeded; no parameter is fit to success labels, and no self-citation carries a load-bearing premise. Resource dimensions (latency, cost, memory) are obtained from container/API measurements independent of the correctness judge. The one definitional circularity is the EASR framing: Eq. (1) encodes the budget penalty, so the Abstract's claim that EASR 'reveals' the non-viability of high accuracy at unbounded cost is a restatement of the metric's construction, not an empirical result. This is a minor circularity because EASR is a secondary contribution; the central agent-comparison claim does not reduce to EASR. Separately, the paper's own text exposes non-circular validity threats: Section IV.G reports only '100% simulated agreement' for the human spot-check rather than real human labels, and the abstract's 100% claim for travel_planning conflicts with the 83.3% shown in Tables III, IV, V, and VIII. These are correctness/validity concerns, not circularity, and they lower confidence in the numerical comparisons without making the derivation circular.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The main free parameters are the declared per-task budgets and the hand-coded specialized agents; both determine EASR and success rates. The correctness labels rely on a non-validated LLM judge, and the empirical comparisons assume N=3 episodes per condition is informative.

free parameters (3)
  • Declared resource budgets for 16 tasks = e.g., fact_qa: 2 CPU/4GB/120s/net on; retail: 4 CPU/8GB/180s/net on; travel: 4 CPU/8GB/600s/net on
    Chosen heuristically from pilot runs (Section VI.E); they set the EASR threshold and determine which successes count as production-viable.
  • Specialized agent designs and heuristics = RetailAgent, WebShopAgent, TravelPlanAgent, CodeGenAgent; no parameter values disclosed
    Hand-authored domain logic; the agents' success is largely a function of these designs, which are not independently derived.
  • Judge configuration (Nemotron Ultra, response_format, metrics list) = not specified in detail
    Determines success/failure labels for every episode; no independent human labels are shown.
axioms (4)
  • domain assumption Correctness labels from the Nemotron Ultra judge match true task outcomes
    All success rates depend on the LLM judge; Section IV.G supports it only with a 20-sample simulated human agreement check.
  • domain assumption Three seeds (episodes) per agent-task pair are enough to estimate success rates
    Used in all tables; Section V.A admits bootstrap CIs are extremely wide with N=3.
  • domain assumption Docker container limit measurements accurately reflect agent resource consumption
    Section III.B/C treat container stats as ground truth; no comparison to host-level measurement is reported.
  • domain assumption General baselines are faithful implementations of ReAct, CoT, Reflexion, and PlanAndSolve
    Their 0% results are attributed to method shortcomings, but baselines were not given task schemas, so harness mismatch could explain failures (Section III.D).

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints." pith.science (2026). https://pith.science/paper/7IBOVWD3

@misc{pith2026260800805,
  author       = {Pith},
  title        = {Pith review of: AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7IBOVWD3}},
  note         = {Machine review of arXiv:2608.00805}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We present AgentSLABench, a resource-aware evaluation framework for autonomous AI agents that measures correctness alongside latency, cost, compute, memory, and network usage under declared resource budgets. Unlike standard benchmarks that report only accuracy, AgentSLABench produces a multi-dimensional profile per agent per task - the same way systems profilers (perf, pprof, cProfile) measure resource consumption of code, but extended with task correctness as a first-class dimension. AgentSLABench provides 16 task environments across 6 categories (5 core: multi-hop QA, retail substitution, code generation, web shopping, travel planning; 11 extended) with isolated Docker containers, declared CPU/memory/time/network budgets, sealed test sets with SHA256 hashes, and a standardized profiling protocol. We profile 5 general-purpose baseline agents (ReAct, PlanAndSolve, Reflexion, CoT, Random) plus 4 task-specialized agents, finding that specialized agents achieve 100% success on 3/5 core tasks (fact_qa, web_shopping, travel_planning) and 66.7-83.3% on retail and code_gen, while general baselines fail entirely on 4/5 domain tasks. Crucially, we report the Efficiency-Adjusted Success Rate (EASR) - success weighted by resource consumption relative to declared budgets - revealing that high accuracy at unbounded cost is not production-viable. We release the full infrastructure, sealed test sets, and profiling results to enable reproducible, resource-aware agent evaluation.

Figures

Figures reproduced from arXiv: 2608.00805 by Meher Bhaskar Madiraju, Meher Sai Preetam Madiraju.

Figure 1
Figure 1. Figure 1: Standard profiler vs. AgentSLABench. (Left) Traditional profilers measure resource usage of code functions without task correctness. (Right) AGENTSLABENCH extends this to autonomous agents, adding task correctness as a first-class dimension alongside latency, cost, compute, memory, and network. budgets — agents are evaluated within these envelopes via Docker-enforced limits. 2) Efficiency-Adjusted Success … view at source ↗
Figure 2
Figure 2. Figure 2: Per-step resource breakdown (TravelPlanningA￾gent, median over 3 seeds). Latency and memory grow with planning depth; peak occurs at constraint validation step. code_gen and retail use 0-1 calls (local reasoning). C. Per-Step Resource Breakdown To understand resource dynamics within episodes, we analyze per-step profiles for the travel task (most resource￾intensive) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 18 canonical work pages · 2 internal anchors

  1. [1]

    WebShop: Towards scalable real-world web interaction with grounded language agents,

    S. Yao, H. Chen, J. Yang, and K. Narasimhan, “WebShop: Towards scalable real-world web interaction with grounded language agents,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 20744–20757, 2022

    work page 2022

  2. [2]

    ALFWorld: Aligning text and embodied environ- ments for interactive learning,

    M. Shridhar, X. Yuan, M.-A. Côté, Y . Bisk, A. Trischler, and M. Hausknecht, “ALFWorld: Aligning text and embodied environ- ments for interactive learning,” inInternational Conference on Learn- ing Representations (ICLR), 2021

    work page 2021

  3. [3]

    SWE-bench: Can language models resolve real- world GitHub issues?,

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real- world GitHub issues?,” inInternational Conference on Learning Representations (ICLR), 2024

    work page 2024

  4. [4]

    GAIA: A benchmark for general AI assistants,

    G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y . LeCun, and G. Synnaeve, “GAIA: A benchmark for general AI assistants,”arXiv preprint arXiv:2311.12983, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    AgentBench: Evaluating LLMs as agents,

    X. Liu, H. Yu, H. Zhang, Y . Xu, X. Lei, H. Lai, Y . Gu, H. Ding, K. Zhao, S. Yao,et al., “AgentBench: Evaluating LLMs as agents,” in International Conference on Learning Representations (ICLR), 2024

    work page 2024

  6. [6]

    ToolLLM: Facilitating large language models to master 16000+ real-world APIs,

    Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, B. Qian,et al., “ToolLLM: Facilitating large language models to master 16000+ real-world APIs,”arXiv preprint arXiv:2307.16789, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    LangSmith: Production observability for LLM applications

    LangChain Team, “LangSmith: Production observability for LLM applications.”https://blog.langchain.dev/ langsmith-production-observability/, 2024

    work page 2024

  8. [8]

    Ultrafast Excitation Exchange in a Maxwell-Fish-Eye Lens

    J. Savelka and K. D. Ashley, “AgentEval: A flexible framework to evaluate AI agents upon user query,”arXiv preprint arXiv:2311.18750, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    AutoGen: Enabling next- gen LLM applications via multi-agent conversation,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, Z. Erkang, L. Li, K. Yang, L. Tran-Thanh, and C. Wang, “AutoGen: Enabling next- gen LLM applications via multi-agent conversation,”arXiv preprint arXiv:2308.08155, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    FrugalGPT: How to use large language models while reducing cost and improving performance,

    L. Chen, M. Zaharia, and J. Zou, “FrugalGPT: How to use large language models while reducing cost and improving performance,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023

    work page 2023

  11. [11]

    Green AI,

    R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green AI,” Communications of the ACM, vol. 63, no. 12, pp. 54–63, 2020

    2020

  12. [12]

    MLPerf training benchmark,

    P. Mattson, C. Cheng, C. Coleman, G. Diamos, P. Micikevicius, D. Patterson, H. Tang, G.-Y . Wei, P. Bailis, V . Bittorf,et al., “MLPerf training benchmark,” inProceedings of Machine Learning and Systems (MLSys), vol. 2, pp. 336–349, 2020

    work page 2020

  13. [13]

    Towards the systematic reporting of the energy and carbon footprints of machine learning,

    P. Henderson, J. Hu, J. Romoff, E. Brunskill, D. Jurafsky, and J. Pineau, “Towards the systematic reporting of the energy and carbon footprints of machine learning,”Journal of Machine Learning Research, vol. 21, no. 248, pp. 1–43, 2020

    2020

  14. [14]

    Eco2AI: Carbon emissions tracking of machine learning models as a step towards sustainable AI,

    S. Budennyy, V . Lazarev, Z. Zakharkina, I. Ryabov,et al., “Eco2AI: Carbon emissions tracking of machine learning models as a step towards sustainable AI,”Scientific Reports, vol. 13, p. 18952, 2023

    work page 2023

  15. [15]

    Langfuse: Open source LLM engineering platform

    Langfuse Team, “Langfuse: Open source LLM engineering platform.” https://langfuse.com/docs, 2024

    work page 2024

  16. [16]

    ReAct: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR), 2023

    2023

  17. [17]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023

    work page 2023

  18. [18]

    LLM-augmenter: Augmenting large language models with plug-and-play modules,

    B. Peng, M. Galley, P. He, C. Brockett, L. Liden, E. Nouri, Z. Yu, B. Dolan, and J. Gao, “LLM-augmenter: Augmenting large language models with plug-and-play modules,”arXiv preprint arXiv:2302.12813, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    RestGPT: Connecting large language models with RESTful APIs,

    Y . Song, W. Xiong, D. Zhu, W. Wu, H. Qian, M. Bo, S. Li, and M. Zhang, “RestGPT: Connecting large language models with RESTful APIs,”arXiv preprint arXiv:2306.06624, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    API-Bank: A comprehensive benchmark for tool-augmented LLMs,

    M. Li, F. Song, B. Yu, H. Yu, Z. Li, F. Huang, and Y . Li, “API-Bank: A comprehensive benchmark for tool-augmented LLMs,” inEmpirical Methods in Natural Language Processing (EMNLP), pp. 3102–3116, 2023

    work page 2023

  21. [21]

    AgentOhana: Design uni- fied agent data with autonomous agent trajectories,

    J. Zhang, R. Hunter, A. Liu,et al., “AgentOhana: Design uni- fied agent data with autonomous agent trajectories,”arXiv preprint arXiv:2402.15506, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    A Survey on Data Selection for LLM Instruction Tuning

    X. Chen, M. Lin, N. Schärli, and D. Zhou, “APBench: A benchmark for evaluating API-based agents,”arXiv preprint arXiv:2402.05123, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    SQuAD: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” inEmpirical Methods in Natural Language Processing (EMNLP), pp. 2383–2392, 2016

    work page 2016

  24. [24]

    Docker: Lightweight linux containers for consistent development and deployment,

    D. Merkel, “Docker: Lightweight linux containers for consistent development and deployment,”Linux Journal, vol. 2014, no. 239, p. 2, 2014

    work page 2014

  25. [25]

    Deep reinforcement learning that matters,

    P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep reinforcement learning that matters,” inAAAI Con- ference on Artificial Intelligence, vol. 32, 2018

    work page 2018

  26. [26]

    Plan-and-solve prompting: Improving zero-shot chain-of- thought reasoning by large language models,

    L. Wang, W. Xu, Y . Lan, Z. Hu, Y . Lan, R. K.-W. Lee, and E.- P. Lim, “Plan-and-solve prompting: Improving zero-shot chain-of- thought reasoning by large language models,” inAnnual Meeting of the Association for Computational Linguistics (ACL), pp. 2609–2634, 2023

    work page 2023

  27. [27]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 24824–24837, 2022

    work page 2022

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.