Pith. sign in

REVIEW 2 major objections 2 minor 45 references

A Bayesian controller maintains belief over code correctness to decide dynamically when to verify or stop.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-25 23:31 UTC pith:T2KXPEYM

load-bearing objection The paper turns coding-agent tool orchestration into cost-sensitive Bayesian sequential testing with a maintained belief state, but the independence and binary-correctness assumptions are the load-bearing part that needs checking. the 2 major comments →

arxiv 2606.24453 v1 pith:T2KXPEYM submitted 2026-06-23 cs.AI cs.CL

Bayesian control for coding agents

Theodore Papamarkou , Vladislav Smirnov , Viktor Mazanov , Artem Vazhentsev , Preslav Nakov , Timothy Baldwin , Artem Shelmanov This is my paper
classification cs.AI cs.CL
keywords Bayesian controlcoding agentssequential hypothesis testinguncertainty quantificationLLM orchestrationcost-sensitive decisionsagent tool use
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper formulates orchestration of coding agents as cost-sensitive sequential hypothesis testing. A Bayesian controller updates its belief about whether a generated candidate is correct using signals from cheap diagnostics and costly verifiers. It then chooses among actions such as gathering more evidence, refining the candidate, running verification, or stopping. This policy is shown to reduce overall cost most effectively when verification is expensive and the available critics supply useful but imperfect information. The same belief state also supplies an interpretable probability of correctness that outperforms token-level probabilities and raw tool-success rates as an uncertainty measure.

Core claim

A Bayesian controller maintains a belief over the binary hypothesis of candidate correctness and uses cost-sensitive sequential testing to decide dynamically whether to gather evidence, refine, verify, or stop, yielding better performance on six generators and nine benchmarks when verification costs are high and critics are informative but imperfect.

What carries the argument

Bayesian controller that maintains and updates a belief distribution over binary correctness using conditionally independent signals from diagnostics and verifiers to minimize expected total cost.

Load-bearing premise

The diagnostics and verifiers supply signals whose informativeness can be captured by a simple Bayesian update over a binary correctness hypothesis.

What would settle it

A new benchmark or setting in which the Bayesian policy incurs higher total verification cost than a fixed-rule baseline while achieving equal or lower final accuracy.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Agents incur lower total verification cost while preserving solution accuracy across multiple generators and benchmarks.
  • The maintained belief serves as a calibrated correctness score superior to token-probability and raw success baselines.
  • Gains appear largest precisely when verification is costly and individual critics are informative but imperfect.
  • Orchestration shifts from fixed rules to sequential, cost-aware decisions that stop early when belief is sufficiently high or low.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same belief-maintenance structure could guide tool-use decisions in non-coding agent domains where actions carry different costs.
  • If signal dependence is stronger than assumed, replacing the simple update with a joint model might further reduce cost.
  • The correctness probability could be exposed to users or downstream systems as an explicit uncertainty flag.
  • Extending the state to track multiple candidate solutions at once might allow parallel refinement under a shared budget.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper formulates orchestration of LLM coding agents as cost-sensitive sequential hypothesis testing. A Bayesian controller maintains a posterior belief over a binary 'correct' hypothesis for candidate solutions and uses it to decide dynamically whether to collect more diagnostic evidence, refine the candidate, invoke an expensive verifier, or stop. Experiments across six generators and nine coding benchmarks show the controller is most valuable when verification is costly and critics are informative but imperfect; the resulting belief state also yields an interpretable correctness score that outperforms token-probability and raw tool-success baselines for uncertainty quantification.

Significance. If the modeling assumptions and empirical claims hold after validation, the work supplies a principled, uncertainty-aware alternative to fixed-rule orchestrators for tool-using coding agents. The explicit separation of control from generation and the use of the belief state for both stopping and UQ are potentially reusable contributions beyond the specific benchmarks.

major comments (2)
  1. [§3.2] §3.2 (Bayesian update): The controller relies on a simple sequential update that treats diagnostic and verifier signals as conditionally independent given the binary correctness hypothesis. No diagnostic checks, sensitivity analysis, or empirical tests for dependence or for the adequacy of the binary state are reported; if signals share latent failure modes or if partial correctness matters, both the cost-sensitive policy and the claimed UQ superiority would be miscalibrated.
  2. [§5] §5 (Experiments): The abstract states that Bayesian control 'proves to be most valuable' under specific conditions, yet the manuscript provides no pre-registered analysis plan, multiple-testing correction, or ablation isolating the contribution of the independence assumption versus other design choices. Without these, it is unclear whether the reported gains survive controls for benchmark selection and hyper-parameter tuning.
minor comments (2)
  1. Notation for the belief state and likelihood functions is introduced without a compact reference table; a single summary table would improve readability.
  2. Figure captions for the cost-sensitivity plots do not state the exact cost ratios used, making it hard to reproduce the 'most valuable when verification is costly' claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We address each major point below and indicate the revisions we will make to strengthen the manuscript.

read point-by-point responses

  1. Referee: [§3.2] §3.2 (Bayesian update): The controller relies on a simple sequential update that treats diagnostic and verifier signals as conditionally independent given the binary correctness hypothesis. No diagnostic checks, sensitivity analysis, or empirical tests for dependence or for the adequacy of the binary state are reported; if signals share latent failure modes or if partial correctness matters, both the cost-sensitive policy and the claimed UQ superiority would be miscalibrated.

    Authors: We agree that the conditional independence assumption merits explicit validation. In the revised version we will add a dedicated sensitivity analysis subsection that (i) derives the effect of pairwise signal dependence on the posterior trajectory under a simple correlation model and (ii) reports empirical posterior calibration on two benchmarks where both diagnostic and verifier outcomes are available for the same candidates. We will also clarify that the binary hypothesis is an operational abstraction chosen because the controller’s cost-sensitive stopping rule is defined with respect to the probability of a fully correct solution; partial correctness is already handled upstream by the generators and is not claimed to be modeled by the belief state itself. revision: partial

  2. Referee: [§5] §5 (Experiments): The abstract states that Bayesian control 'proves to be most valuable' under specific conditions, yet the manuscript provides no pre-registered analysis plan, multiple-testing correction, or ablation isolating the contribution of the independence assumption versus other design choices. Without these, it is unclear whether the reported gains survive controls for benchmark selection and hyper-parameter tuning.

    Authors: The experiments were exploratory and the primary claims rest on consistent qualitative patterns across six generators and nine benchmarks rather than on formal hypothesis tests. We acknowledge that a pre-registered analysis plan was not used. In revision we will (i) add an explicit ablation that replaces the independence assumption with a simple joint likelihood model on a subset of tasks and (ii) include a supplementary table that recomputes all headline metrics after re-tuning the controller hyper-parameters on a held-out benchmark split. We will also add a limitations paragraph discussing the absence of pre-registration and multiple-testing correction. revision: partial

Circularity Check

0 steps flagged

No circularity: standard Bayesian update applied to tool orchestration without self-referential fitting or load-bearing self-citations.

full rationale

The abstract formulates orchestration as cost-sensitive sequential hypothesis testing with a Bayesian belief over binary correctness, updated from diagnostics and verifiers. No equations appear that define a parameter from data and then rename its output as a prediction. No self-citation chains, uniqueness theorems, or ansatzes are invoked. The claimed superiority is presented as an empirical result across generators and benchmarks rather than a tautological consequence of the modeling assumptions. The reader's assessment of score 2 aligns with the absence of any load-bearing reduction; the modeling choice (conditional independence, binary state) is an explicit assumption open to falsification, not a hidden circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only; no free parameters, axioms, or invented entities are mentioned. The Bayesian update itself is treated as a standard tool rather than derived in the paper.

pith-pipeline@v0.9.1-grok · 5645 in / 1141 out tokens · 22227 ms · 2026-06-25T23:31:49.882940+00:00 · methodology

0 comments
read the original abstract

Modern coding agents pair LLM generators with various tools, including cheap diagnostics and expensive verifiers. The tool-use decisions are typically governed by orchestrators that often use fixed rules and ignore uncertainty. We formulate orchestration as cost-sensitive sequential hypothesis testing: a Bayesian controller maintains a belief over candidate correctness and dynamically decides whether to gather more evidence, refine the candidate, verify it, or stop. Across six generators and nine coding benchmarks, Bayesian control proves to be most valuable when verification is costly and critics are informative but imperfect. Beyond control, the belief state yields an interpretable correctness score that outperforms token-probability and raw tool-success baselines for uncertainty quantification.

Figures

Figures reproduced from arXiv: 2606.24453 by Artem Shelmanov, Artem Vazhentsev, Preslav Nakov, Theodore Papamarkou, Timothy Baldwin, Viktor Mazanov, Vladislav Smirnov.

Figure 1
Figure 1. Figure 1: Illustration of the Bayesian control loop. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-policy ∆ utility against always_verify on SWE-Bench Lite / claude-haiku-4.5. Gold stars mark per-cell winners; numbers above bars are abso￾lute mean utility; error bars are paired-bootstrap 95% CIs (B=1000). Discussion in §6.2; per-cell evaluation￾set bars across all (benchmark, generator) pairs in Appendix D.1; full train-vs-evaluation panel in Ap￾pendix D.2. saving that does not justify the loss. 6.2… view at source ↗
Figure 2
Figure 2. Figure 2: Empirical decision regions over the (P(Y =1), Cver/R) plane: k-NN (k=300) over 7,020 sweep points. Colors: A (green) Bayesian; B (blue) gate(Crtest); C (yellow) always_verify; other (coral). See §6.1 for the full regime description. cost, the optimal Bayes decision often is a gate on the public-test critic. On cells where Crtest is highly informative (γtest ≳ 0.7 on LCB and on most Hu￾manEvalFix and CodeCo… view at source ↗
Figure 4
Figure 4. Figure 4: sweeps Cver ∈ [1, 100] on MBPP+ / claude-haiku-4.5 at fixed R=100, with the other costs held at their default values. The sweep traverses all three regimes within a single cell, illustrating the C→ B →A handoff: at low Cver, always_verify wins because the prior is high enough that blind verification has a positive expected value; at moderate Cver, gate(Crtest) exploits the public-test critic; at high Cver,… view at source ↗
Figure 5
Figure 5. Figure 5: Critic informativeness γi = Pi(z=pass | Y =1) − Pi(z=pass | Y =0) vs prior P(Y =1) across the available benchmark–generator pairs, one subplot per critic layer. Point color indicates the Bayesian ∆ for each benchmark–generator pair. SWE-Bench cells do not appear in the Crtest (public tests) panel: SWE-Bench has no public-test split distinct from the oracle verifier (the test_patch is the verifier), so ther… view at source ↗
Figure 6
Figure 6. Figure 6: Per-benchmark location of each generator in ( [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Held-out eval policy comparison on LCB-hard, all generators on one panel. Bayesian variants dominate [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Held-out eval policy comparison on LCB-medium, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Held-out eval policy comparison on LCB-easy, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Held-out eval policy comparison on MBPP+, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Held-out eval policy comparison on HumanEval+, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Held-out eval policy comparison on HumanEvalFix, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Held-out eval policy comparison on CodeContests, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p022_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Held-out eval policy comparison on SWE-Bench Lite, all generators on one panel (the [PITH_FULL_IMAGE:figures/full_fig_p022_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Held-out eval policy comparison on SWE-Bench Verified, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Train-vs-eval policy comparison on LCB-hard. Left column: 75% train (in-sample fit). Right column: [PITH_FULL_IMAGE:figures/full_fig_p024_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Train-vs-eval policy comparison on LCB-medium. [PITH_FULL_IMAGE:figures/full_fig_p025_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Train-vs-eval policy comparison on LCB-easy. [PITH_FULL_IMAGE:figures/full_fig_p026_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Train-vs-eval policy comparison on MBPP+. [PITH_FULL_IMAGE:figures/full_fig_p027_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Train-vs-eval policy comparison on HumanEval+. [PITH_FULL_IMAGE:figures/full_fig_p028_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Train-vs-eval policy comparison on HumanEvalFix. [PITH_FULL_IMAGE:figures/full_fig_p029_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Train-vs-eval policy comparison on CodeContests. [PITH_FULL_IMAGE:figures/full_fig_p030_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Train-vs-eval policy comparison on SWE-Bench Lite. The [PITH_FULL_IMAGE:figures/full_fig_p031_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Train-vs-eval policy comparison on SWE-Bench Verified. [PITH_FULL_IMAGE:figures/full_fig_p032_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Cver sweep on MBPP+, one sub-panel per generator (companion to [PITH_FULL_IMAGE:figures/full_fig_p034_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Cver sweep on LCB-hard, one sub-panel per generator. LCB-hard’s low prior (P(Y =1) ≈ 0.17) places the analytic crossover at C ∗ ver ≈ 17, so the cell sits in regime A for nearly the entire sweep – the Bayesian variants lead above Cver ≈ 20. The two Bayesian curves overlap because the measured P(fix | broken) ≈ 0.07 makes multi-step refinement negative-EV, so bayesian_DP collapses to the same myopic policy… view at source ↗
Figure 27
Figure 27. Figure 27: Cver sweep on LCB-medium, one sub-panel per generator [PITH_FULL_IMAGE:figures/full_fig_p035_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Cver sweep on LCB-easy, one sub-panel per generator. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Cver sweep on HumanEval+, one sub-panel per generator [PITH_FULL_IMAGE:figures/full_fig_p036_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Cver sweep on HumanEvalFix, one sub-panel per generator. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: Cver sweep on CodeContests, one sub-panel per generator [PITH_FULL_IMAGE:figures/full_fig_p037_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: Cver sweep on SWE-Bench Lite, one sub-panel per generator. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_32.png] view at source ↗
Figure 33
Figure 33. Figure 33: Cver sweep on SWE-Bench Verified, one sub-panel per generator. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_33.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. CodeT : C ode generation with generated tests. In International Conference on Learning Representations (ICLR)

    2023

  2. [2]

    Deepro Choudhury, Sinead Williamson, Adam Goli \'n ski, Ning Miao, Freddie Bickford Smith, Michael Kirchhof, Yizhe Zhang, and Tom Rainforth. 2026. BED-LLM : I ntelligent information gathering with LLM s and B ayesian experimental design. In International Conference on Learning Representations (ICLR)

    2026

  3. [3]

    Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Fr \'e d \'e ric Blain, Francisco Guzm \'a n, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539--555

    2020

  4. [4]

    Ronald A. Howard. 1966. Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2(1):22--26

    1966

  5. [6]

    Lahiri, Madanlal Musuvathi, and Jianfeng Gao

    Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnaci \'o n, Shuvendu K. Lahiri, Madanlal Musuvathi, and Jianfeng Gao. 2022. Fault-aware neural code rankers. In Advances in Neural Information Processing Systems (NeurIPS)

    2022

  6. [7]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Live C ode B ench: H olistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations (ICLR)

    2025

  7. [8]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench : C an language models resolve real-world Github issues? In International Conference on Learning Representations (ICLR)

    2024

  8. [9]

    Littman, and Anthony R

    Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. 1998. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1--2):99--134

    1998

  9. [10]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R\' e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, and 7 others. 2022. Competition-level code generation with A...

    2022

  10. [11]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In International Conference on Neural Information Processing Systems (NeurIPS)

    2023

  11. [12]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self- R efine: I terative refinement with self-feedback. In Advances in Neural Information Proc...

    2023

  12. [13]

    Andrey Malinin and Mark Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations (ICLR)

    2021

  13. [14]

    Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2024. Octo P ack: I nstruction tuning code large language models. In International Conference on Learning Representations (ICLR)

    2024

  14. [15]

    Theodore Papamarkou, Pierre Alquier, Matthias Bauer, Wray Buntine, Andrew Davison, Gintare Karolina Dziugaite, Maurizio Filippone, Andrew Y. K. Foong, Vincent Fortuin, Dimitris Fouskakis, Jes Frellsen, Eyke Hüllermeier, Theofanis Karaletsos, Mohammad Emtiyaz Khan, Nikita Kotelevskii, Salem Lahlou, Yingzhen Li, Fang Liu, Clare Lyle, and 11 others. 2026. Po...

    2026

  15. [16]

    Narasimhan, and Shunyu Yao

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R. Narasimhan, and Shunyu Yao. 2023. Reflexion: L anguage agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS)

    2023

  16. [18]

    Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2025. Benchmarking uncertainty quantification methods for large language models with LM - P olygrap...

    2025

  17. [19]

    Abraham Wald. 1947. Sequential analysis. John Wiley & Sons

    1947

  18. [20]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, and 5 others. 2025. Open H ands: A n open platform for AI software developers as generalist agents. In Int...

    2025

  19. [21]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R. Narasimhan, and Ofir Press. 2024. SWE-agent : A gent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS)

    2024

  20. [22]

    Hashimoto, Mike Lewis, Wen-Tau Yih, Daniel Fried, and Sida I

    Tianyi Zhang, Tao Yu, Tatsunori B. Hashimoto, Mike Lewis, Wen-Tau Yih, Daniel Fried, and Sida I. Wang. 2023. Coder reviewer reranking for code generation. In International Conference on Machine Learning (ICML)

    2023

  21. [23]

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2024. Language agent tree search unifies reasoning, acting, and planning in language models. In International Conference on Learning Representations (ICLR)

    2024

  22. [24]

    2024 , publisher =

    Du, Xueying and Liu, Mingwei and Wang, Kaixin and Wang, Hanlin and Liu, Junwei and Chen, Yixuan and Feng, Jiayi and Sha, Chaofeng and Peng, Xin and Lou, Yiling , title =. 2024 , publisher =

    2024

  23. [25]

    and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R

    Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R. , title =. International Conference on Learning Representations (ICLR) , year =

  24. [26]

    2023 , booktitle =

    Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming , title =. 2023 , booktitle =

    2023

  25. [27]

    Niklas Muennighoff and Qian Liu and Armel Randy Zebaze and Qinkai Zheng and Binyuan Hui and Terry Yue Zhuo and Swayam Singh and Xiangru Tang and Leandro Von Werra and Shayne Longpre , booktitle=. Octo

  26. [28]

    Competition-level code generation with

    Li, Yujia and Choi, David and Chung, Junyoung and Kushman, Nate and Schrittwieser, Julian and Leblond, R\'. Competition-level code generation with. Science , publisher =. 2022 , pages =

    2022

  27. [29]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Madaan, Aman and Tandon, Niket and Gupta, Prakhar and Hallinan, Skyler and Gao, Luyu and Wiegreffe, Sarah and Alon, Uri and Dziri, Nouha and Prabhumoye, Shrimai and Yang, Yiming and Gupta, Shashank and Majumder, Bodhisattwa Prasad and Hermann, Katherine and Welleck, Sean and Yazdanbakhsh, Amir and Clark, Peter , title =. Advances in Neural Information Pro...

  28. [30]

    and Yao, Shunyu , title =

    Shinn, Noah and Cassano, Federico and Gopinath, Ashwin and Narasimhan, Karthik R. and Yao, Shunyu , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  29. [31]

    Wald, Abraham , title =

  30. [32]

    , title =

    Howard, Ronald A. , title =. IEEE Transactions on Systems Science and Cybernetics , volume =

  31. [33]

    Naman Jain and King Han and Alex Gu and Wen-Ding Li and Fanjia Yan and Tianjun Zhang and Sida Wang and Armando Solar-Lezama and Koushik Sen and Ion Stoica , booktitle=. Live

  32. [34]

    Papamarkou, Theodore and Alquier, Pierre and Bauer, Matthias and Buntine, Wray and Davison, Andrew and Dziugaite, Gintare Karolina and Filippone, Maurizio and Foong, Andrew Y. K. and Fortuin, Vincent and Fouskakis, Dimitris and Frellsen, Jes and Hüllermeier, Eyke and Karaletsos, Theofanis and Khan, Mohammad Emtiyaz and Kotelevskii, Nikita and Lahlou, Sale...

  33. [35]

    and Cassandra, Anthony R

    Kaelbling, Leslie Pack and Littman, Michael L. and Cassandra, Anthony R. , title =. Artificial Intelligence , volume =

  34. [36]

    International Conference on Learning Representations (ICLR) , year =

    Choudhury, Deepro and Williamson, Sinead and Goli. International Conference on Learning Representations (ICLR) , year =

  35. [37]

    Structured Uncertainty guided Clarification for LLM Agents

    Suri, Manan and Mathur, Puneet and Lipka, Nedim and Dernoncourt, Franck and Rossi, Ryan A. and Manocha, Dinesh , title =. arXiv preprint arXiv:2511.08798 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  36. [38]

    International Conference on Learning Representations (ICLR) , year =

    Chen, Bei and Zhang, Fengji and Nguyen, Anh and Zan, Daoguang and Lin, Zeqi and Lou, Jian-Guang and Chen, Weizhu , title =. International Conference on Learning Representations (ICLR) , year =

  37. [39]

    Fault-aware neural code rankers , booktitle =

    Inala, Jeevana Priya and Wang, Chenglong and Yang, Mei and Codas, Andres and Encarnaci. Fault-aware neural code rankers , booktitle =

  38. [40]

    and Lewis, Mike and Yih, Wen-Tau and Fried, Daniel and Wang, Sida I

    Zhang, Tianyi and Yu, Tao and Hashimoto, Tatsunori B. and Lewis, Mike and Yih, Wen-Tau and Fried, Daniel and Wang, Sida I. , title =. International Conference on Machine Learning (ICML) , year =

  39. [41]

    AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation

    Huang, Dong and Zhang, Jie M. and Luck, Michael and Bu, Qingwen and Qing, Yuhao and Cui, Heming , title =. arXiv preprint arXiv:2312.13010 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  40. [42]

    International Conference on Learning Representations (ICLR) , year =

    Zhou, Andy and Yan, Kai and Shlapentokh-Rothman, Michal and Wang, Haohan and Wang, Yu-Xiong , title =. International Conference on Learning Representations (ICLR) , year =

  41. [43]

    and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik R

    Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik R. and Press, Ofir , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  42. [44]

    Xu and Xiangru Tang and Mingchen Zhuge and Jiayi Pan and Yueqi Song and Bowen Li and Jaskirat Singh and Hoang H

    Xingyao Wang and Boxuan Li and Yufan Song and Frank F. Xu and Xiangru Tang and Mingchen Zhuge and Jiayi Pan and Yueqi Song and Bowen Li and Jaskirat Singh and Hoang H. Tran and Fuqiang Li and Ren Ma and Mingzhang Zheng and Bill Qian and Yanjun Shao and Niklas Muennighoff and Yizhe Zhang and Binyuan Hui and Junyang Lin and Robert Brennan and Hao Peng and H...

  43. [45]

    Benchmarking uncertainty quantification methods for large language models with

    Vashurin, Roman and Fadeeva, Ekaterina and Vazhentsev, Artem and Rvanova, Lyudmila and Vasilev, Daniil and Tsvigun, Akim and Petrakov, Sergey and Xing, Rui and Sadallah, Abdelrahman and Grishchenkov, Kirill and Panchenko, Alexander and Baldwin, Timothy and Nakov, Preslav and Panov, Maxim and Shelmanov, Artem , journal =. Benchmarking uncertainty quantific...

    2025

  44. [46]

    International Conference on Learning Representations (ICLR) , year =

    Andrey Malinin and Mark Gales , title =. International Conference on Learning Representations (ICLR) , year =

  45. [47]

    Transactions of the Association for Computational Linguistics , volume =

    Unsupervised quality estimation for neural machine translation , author =. Transactions of the Association for Computational Linguistics , volume =. 2020 , publisher =

    2020