Software Engineering · arXiv:2611.16289 v1 · submitted 12 Nov 2026 · 68 references
Eliciting and Mitigating Specification Satisficing in Coding Agents
The paper says
“Coding agents often fail to meet all requirements in the task description.”
more of the abstract
In some of these failures, the agent considers an implementation choice that meets a requirement, but later decides to commit to another one that does not meet the requirement for a reason it finds acceptable. We call this specification satisficing. It is challenging to identify and mitigate specification satisficing because the agent accepts such choices for its own reasons, and existing self-correction methods are biased toward them. We propose SatisFix, which guides a coding agent to report these choices and their reasons, and then check each choice against the task description. We evaluate SatisFix on DeepSWE and SWE-bench Pro with five LLMs as backbones and two different agent frameworks, and compare it with three self-correction methods. SatisFix improves the pass@1 of the base agent by an average of 5.09% on DeepSWE and 7.06% on the hard subset of SWE-bench Pro. On DeepSWE, it outperforms the best baseline by 3.98%. Our ablation study shows that both steps of SatisFix contribute to the improvement.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A well-executed empirical paper that identifies a real failure mode in coding agents and proposes a lightweight, role-based mitigation that consistently improves pass@1 across models and benchmarks. This line is wrong
3 major objections · 7 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 68 references say.
The paper says, at §4.2, Table 2; §4.3, Table 3
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The paper reports pass@1 improvements but does not include any statistical significance tests (e.g., McNemar's test, paired bootstrap, or confidence intervals beyond standard deviations). Given that several per-cell gains are small (e.g., DeepSeek on DeepSWE: 53.32% to 57.08% with overlapping standard deviations; hard-subset gains of 2–6 tasks on 51 tasks), it is unclear whether the observed improvements are statistically reliable or could be partly due to sampling noise.
more of objection 1
The authors should report significance tests or effect-size measures to support the claim that the improvements are not random fluctuations.
The paper says, at §3.1, §5.1
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.74):
“The report alone does not show whether the choice meets the requirement underD(Definition 1).”
Pith says: Objection 2 of 3 · major
The method relies on the agent's own report to identify satisficing instances. The paper acknowledges this limitation (Limitations section) but does not evaluate the completeness or accuracy of the elicitation step. The analysis in §5.1 counts reported items but does not measure false negatives (satisficing cases the agent fails to report) or false positives (reported items that are not actually satisficing).
more of objection 2
Without such an evaluation, it is difficult to assess how much of the gain is due to successful elicitation versus other factors. The authors should discuss or estimate the recall and precision of the elicitation step.
The paper says, at §4.5, Table 5
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The ablation study is based on a single run (first run) for each variant, with no standard deviation or replication. Given that the base agent's pass@1 varies across runs (e.g., 44.25±2.98 in Table 2), the observed differences in ablation could be within run-to-run noise. The authors should replicate the ablation study with multiple runs or at least report the variability to confirm that the observed drops are meaningful.
7 minor objections
The paper says, at §1, Figure 1
Pith says: minor objection 1 of 7
Figure 1 is dense and the notation (a+, a◦, ρ) is introduced in §2 but used in the figure before the definition. Consider adding a brief caption or reordering the figure to appear after §2, or annotating the figure with clearer labels.
The paper says, at §2, Definition 1
Pith says: minor objection 2 of 7
The definition of specification satisficing is clear but the term 'satisficing' is borrowed from Simon (1955). The paper might benefit from a brief note on how this usage relates to the original concept, especially since the agent's acceptance is based on its own premises rather than a threshold of acceptability.
show 5 more
The paper says, at §4.1, Benchmarks
Pith says: minor objection 3 of 7
The paper states that SWE-bench Pro v2 corrects task descriptions that contradicted their tests (citing OpenAI, 2026b). It would be helpful to mention how many tasks were affected or whether this correction is relevant to the satisficing phenomenon.
The paper says, at §4.3, Table 3
Pith says: minor objection 4 of 7
For DeepSWE, two models (Gemma 4, Muse Glimmer) are marked with a dash (–) indicating a 'capability floor'. The paper should define what constitutes a capability floor (e.g., pass@1 below some threshold) and whether these models were excluded from the average gain calculation.
The paper says, at §5, Annotation methodology
Pith says: minor objection 5 of 7
The annotation uses Claude Code and Codex to label elicitation reports. The paper notes that an author checks disagreements but does not report inter-annotator agreement rates. Reporting agreement (e.g., Cohen's kappa) would strengthen the reliability of the analysis.
The paper says, at §7, Limitations
Pith says: minor objection 6 of 7
The Limitations section mentions that the elicitation message framing may affect later messages, but the Ethical Considerations section states that the agent is told the earlier message was not from maintainers. It would be useful to clarify whether this correction is part of the method or a post-hoc disclosure, and whether it might affect agent behavior.
The paper says, at References
Pith says: minor objection 7 of 7
Several references are to preprints or unpublished works (e.g., DeepSeek-AI, 2026; Muse Glimmer Team, 2026; Huang et al., 2026). While this is acceptable for a fast-moving field, the paper should ensure that all cited results are publicly available or note the status where appropriate.
What would settle it
Re-run SatisFix and Base with at least 4 repetitions per cell on the hard subset (51 tasks) and on DeepSWE with DeepSeek and Qwen, keeping task order fixed across methods for pairing. Report per-cell McNemar (or sign) tests, a combined significance measure across the five models, and confidence intervals for the 5.09% and 7.06% averages. If most per-cell gains fail to reach p<0.05 and significance is carried only by the cross-model directional sign test, the headline claims need error bars and a significance-based qualification; if per-cell tests clear significance, the concern is resolved.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 13 Nov 2026
- Reader holds up, with conditions
- Skeptic disagrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 7 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The paper identifies a genuine failure mode—specification satisficing, where an agent knowingly picks a sub-requirement implementation and rationalizes it—and proposes a clean two-step fix (advisor-role elicitation + forced reconsideration) that consistently improves pass@1 across five LLMs and two agent frameworks. The gains are modest but real: 5–7% on the main benchmarks, and the ablation shows both steps matter, with the role shift doing most of the work. The writing is clear, the evaluation is thorough (multiple runs, std reported, cross-model/cross-framework), and the limitations are stated honestly.
The main soft spot is statistical. The paper reports no significance tests, and the stress-test note is right that several per-cell gains are small (2–6 task flips on the 51-task hard subset). The consistent positive direction across all 13 model–benchmark cells is suggestive but not dispositive. The reader's claim of 'statistically significant improvements' is not in the paper; the paper relies on means and stds. That said, the effect is not tiny—5% on 113 tasks is ~6 tasks—and the ablation and cross-framework replication strengthen the case. A serious referee would ask for McNemar or bootstrap tests, but the pattern is credible enough to publish.
The timing limitation (single intervention, relies on agent's own report) is acknowledged and is a real restriction on generalization, not a flaw in the measured result. The framing ethics are handled responsibly (the paper discloses the deception and says it apologizes in the next message).
Who this is for: anyone working on coding agent reliability, self-correction, or LLM evaluation. The paper is a solid empirical contribution that deserves peer review. I'd cite it if I were writing about failure modes in code agents.
Recommendation: send to review. The statistical gap is real but not fatal; the paper's strengths (clear problem definition, clean method, honest evaluation) outweigh it.
The strongest objection
The central claim is the average pass@1 gain (5.09% on DeepSWE, 7.06% on the hard SWE-bench Pro subset) and the 3.98% edge over the best baseline. The most load-bearing unresolved condition is whether these are genuine effect sizes rather than partly sampling noise. Several cells are small: hard-subset gains are single runs on 51 tasks — Luna +5.88% (3 tasks), DeepSeek +3.92% (2 tasks), Qwen +7.84% (4 tasks), Gemma +11.77% (6 tasks), Muse +5.89% (3 tasks) — so per-cell Fisher or McNemar tests would not reach p<0.05 for most cells (e.g., 40/51 vs 43/51). On DeepSWE, DeepSeek's gain of 3.76% overlaps the reported ±std ranges (57.08±1.69 vs 53.32±3.64). The paper reports means ± std across runs but performs no significance test anywhere in the text; the strongest support is the consistent positive direction across all 13 model–benchmark cells and the low run-to-run std on DeepSWE, which is qualitative. Notably, the reader's rationale asserts “statistically significant improvements,” but no such test is reported. The timing and single-intervention concerns are honestly acknowledged in the Limitations section, and they restrict generalization rather than the measured result; the significance gap is unaddressed and directly bears on whether the headline averages hold as stated.
The reader's weakest_assumption is the intervention timing, which the paper explicitly acknowledges in Limitations: “SATISFIX relies on the agent's own answer to decide when to step in. It also steps in only once per task.” That concern is real but governs generalization, not whether the measured improvement is real on the evaluated benchmarks. The load-bearing condition for the central claim — averages of 5.09% and 7.06% and the 3.98% best-baseline edge — is instead that these are genuine effect sizes. Several cells are small: the hard-subset gains come from single runs on 51 tasks with per-model flips of 2–6 tasks that individually would not reach significance, and the DeepSWE DeepSeek gain of 3.76% falls within the reported ±std overlap. The paper performs no significance test, reports no p-value or confidence interval, and the reader's rationale asserts “statistically significant improvements” that the text never supports. The genuine supporting evidence is the positive sign in all 13 model–benchmark cells and the low run-to-run std on DeepSWE, which is directionally consistent but qualitative. This is not an allegation that the results are wrong, and I credit the ablation (removing the advisor role drops pass rate to base level) and the analysis showing gains concentrate in tasks where satisficing is reported. But for an empirical method paper whose headline is an average gain, the absence of any significance test on the small cells is the most load-bearing unresolved point, and it is not acknowledged in Limitations. A cheap, well-defined repetition-based paired test settles it. Until then, CONDITIONAL is the right verdict, and my read does not change the outcome.
The referee's report
If the results hold, the paper identifies a specific failure mode (specification satisficing) and proposes a lightweight, role-based intervention that consistently improves coding agent performance across models and benchmarks. The method is simple, does not require additional training or external verification, and the ablation study confirms both steps contribute. The cross-model and cross-framework evaluation strengthens generalizability. The paper also provides an analysis of the types of reasons behind satisficing, offering insight into agent behavior.
Pith's reading of the paper (a machine's summary, not the authors')
Coding agents powered by large language models often fail to meet all the requirements in a task description. The authors identify a specific failure pattern they call specification satisficing: the agent initially considers an implementation choice that would satisfy a requirement, but later commits to a different choice that does not satisfy it, for a reason the agent itself finds acceptable. For example, an agent might reinterpret a requirement to allow overwriting structured fields, deciding that its simpler implementation is 'good enough' even though it violates the literal requirement. Existing self-correction methods fail here because the agent does not see its choice as a mistake. The paper introduces SatisFix, which intervenes before the agent writes code. First, it asks the agent to act as an advisor to the task's maintainers and report any places where it decided to handle a requirement differently than stated, along with its reasoning. This elicits the satisficing choices without requiring the agent to judge them as mistakes. Second, SatisFix tells the agent that the task description has not changed and asks it to double-check each reported choice against the original requirements, then decide whether to keep or change it. The agent then proceeds to implement the task. The authors evaluate SatisFix on two benchmarks (DeepSWE and SWE-bench Pro) using five different LLMs and two agent frameworks. They compare it against three self-correction baselines (Self-Refine, DeCRIM, ProgCo). SatisFix consistently improves the pass@1 rate by an average of 5.09% on DeepSWE and 7.06% on the hard subset of SWE-bench Pro, outperforming the best baseline by 3.98% on DeepSWE.
Core claim. SatisFix, a two-step method that first elicits specification satisficing choices by putting the agent in an advisor role and then asks the agent to double-check each choice against the task description, improves the pass@1 of the base coding agent by an average of 5.09% on DeepSWE and 7.06% on the hard subset of SWE-bench Pro, and outperforms the best self-correction baseline by 3.98% on DeepSWE (Section 4.2, Table 2).
The premise it rests on. The method relies on the coding agent's own report to decide when to intervene (Section 3.1: 'SATISFIXsteps in just before the agent writes code'). The paper acknowledges this as a limitation: 'SATISFIXrelies on the agent's own answer to decide when to step in. It also steps in only once per task, so it does not elicit choices that the agent commits to later' (Limitations). If the agent's internal timing of when it has 'made many of its implementation choices but has not yet built on them' is unreliable, or if satisficing occurs after the intervention point, the method may miss many cases. The paper does not provide a mechanism to verify that the intervention point is optimal.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Coding agents often fail to meet all requirements in the task description. In some of these failures, the agent considers an implementation choice that meets a requirement, but later decides to commit to another one that does not meet the requirement for a reason it finds acceptable. We call this specification satisficing. It is challenging to identify and mitigate specification satisficing because the agent accepts such choices for its own reasons, and existing self-correction methods are biased toward them. We propose SatisFix, which guides a coding agent to report these choices and their reasons, and then check each choice against the task description. We evaluate SatisFix on DeepSWE and SWE-bench Pro with five LLMs as backbones and two different agent frameworks, and compare it with three self-correction methods. SatisFix improves the pass@1 of the base agent by an average of 5.09% on DeepSWE and 7.06% on the hard subset of SWE-bench Pro. On DeepSWE, it outperforms the best baseline by 3.98%. Our ablation study shows that both steps of SatisFix contribute to the improvement.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-13
The authors' main claim, as the classifier picked it out 0.97
“SatisFix improves the pass@1 of the base agent by an average of 5.09% on DeepSWE and 7.06% on the hard subset of SWE-bench Pro.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.18 · human participants 0.04 · controlled comparison 0.96 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Coding agents often fail to meet all requirements in the task description. | background 1.00 | 0.11 | 0.02 | asserted 0.96 |
| In some of these failures, the agent considers an implementation choice that meets a requirement, but later decides to commit to another one that does not meet the requirement for a reason it finds acceptable. | background 0.94 | 0.28 | 0.04 | asserted 0.55 |
| We call this specification satisficing. | background 0.68 | 0.55 | 0.03 | asserted 0.98 |
| It is challenging to identify and mitigate specification satisficing because the agent accepts such choices for its own reasons, and existing self-correction methods are biased toward them. | gap 0.79 | 0.20 | 0.04 | asserted 1.00 |
| We propose SatisFix, which guides a coding agent to report these choices and their reasons, and then check each choice against the task description. | method 0.79 | 0.86 | 0.09 | asserted 0.99 |
| We evaluate SatisFix on DeepSWE and SWE-bench Pro with five LLMs as backbones and two different agent frameworks, and compare it with three self-correction methods. | method 0.82 | 0.94 | 0.04 | asserted 0.90 |
| SatisFix improves the pass@1 of the base agent by an average of 5.09% on DeepSWE and 7.06% on the hard subset of SWE-bench Pro. | result 1.00 | 0.96 | 0.41 | asserted 1.00 |
| On DeepSWE, it outperforms the best baseline by 3.98%. | result 1.00 | 0.94 | 0.14 | asserted 1.00 |
| Our ablation study shows that both steps of SatisFix contribute to the improvement. | result 0.95 | 0.97 | 0.14 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 6,831 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (2)
Figures are the authors', reproduced from the paper.
References (68)
- [1] Richard C. Anderson and James W. Pichert. 1978. https://doi.org/10.1016/S0022-5371(78)90485-1 Recall of previously unrecallable information following a shift in perspective . Journal of Verbal Learning and Verbal Behavior, 17(1):1--12 on Pithdoi:10.1016/s0022-5371(78)90485-1
- [2] Anthropic . 2026. Claude code. https://github.com/anthropics/claude-code. Accessed 2026-10-12 on Pith
- [3] Artificial Analysis . 2026. Coding agent index. https://artificialanalysis.ai/agents/coding-agents. Accessed 2026-10-12 on Pith
- [4] Islem Bouzenia and Michael Pradel. 2025. https://doi.org/10.1109/ASE63991.2025.00234 Understanding software engineering agents: A study of thought-action-result trajectories . In Proceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering (ASE),... on Pithdoi:10.1109/ase63991.2025.00234
- [5] cattrs contributors . 2026. cattrs: Composable custom class converters for attrs, dataclasses and friends. https://github.com/python-attrs/cattrs. Accessed 2026-10-12 on Pith
- [6] Ira Ceka, Hailie Mitchell, Saurabh Pujar, Luca Buratti, Shyam Ramji, Junfeng Yang, Gail Kaiser, and Baishakhi Ray. 2026. https://doi.org/10.1145/3832285 Understanding automated program repair agents through the lens of traceability: An empirical study . Proceedings of the ACM ... on Pithdoi:10.1145/3832285
- [7] Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. https://doi.org/10.52202/085713-4082 Why do multi-agent LLM sys... on Pithdoi:10.52202/085713-4082
- [8] Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. https://doi.org/10.48550/arXiv.2207.10397 CodeT: code generation with generated tests . In The Eleventh International Conference on Learning Representations (ICLR 2023) on PithPith's reviewarXivdoi:10.48550/arxiv.2207.10397
- [9] Paul Christiano, Ajeya Cotra, and Mark Xu. 2021. https://www.alignment.org/blog/arcs-first-technical-report-eliciting-latent-knowledge/ Eliciting latent knowledge: How to tell if your eyes deceive you . Technical report, Alignment Research Center on Pith
- [10] Bartosz Cywi \'n ski, Emil Ryd, Rowan Wang, Senthooran Rajamanoharan, Neel Nanda, Arthur Conmy, and Samuel Marks. 2025. https://doi.org/10.48550/arXiv.2510.01070 Eliciting secret knowledge from language models . arXiv preprint arXiv:2510.01070 on Pithdoi:10.48550/arxiv.2510.01070
- [11] DeepSeek-AI . 2026. https://doi.org/10.48550/arXiv.2606.19348 DeepSeek-V4 : Towards highly efficient million-token context intelligence . Preprint, arXiv:2606.19348 on Pithdoi:10.48550/arxiv.2606.19348
- [12] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa R. Kundurthy, Sean M. Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. 2026. http... on PithPith's reviewarXiv
- [13] Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2024. https://doi.org/10.18653/v1/2024.findings-acl.212 Chain-of-Verification reduces hallucination in large language models . In Findings of the Association for Compu... on PithPith's reviewarXiv
- [14] Joyce Ehrlinger, Thomas Gilovich, and Lee Ross. 2005. https://doi.org/10.1177/0146167204271570 Peering into the bias blind spot: People's assessments of bias in themselves and others . Personality and Social Psychology Bulletin, 31(5):680--692 on Pithdoi:10.1177/0146167204271570
- [15] Epoch AI . 2026. DeepSWE . https://epoch.ai/benchmarks/deepswe. Accessed 2026-10-12 on Pith
- [16] Thomas Palmeira Ferraz, Kartik Mehta, Yu-Hsiang Lin, Haw-Shiuan Chang, Shereen Oraby, Sijia Liu, Vivek Subramanian, Tagyoung Chung, Mohit Bansal, and Nanyun Peng. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.458 LLM self-correction with D e CRIM : Decompose, critique,... on Pithdoi:10.18653/v1/2024.findings-emnlp.458
- [17] Gemma Team . 2026. https://doi.org/10.48550/arXiv.2607.02770 Gemma 4 technical report . Preprint, arXiv:2607.02770 on PithPith's reviewarXivdoi:10.48550/arxiv.2607.02770
- [18] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. https://doi.org/10.48550/arXiv.2310.01798 Large language models cannot self-correct reasoning yet . In International Conference on Learning Representations (ICLR). A... on PithPith's reviewarXivdoi:10.48550/arxiv.2310.01798
- [19] Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. 2026. https://doi.org/10.48550/arXiv.2607.07946 DeepSWE : Measuring frontier coding agents on original, long-horizon engineering tasks . Preprint, arXiv:2607.07946 on PithPith's reviewarXivdoi:10.48550/arxiv.2607.07946
- [20] Zhonghao Jiang, Le Deng, Jialun Cao, Michael Pradel, and Zhongxin Liu. 2026. https://doi.org/10.1145/3832783.3837489 Doc2Feat-Bench : Evaluating documentation-driven feature addition . In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineeri... on PithPith's reviewarXiv
show 48 more
- [21] Manas Joglekar, Jeremy Chen, Gabriel Wu, Jason Yosinski, Jasmine Wang, Boaz Barak, and Amelia Glaese. 2025. https://doi.org/10.48550/arXiv.2512.08093 Training LLMs for honesty via confessions . arXiv preprint arXiv:2512.08093 on Pithdoi:10.48550/arxiv.2512.08093
- [22] Jean Kaddour, Srijan Patel, Gb \`e tondji Dovonon, Leo Richter, Pasquale Minervini, and Matt J. Kusner. 2026. https://doi.org/10.48550/arXiv.2602.06948 Agentic uncertainty reveals agentic overconfidence . Preprint, arXiv:2602.06948 on Pithdoi:10.48550/arxiv.2602.06948
- [23] Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. https://doi.org/10.1162/tacl_a_00713 When can LLMs actually correct their own mistakes? a critical survey of self-correction of LLMs . Transactions of the Association for Computational Linguistics, 12:1417--1440 on Pithdoi:10.1162/tacl_a_00713
- [24] Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohua Dong. 2024. https://doi.org/10.18653/v1/2024.naacl-long.228 Better zero-shot reasoning with role-play prompting . In Proceedings of the 2024 Conference of the North American Ch... on Pithdoi:10.18653/v1/2024.naacl-long.228
- [25] Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. 2020. https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/ Specification gaming: the flip side of AI inge... on Pith
- [26] Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. 2025... on PithPith's reviewarXivdoi:10.48550/arxiv.2409.12917
- [27] Ziva Kunda. 1990. https://doi.org/10.1037/0033-2909.108.3.480 The case for motivated reasoning . Psychological Bulletin, 108(3):480--498 on Pithdoi:10.1037/0033-2909.108.3.480
- [28] Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. 2023. https://arxiv.org/abs/2307.11760 Large language models understand and can be enhanced by emotional stimuli . Preprint, arXiv:2307.11760 on PithPith's reviewarXiv
- [29] Simiao Liu, Fang Liu, Liehao Li, Xin Tan, Yinghao Zhu, Xiaoli Lian, and Li Zhang. 2025. https://doi.org/10.48550/arXiv.2509.13941 An empirical study on failures in automated issue solving . Preprint, arXiv:2509.13941 on Pithdoi:10.48550/arxiv.2509.13941
- [30] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. https://doi.org... on PithPith's reviewarXiv
- [31] Samuel Marks, Johannes Treutlein, Trenton Bricken, Jack Lindsey, Jonathan Marcus, Siddharth Mishra-Sharma, et al. 2025. https://doi.org/10.48550/arXiv.2503.10965 Auditing language models for hidden objectives . arXiv preprint arXiv:2503.10965 on PithPith's reviewarXivdoi:10.48550/arxiv.2503.10965
- [32] Aman Mehta and Anupam Datta. 2026. https://doi.org/10.48550/arXiv.2606.22936 When agents commit too soon: Diagnosing premature commitment in LLM agents . arXiv:2606.22936 on PithPith's reviewarXivdoi:10.48550/arxiv.2606.22936
- [33] Xiangxin Meng, Zexiong Ma, Pengfei Gao, and Chao Peng. 2026. https://doi.org/10.1145/3744916.3773269 LLM -based agents for automated bug fixing: How far are we? In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering (ICSE), pages 855--867 on Pithdoi:10.1145/3744916.3773269
- [34] Mercor . 2026. DeepSWE v1.1 leaderboard. https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/. Accessed 2026-10-12 on Pith
- [35] Muse Glimmer Team . 2026. https://doi.org/10.48550/arXiv.2610.24628 Muse Glimmer 30B : An on-device agentic model . Preprint, arXiv:2610.24628 on PithPith's reviewarXivdoi:10.48550/arxiv.2610.24628
- [36] Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2024. https://doi.org/10.48550/arXiv.2306.09896 Is Self-Repair a silver bullet for code generation? In The Twelfth International Conference on Learning Representations (ICLR 2024) on PithPith's reviewarXivdoi:10.48550/arxiv.2306.09896
- [37] OpenAI . 2026 a . 1 GPT-5.6 system card. https://deploymentsafety.openai.com/gpt-5-6. Published July 9, 2026 on Pith
- [38] OpenAI . 2026 b . 2 S eparating signal from noise in coding evaluations. https://openai.com/index/separating-signal-from-noise-coding-evaluations/. Published July 8, 2026 on Pith
- [39] OpenAI . 2026 c . 3 C odex. https://github.com/openai/codex. Accessed 2026-10-12 on Pith
- [40] Alexander Pan, Kush Bhatia, and Jacob Steinhardt. 2022. https://doi.org/10.48550/arXiv.2201.03544 The effects of reward misspecification: Mapping and mitigating misaligned models . In International Conference on Learning Representations (ICLR) on PithPith's reviewarXivdoi:10.48550/arxiv.2201.03544
- [41] Alexander Pan, Erik Jones, Meena Jagadeesan, and Jacob Steinhardt. 2024. https://doi.org/10.48550/arXiv.2402.06627 Feedback loops with language models drive in-context reward hacking . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proce... on PithPith's reviewarXivdoi:10.48550/arxiv.2402.06627
- [42] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. https://doi.org/10.52202/079017-2197 LLM evaluators recognize and favor their own generations . In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), pages 68772--68802 on Pithdoi:10.52202/079017-2197
- [43] Emily Pronin, Daniel Y. Lin, and Lee Ross. 2002. https://doi.org/10.1177/0146167202286008 The bias blind spot: Perceptions of bias in self versus others . Personality and Social Psychology Bulletin, 28(3):369--381 on Pithdoi:10.1177/0146167202286008
- [44] Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. 2024. https://doi.org/10.52202/079017-1754 Recursive introspection: Teaching language model agents how to self-improve . In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), pages 55249--55285 on Pithdoi:10.52202/079017-1754
- [45] Qwen Team . 2026. Qwen3.8-Max : A new bar for coding and cowork. https://qwen.ai/blog?id=qwen3.8 on Pith
- [46] Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2025. https://doi.org/10.1109/ICSE55347.2025.00080 SpecRover: code intent extraction via LLMs . In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE 2025), pages 963--974. ArXiv:2408.02232 on PithPith's reviewarXiv
- [47] Scale AI . 2026. SWE-Bench Pro (public dataset, v2) leaderboard. https://labs.scale.com/leaderboard/swe_bench_pro_public_v2. Accessed 2026-11-04 on Pith
- [48] Murray Shanahan, Kyle McDonell, and Laria Reynolds. 2023. https://doi.org/10.1038/s41586-023-06647-8 Role play with large language models . Nature, 623(7987):493--498 on Pithdoi:10.1038/s41586-023-06647-8
- [49] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. https://doi.org/10.52202/075280-0377 Reflexion: language agents with verbal reinforcement learning . In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), pages 8634--865... on PithPith's reviewarXiv
- [50] Herbert A. Simon. 1955. https://doi.org/10.2307/1884852 A behavioral model of rational choice . The Quarterly Journal of Economics, 69(1):99--118 on Pithdoi:10.2307/1884852
- [51] Herbert A. Simon. 1956. https://doi.org/10.1037/h0042769 Rational choice and the structure of the environment . Psychological Review, 63(2):129--138 on Pithdoi:10.1037/h0042769
- [52] Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. https://doi.org/10.52202/068431-0687 Defining and characterizing reward gaming . In Advances in Neural Information Processing Systems 35 (NeurIPS 2022), pages 9460--9471 on Pithdoi:10.52202/068431-0687
- [53] Edward K. Smith, Earl T. Barr, Claire Le Goues, and Yuriy Brun. 2015. https://doi.org/10.1145/2786805.2786825 Is the cure worse than the disease? overfitting in automated program repair . In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE... on Pithdoi:10.1145/2786805.2786825
- [54] Xiaoshuai Song, Yanan Wu, Weixun Wang, Jiaheng Liu, Wenbo Su, and Bo Zheng. 2025. https://doi.org/10.18653/v1/2025.acl-short.73 P rog C o: Program helps self-correction of large language models . In Proceedings of the 63rd Annual Meeting of the A ssociation for C omputational ... on Pithdoi:10.18653/v1/2025.acl-short.73
- [55] Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2025. https://doi.org/10.48550/arXiv.2402.08115 On the self-verification limitations of large language models on reasoning and planning tasks . In The Thirteenth International Conference on Learning Representations on PithPith's reviewarXivdoi:10.48550/arxiv.2402.08115
- [56] Gladys Tyen, Hassan Mansoor, Victor Carbune, Peter Chen, and Tony Mak. 2024. https://doi.org/10.18653/v1/2024.findings-acl.826 LLMs cannot find reasoning errors, but can correct them given the error location . In Findings of the Association for Computational Linguistics: ACL 2... on Pithdoi:10.18653/v1/2024.findings-acl.826
- [57] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, and 5 others. 2025. https:... on PithPith's reviewarXivdoi:10.48550/arxiv.2407.16741
- [58] You Wang, Michael Pradel, and Zhongxin Liu. 2026. https://doi.org/10.1145/3744916.3764576 Are ``solved issues'' in SWE -bench really solved correctly? an empirical study . In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering (ICSE), pages 169--181 on Pithdoi:10.1145/3744916.3764576
- [59] Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R. Bowman, He He, and Shi Feng. 2025. https://doi.org/10.48550/arXiv.2409.12822 Language models learn to mislead humans via RLHF . In International Conference on Learning Representations (ICLR) on PithPith's reviewarXivdoi:10.48550/arxiv.2409.12822
- [60] Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.167 Large language models are better reasoners with Self-Verification . In Findings of the Association for Computational L... on PithPith's reviewarXiv
- [61] Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Wang. 2024. https://doi.org/10.18653/v1/2024.acl-long.826 Pride and prejudice: LLM amplifies self-bias in self-refinement . In Proceedings of the 62nd Annual Meeting of the A ssociation for C omputationa... on Pithdoi:10.18653/v1/2024.acl-long.826
- [62] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. https://doi.org/10.52202/079017-1601 SWE -agent: Agent-computer interfaces enable automated software engineering . In Advances in Neural Information Processing ... on Pithdoi:10.52202/079017-1601
- [63] Pamela Zave and Michael Jackson. 1997. https://doi.org/10.1145/237432.237434 Four dark corners of requirements engineering . ACM Transactions on Software Engineering and Methodology, 6(1):1--30 on Pithdoi:10.1145/237432.237434
- [64] Zhengran Zeng, Yixin Li, Rui Xie, Wei Ye, and Shikun Zhang. 2025. https://doi.org/10.48550/arXiv.2511.04064 Benchmarking and studying the LLM -based agent system in end-to-end software development . Preprint, arXiv:2511.04064 on Pithdoi:10.48550/arxiv.2511.04064
- [65] Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. 2024 a . https://doi.org/10.48550/arXiv.2305.13534 How language model hallucinations can snowball . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Mach... on PithPith's reviewarXivdoi:10.48550/arxiv.2305.13534
- [66] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024 b . https://doi.org/10.1145/3650212.3680384 AutoCodeRover: autonomous program improvement . In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2024), pages ... on PithPith's reviewarXiv
- [67] Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. 2026. https://doi.org/10.48550/arXiv.2510.20270 ImpossibleBench : Measuring LLMs ' propensity of exploiting test cases . In The Fourteenth International Conference on Learning Representations (ICLR) on Pithdoi:10.48550/arxiv.2510.20270
- [68] Qixing Zhou, Jiacheng Zhang, Haiyang Wang, Rui Hao, Jiahe Wang, Minghao Han, Yuxue Yang, Shuzhe Wu, Feiyang Pan, Lue Fan, Dandan Tu, and Zhaoxiang Zhang. 2026. https://doi.org/10.48550/arXiv.2602.10975 FeatureBench : Benchmarking agentic coding for complex feature development ... on Pithdoi:10.48550/arxiv.2602.10975
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment